The Case Against Economic Values in the Orbitofrontal Cortex (or Anywhere Else in the Brain)

The past 20 years have seen a great deal of interest in understanding how our brains implement economic choices  (Camerer et al., 2005; Glimcher & Fehr, 2013; Loewenstein et al., 2008; Padoa-Schioppa, 2011; Rangel et al., 2008; Rushworth et al., 2011) . Much research in this field of neuroeconomics rests on the assumption that choices between options rely on an explicit valuation process  (Kable & Glimcher, 2009; Levy & Glimcher, 2012; Montague & Berns, 2002; O'Doherty, 2014; Padoa-Schioppa, 2011) . That is, that the brain first assigns value to each option and then compares those values to determine choice. The concept of valuation-which stems from economic theory-is so ingrained that it may seem inevitable. How else would one literally compare apples to oranges? Indeed, the idea that value exists on a single cardinal scale, also called a "common currency," has been extended to encompass not only goods, but also effort costs and time delays; everything an agent needs to bundle into evaluation of a mode of action that is aimed at procuring a specific outcome. In parallel, the computational framework of reinforcement learning, which has been a cornerstone of neuroeconomics, also makes a scalar value signal central to its implementation  (Niv, 2009; Sutton & Barto, 2018) . Specifically, in reinforcement learning, the value of an option-a state or an action-is the expected sum of future rewards contingent upon that choice. As a sum of future rewards that may be of different types (and include costs as negative rewards), reinforcement-learning value is naturally calculated in some unitless common currency. However, not all reinforcement learning algorithms rely on or even calculate values  (Sutton & Barto, 2018) , and reinforcementlearning values, as sums of future rewards, are not synonymous with economic values of specific goods. Likewise, many empirically supported process models of choice get by with no valuation  (Gigerenzer & Gaissmaier, 2011; Miller et al., 2019; Vlaev et al., 2011) . The fact that the brain can compute values to compare apples and oranges does not mean that it routinely does so, or that valuation is the primary process underlying choice. In this opinion paper, we argue that in many choice scenarios the brain may not be computing values at all, despite appearing to do so. We will demonstrate that the rationale for a value signal is weak, as are both behavioral and neural evidence supporting value computation. Finally, we propose an alternative-direct learning of action policies-and suggest there is scant direct evidence for value learning that cannot be explained by this and other alternative theories. Our hypothesis is important as it suggests a different interpretation of previous data and requires that studies attempting to resolve mechanisms of value computation in the brain first establish that in the specific situation studied, valuation is indeed occurring. In particular, much work has associated the orbitofrontal cortex (OFC) in representing the expected economic value of different goods or options  (Bartra et al., 2013; Levy & Glimcher, 2012 )-a role that must be critically reconsidered if we agree that the brain may not necessarily be representing such values in many of the experiments so far used to test this hypothesis.
 Why Argue Against Value? Intuitively, our thesis is that while we may know that we prefer an orange to an apple (for one thing, oranges don't brown when exposed to oxygen), we may make this judgment without consulting an internal scalar (cardinal) or even a universal ordinal value signal. Consequently, we may be hard-pressed to express the precise value that we put on an orange. This is not merely an issue of conscious access to our internal value of oranges 1 , but may be due to the fact that deciding on preferences can utilize many alternative mechanisms that don't require or rely on calculation of such a value. Valuation is hard  (Payne et al., 1992) . It is also often unnecessary: when choosing between, say, an orange and a car, it is immediately clear that one is better than the other without calculating the precise value of either. If you are extremely thirsty, you might choose the orange, whereas if you need to go somewhere a car is the only relevant option. Arguably, many real-life choices are between options that are sufficiently different in the needs that they fulfill as to be more similar to this extreme example than they are to the choice between apples and oranges  (Juechems & Summerfield, 2019) . Valuation may only be necessary when choosing between two very similarly valued items. However, if the items are sufficiently similar in how they satisfy our needs, the brain may decide to choose randomly, according to some valuefree heuristic, or according to past choices-and move on  (Chater, 2018) . Alternatively, the brain can try to calculate the exact values of the options to the necessary precision that arbitrates between them. Indeed, choosing between similar options often takes longer than choosing between very different options, even when making rather trivial choices that do not warrant the time and effort invested  (Pirrone et al., 2018; Teodorescu et al., 2016 ). An example is the inordinate amount of time some of us may spend deciding what brand of tuna fish to buy, despite the fact that our time (as per our salary) is worth much more than the money we will save by correctly evaluating which brand provides the best "value for money." This scenario also illustrates how inept we are at comparing the value of different kinds-here, time and money-which should have been straightforward if the modus operandi of the brain was to evaluate everything in terms of some common currency (see below). So perhaps the brain can, when needed, calculate values. However, we argue that this is not the main means by which the brain makes decisions, and perhaps not the natural mode of decision making.
 What Is Value? Before moving on, we would like to delineate precisely how we are defining value for the sake of our argument. The term "value" has multiple uses (the problems this multiplicity raises are carefully laid out by O'Doherty, 2014). We consider "value" a hypothesized scalar variable that reflects the worth of a specific item or outcome. 2 Because it is scalar, it is necessarily abstract. It can refer to any good, and makes use of a common currency code that is comparable across goods of different types (e.g., food, water, recreation time). Value, by this definition, is cardinal, not ordinal, meaning it can be defined for an option per se, rather than solely relative to other options. That is, it reifies the idea of "utils"-quantifiable units of value, often used in a jocular manner in Economics classes. The idea of a common currency is key as it implies that all relative calculations have already happened-the value of an option is in some denomination that objectively defines it as compared to other such values. One might argue that this is too narrow a definition, but we believe this is what neuroeconomists have in mind when talking about representation of common currency values in the brain. For example, we know of no neuroeconomic model that imagines a neural implementation of an ordinal value scale. This having been said, even an ordinal scale should not change based on the comparison set, and should satisfy transitivity-which many of the neural signals attributed to value do not, as we detail below. Our definition of value, for the purpose of this paper, is different from other types of value that are discussed in reinforcement learning. Specifically, here we are discussing value as the reward worth of a single item/event, not the expected sum of all future rewards (R in reinforcement learning models, not V). One might argue that in order to compute such an expected sum V, the subjective worth of all individual rewards must be translated to a common currency that can be added. In this sense, reinforcement learning models do presuppose a common currency reward value. However, they don't necessarily commit to economic properties of such values, such as transitivity and consistency (see also  Juechems & Summerfield, 2019 ), and we discuss below a class of reinforcement learning algorithms that can make do with evaluations that are only relative to the current options, and not applicable in other situations. One important feature of our definition of value is that in our view, although value is inferred from choice, it is not strictly identical to choice, nor necessarily implied by choice. For instance, if we observe a consistent preference for A over B, A is assumed to have a higher value than B. But choice of A over B isn't sufficient to infer the existence of a self-consistent value function: a decisionmaker may adhere to a heuristic policy that results in stable preference for  Slovic, 2006) . Moreover, choice can be altered without manipulating the reward value of an item  (Schonberg & Katz, 2020) , which suggests that value, as inferred from choice, is not untarnished by processes that are not economic in nature. Thus, value can be inferred from behavior given certain assumptions, but preferences do not always lead to a value function. [A > B], [B > C], and [C > A] (Lichtenstein & This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
 The Common-Currency Hypothesis The notion that economic decision-makers make use of internal value functions to compare options is often dated to Daniel Bernoulli's proposal in 1738 of a logarithmic utility curve to explain preferences in the St. Petersburg Paradox  (Martin, 2011) . Here, a decision-maker is offered a chance to play a game in which they win $2 (n−1) with n being the first time, in a series of coin tosses, that a coin falls on "heads." The paradox is that although the expected monetary value of this game (that is, the sum of expected wins multiplied by their probability P ∞ n=1 2 ðn−1Þ • 0.5 n = P ∞ n=1 0.5) is infinity, people are not willing to pay even $10 to play this game  (Hayden & Platt, 2009) . The explanation that Bernoulli proposed proved foundational within microeconomic theory. He argued that decision-makers don't base calculations on the nominal, objective cash value of the potential gain, but rather on its subjective value. If the subjective value grows more slowly than the objective value (the idea of "diminishing marginal utility"), an optimizing decision-maker will appear riskaverse, for instance, when evaluating the St. Petersburg gamble. Utility, or subjective value, has been central to many if not most microeconomic models. These include the axiomatic approaches of Pareto, Von Neumann and Morgenstern, and Samuelson. It is also central to behavioral theories such as prospect theory and decision field theory  (Busemeyer & Townsend, 1993; Kahneman & Tversky, 1979) . Ironically, however, explaining choices in the St. Petersburg paradox using such a subjective value function requires utility for money that diminishes so rapidly that it does not generalize to choices in other contexts. Instead, heuristic accounts-accounts that don't rely on ideas about value maximization-provide better quantitative matches for St. Petersburg choices  (Hayden & Platt, 2009) . Despite its central importance in economics, economists are typically agnostic about whether the concept of value is just a convenient description, or whether it is instantiated in the brain. As Friedman argued, decision-makers behave as if we compute and compare values, but we cannot conclude that we actually do so  (Friedman, 1953) . In that work, he famously compared economic agents to a trained billiards player who makes excellent shots as if having a sophisticated grasp of Euclidean geometry, although in fact such a theoretical understanding is not necessary for good billiards skill. Economists typically stop at the point of saying that economic models are "as if" models, with some arguing that the question of the underlying reality of economic variables is outside the domain of economics  (Gul & Pesendorfer, 2008; Harrison, 2008) . In contrast, neuroeconomics has generally taken as a default assumption that these "as if" theories are reified in the brain  (Kable & Glimcher, 2009; Levy & Glimcher, 2012; Montague & Berns, 2002; O'Doherty, 2014; Padoa-Schioppa, 2011; Rich & Wallis, 2016) . Modern tools such as neuroimaging and single unit physiology provide the opportunity to assess the implementation of decision making directly. Indeed, neuroscientists have had little trouble identifying correlates of value in several brain areas  (Knutson et al., 2001; Levy & Glimcher, 2012; Plassmann et al., 2007; Wallis, 2007) , leading to the suggestion that the OFC is a nexus for representing the economic value of goods in the brain (see, e.g.,  Rushworth et al., 2011; Wallis, 2007) , alongside other brain areas that are important for computing and representing value such as the ventromedial prefrontal cortex  (Bartra et al., 2013)  and ventral striatum  (Haber & Knutson, 2010) . However, it is not clear that the signals identified in these studies actually represent value as proposed-on a common-currency scale, for comparing options and making choices. Moreover, so called "value signals" only correlate with value, so may be signaling other quantities, such as attention, action plans, vigor, or preference, which also correlate with value  (Maunsell, 2004; O'Doherty, 2014; Wallis & Rich, 2011) . Hence it is important to carefully consider alternative interpretations for these findings, as we do further below, after discussing some practical constraints and philosophical conundrums.
 Alternatives to Valuation in Decision Making While common-currency value provides a convenient and general mechanism for making choices, there are many possible alternatives  (Gigerenzer & Gaissmaier, 2011; Kahneman et al., 1982; Lichtenstein & Slovic, 2006; Vlaev et al., 2011) , some of which are quite general and robust. Consider for example the "priority heuristic"  (Brandstätter et al., 2006) . This de minimis heuristic approach proposes that decision-makers first identify a dimension along which options vary and then compare options along that dimension. If that results in a choice, they stop; otherwise, they move on to the next dimension. This heuristic can explain many phenomena, including Allais preferences  (Allais, 1953) , the reflection effect  (Fishburn & Kochenberger, 1979) , the certainty effect  (Kahneman & Tversky, 1979) , the fourfold pattern in risky choice that motivates prospect theory  (Tversky & Fox, 1995) , and several intransitivities-all without ever requiring computing of value  (Brandstätter et al., 2006) . And the priority heuristic is one of a large number of heuristics that do a remarkable job at describing behavior  (Gigerenzer & Gaissmaier, 2011) . These heuristics are generally motivated by psychological observations, and thus are consistent with known data from the psychology-if not the neuroscience-of choice. In particular, they reflect the assumption that calculating value is difficult, that humans typically use shortcuts whenever possible, and that heuristics are a good shortcut  (Gigerenzer & Gaissmaier, 2011; Lieder & Griffiths, 2020; Payne et al., 1992) . Moreover, these heuristics are not limited to humans, but apply to other species, including monkeys  (Heilbronner & Hayden, 2016; Marsh, 2002; Santos & Rosati, 2015; Shafir et al., 2002) . Importantly for our argument, the success of heuristic approaches demonstrates that value calculations are not a priori essential for neuroeconomic theories of choice. In particular, heuristic theories can solve many problems for which value is proposed to be needed  (Kahneman et al., 1982; Lichtenstein & Slovic, 2006; Piantadosi & Hayden, 2015; Stevens, 2016; Tversky, 1969; Vlaev et al., 2011) . Heuristics readily allow for comparison of multi-dimensional goods and bundles, and for choice across dissimilar goods. While heuristics are not perfect, and lead to many choice anomalies, choice is, empirically, full of anomalies. Moreover, several non-heuristic process models also eschew value computation steps, including decision by sampling, query theory, and fuzzy trace theory; these all account for a wide range of choice behavior as well  (Reyna, 2008; Stewart & Simpson, 2008; Stewart et al., 2006; Weber et al., 2007) .
 Practical Issues in the Neuroscience of Value Representation Despite the above, it is often considered axiomatic that an internal, neural, value scale must exist in the brain. The job of neuroscientists is then to find this signal-to pinpoint brain activity This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. that correlates with value. In this section, we challenge this viewpoint by considering some basic issues that come up in the neuroscience of value. The first problem is that it is impossible to precisely measure value. We can measure preferences between options and use the data to infer option values, but elicited preferences are noisy measurements that also reflect factors other than value. For example, depending on how preferences are elicited, they may also reflect the tendency to press the same button repeatedly rather than change actions, the tendency to switch between options due to a prior belief about depleting resources, or the amount of attention or looking time for each of the options  (Armel et al., 2008; Schonberg & Katz, 2020; Shimojo et al., 2003; Sugrue et al., 2005) . Finally, value may shift from trial to trial, even depending on recent outcomes, meaning that methods that average across trials produce misleading value estimates  (Sugrue et al., 2005) . One could, in theory, incorporate these factors into the inferred value of an option. For example, there may be inherent value in choosing the same thing twice in a row. However, it becomes unclear what the definition of the "option" is that is being evaluated: if the value of an apple after eating (or choosing, or even just viewing) an orange is different from the value of an apple without that proximal experience, can we determine the value of goods at all? And if we can change the value (read: preference) for an option just by directing more attention to it  (Salomon et al., 2018; Schonberg et al., 2014) , or inducing choice of it  (Izuma et al., 2010; Sharot et al., 2012; Voigt et al., 2017) , is preference really measuring the subjective economic worth of a good? The risk is circularity-if any choice behavior can be explained by supposing value that depends on local history, then the concept of value adds no additional explanatory power beyond that of recent events and choices. Different ways to measure value raise a second challenge of inconsistent measures. Indeed, if we had a single neural value function we called on, values elicited by different measures would match. Unfortunately, they do not  (Lichtenstein & Slovic, 2006) . To explain this, in their seminal work, Sarah Lichtenstein and Paul Slovic proposed that preferences do not arise from any internal value function, but instead are constructed at the time of elicitation (see also  Ariely et al., 2003; Payne et al., 1992; Tversky & Shafir, 1992) . That is, in the view of these and like-minded scholars, value doesn't sit in the brain waiting to be used; rather, preference is a complex and active process that takes place at the time the decision is made. Critically, in this theory we compute choices in a largely ad-hoc manner based on the available options, without an intermediary common-currency valuation stage. As such, there is no guarantee of consistency or reliability; any consistency or reliability observed may be explained as a result of strong attractor states in the way the system determines the choice, and deviations from consistency are evidence for the specific nature of the algorithm and its idiosyncrasies. This hypothesis, while denouncing value as a latent construct in the brain, nevertheless invites neuroscientific research to understand the active processes involved in choice, and how these relate to (relative) evaluation, if not to economic values. This having been said, experimenters can roughly estimate value, even if not measure it precisely. For example, in a given experiment they can examine choices and determine with confidence that a monkey (behaves as if it) places more value on a gamble as compared to a safe option with a matched expected value. One could then use this fact to try to identify a neural correlate of value. However, for this neural endeavor to be valid, it is important to identify all confounding variables and regress them out. O'Doherty's (2014) review delineates the practical difficulty of doing so, using the overall rubric of "visceral, autonomic, and skeletomotor" activity. In practice, confounding variables include both stimulus and outcome identity, information about the state or structure of the world, the surpriseness, informativeness, and informational value of stimuli, details of the action associated with selecting of consuming the reward, including its likelihood and vigor, and the attention and arousal engendered by the stimulus (e.g.,  Blanchard et al., 2015; Botvinik-Nezer et al., 2020; Niv et al., 2007; O'Doherty, 2007; Roesch & Olson, 2004; Roesch et al., 2006; Wilson et al., 2014; . Indeed, in studies that separately assess encoding of outcome identity versus outcome value, activity in brain areas that are often considered to be emblematic of economic value (in particular, the OFC) turns out to correlate with outcome identity instead  (Klein-Flügge et al., 2013) . A final insurmountable problem is that it may be impossible, even in theory, to obtain a brain measure of value that is independent of behavior. Suppose for example, that we identify a particular class of neurons whose firing rates are perfectly correlated with value down to our ability to measure it through preference. Supporting this idea, we observe that any procedure that modifies value (as inferred from behavior) changes the firing rate of these neurons in a manner consistent with our predictions. We may tentatively hypothesize that these neurons are (or are among) the value neurons of the brain. However, as shown by Schonberg and colleagues, preference can be changed irrespective of changing the economic worth of goods  (Schonberg & Katz, 2020) . Therefore, to differentiate value neurons from preference neurons, we would need to show that these neurons do not strictly follow expressed preference when the value function diverges from it. This is, of course, not possible if the value function never measurably diverges from preference. And if we assume that values can diverge from preference, it is not clear how to define values to start with. We call this "the neuroeconomic relativity problem" because, like Einstein's relativity problem, it reflects the fact that there is no external reference frame to which one can calibrate value inferences.
 Reconsidering the Motivation for Common Currency It may be worth asking, then, what does having a single common currency buy you? Why would the brain invest in such an organization? One advantage of a common-currency scale is that it simplifies comparing options that differ along multiple dimensions. For example, when hunting for an apartment, the options may differ along dimensions of price, area, neighborhood, and amenities. The logic is that these dimensions must be first combined into a single scalar per apartment so that the scalars can be compared. However, this is not the only way to solve the problem, and importantly, may not be the way humans make their decisions. For example, the apartment shopper may choose a single dimension and pick the winner along that dimension, as discussed above, or may compare separately on each dimension and choose the apartment that wins on most counts  (Tversky, 1972) . Laboratory studies where humans can choose what attributes to view indeed suggest that people don't uncover all the attributes of one option (to calculate its value) and then continue to the next option, but rather prefer to view information for all options attribute by attribute  (Fellows, 2006; Hunt et al., 2014) . Indeed, as mentioned, much empirical work indicates that human decision-makers broadly favor heuristic approaches that This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. THE CASE AGAINST ECONOMIC VALUES eschew a value stage in a large number of contexts  (Brandstätter et al., 2006; Gigerenzer & Gaissmaier, 2011; Kahneman et al., 1982; Lieder & Griffiths, 2020) . Notably, systems that make use of heuristics may generate internal variables that are conceptually distinct from value but that correlate with value, thus leading to an interpretational confound. Consider, for example, a relatively well-understood implementation of choice in a (non-brain) distributed system: the selection of hive sites in bees (Apis mellifera,  Seeley, 2010; Seeley & Buhrman, 1999; Seeley et al., 2006) . Bee swarms select a hive site by sending out scouts to investigate potential sites. Each site differs along roughly 20 dimensions of varying but measurable importance (size, safety from predators, exposure to sunlight, wind, etc.). Each scout bee that encounters the hive site performs an extremely poor estimation of the value of the site-it typically will only sample three-to-four of the dimensions and even then, estimate them poorly. The bee then returns and if the estimated site quality is sufficiently good, indicates an assessment of its quality to the other bees in the swarm. The quality it signals will be correlated with the overall value of the hive site, but only weakly, and, critically, will only integrate a subset of relevant value components. The overall comparison between options is indirect-the options race to attract adherents; the majority of the decision is made by a positive feedback process. This example is particularly relevant because scholars of perceptual decision making may recognize this process as strongly related to ideas about how the brain decides between options by racing to bounds, and there is evidence that deliberative processes in the brain follow the same principles as choice processes in bee swarms  (Eisenreich et al., 2017; Franks et al., 2003; Mitchell, 2009; Pais et al., 2013; Passino et al., 2008; Pirrone et al., 2018) .
 Evaluating the Neural Evidence for the Common-Currency Hypothesis Evidence for the common-currency hypothesis comes from the observation that firing rates of single neurons or hemodynamic responses of voxels correlate with values of offers and outcomes  (Kennerley et al., 2009 (Kennerley et al., , 2011 Levy & Glimcher, 2012; Padoa-Schioppa, 2011; Rangel et al., 2008) . These responses depend on multiple elements of offers (e.g., the expected reward, as well as the associated response costs), and are modulated by factors that affect subjective value, such as context or level of satiety (see  Conen & Padoa-Schioppa, 2019; O'Neill & Schultz, 2010; Rudebeck et al., 2017  for only a few examples). Often, variations in these firing rates predict variations in choice  (Conen & Padoa-Schioppa, 2015; Strait et al., 2014; Sugrue et al., 2004) . Such patterns have been taken as evidence that the neural signal encodes value, in particular implicating the OFC  (Bartra et al., 2013; Levy & Glimcher, 2012; O'Reilly, 2020) . However, such patterns of neural responding do not definitively demonstrate a common-currency value signal. For example, in the original finding of value-encoding neurons in the OFC  (Tremblay & Schultz, 1999) , neurons responded more strongly to a high-valued option than to a low-valued one. However, the principle of common currency only held within a given choice pair-responses for Option B were low when this option was paired with a better Option A, but were high when this same option was paired with a worse Option C. Other studies found value-correlated responses to be more stable across contexts  (Padoa-Schioppa & Assad, 2008) , but most findings support an encoding that depends on the alternatives (e.g.,  Kobayashi et al., 2010; Padoa-Schioppa, 2009; Zimmermann et al., 2018) . One might interpret this finding as reflecting simple range adaptation of neurons that have a finite firing rate; however, a more parsimonious explanation is that the firing pattern is consistent with a relative preference code rather than an abstract value code. This interpretation places the putative neural representation of value one stage later than we would expect from a common-currency codeimmediately after comparison (because a relative valuation is itself a comparison) rather than as an input to comparison, raising the question of where the input came from. An interpretation of this code as something other than economic value would obviate this worry. Importantly, with a code that depends on alternatives, there is no real sense in which we can read out the "true subjective value" of an option. We can only know if an option's value is higher than another value-similar to the information provided by preferences and choice behavior. Another corollary of such a relative code is that it is unclear to what extent we can read out a meaningful value signal when only one option is available. Indeed, presenting a subject with one option at a time should, theoretically, provide the best ability to read out option values from neural activity in areas associated with value, such as OFC, ventromedial prefrontal cortex, and dorsal and subgenual anterior cingulate cortices. Doing so reveals that while neural activity in these regions does correlate with the value of the first option, neural responses to the second (alternative) option (presented on its own) correlate with the value difference, that is, they reflect the result of value comparison rather than valuation per se  Hunt et al., 2018; Strait et al., 2014) . Moreover, even responses to the first offer do not simply encode its value, but also contain information about the likelihood that option will be chosen (presumably relative to the expected value of the second offer,  Azab & Hayden, 2017) . These results suggest that even when evaluation is experimentally segregated from comparison, pure value encodings may not exist. In fact, the idea that prefrontal neurons are selective or responsive to a single experimenter-defined variable has been increasingly falling out of favor  (Fusi et al., 2016; Raposo et al., 2014; Rigotti et al., 2013) , with "mixed selectivity" appearing to be a core operating principle of prefrontal cortex, including in ostensible value regions  (Blanchard et al., 2018; Hayden & Platt, 2010; Kimmel et al., 2020; . A third issue is that although the activity of some neurons in the OFC is correlated with value (but not necessarily linearly related to value), activity of other neurons in this same area is anticorrelated with value, with the majority of neurons showing no relationship at all with value. This raises the possibility that the neurons are encoding something other than value, for instance, a distributed representation of the identity of each of the three options, A, B, and C above, or the identity of the stimulus representing the offer. With the small number of options evaluated in most experiments, such a distributed code of outcome or stimulus identity (but not value) can easily result in some neurons randomly firing most strongly for the higher-valued of the three options and least for the worse option, while in other neurons the relationship would be in the opposite direction. This would also explain why many neurons show a nonmonotonic relationship between value and their firing. Recently, using ensemble recordings in the OFC that allow analysis on the level of a single trial, Wallis and colleagues have attempted a more direct test of the hypothesis that OFC encodes value  (Rich & Wallis, 2016) . In their task, monkeys were offered two options (denoted by images corresponding to the options) and asked to choose between them. On some trials, only one option was offered. Classifiers were trained to classify options corresponding to each of four offer-value levels (0.05, 0.10, 0.18, and 0.30 ml juice, or, in separate blocks, four levels of second-order reinforcer), aggregating over the two images corresponding to each value level. Using single-unit activity and local-field-potential recordings in the OFC, the authors could classify the offered value above chance, suggesting that OFC neurons encoded information about value. However, here too, offer values may have been encoded as different (outcome stimulus) identities, not different (cardinal) values in a common currency. One stringent test of the value-coding hypothesis would be to show that a combination of two offers of value level 0.05 ml resulted in similar activation pattern to a single offer of value level 0.10 ml. Ensemble recordings that give ample data in a single trial allow testing these hypotheses with novel combinations that have not been trained to predict the same identity of reward through multiple presentations, therefore testing the assumptions of the additivity of common currency directly. This critical test has not been conducted, to our knowledge. As mentioned, many factors that are conceptually distinct from value can influence choice-and are therefore closely correlated with value  (Maunsell, 2004) . Reward value drives attention, promotes both short-and long-term learning, primes behavioral adjustments, updates internal models, activates circuitry that detect both positive and negative surprises, and elicits mental computations of cost-benefit tradeoffs as well as comparisons with what could have been chosen. All of these are different from scalar, commoncurrency value, but are known to drive neural activity in the regions usually associated with economic value. As such, they confound that interpretation. This problem is a long-standing and notorious one in neuroeconomics  (Maunsell, 2004; O'Doherty, 2014; Roesch & Olson, 2003) . To overcome some of these potential confounds, a strong tradition in neuroeconomics research is to control for extraneous factors such as salience and response cost (although we note that many factors-for example, covert attention-cannot be controlled for). As a practical issue, it is difficult to control for alternative interpretations without making the task so convoluted that it becomes unnatural for the animal, in the ethological sense  (O'Doherty, 2014) . As a result, it is not clear if findings of value computation (if we believe them to be so) in these tasks would naturally translate to how the brain computes choice in naturalistic situations. But the deeper issue remains that value as inferred in most experiments is definitionally a summary of aggregated choice behavior, and therefore, any variable that influences choice behavior will necessarily be correlated with value. It is for this reason that we suggest that a stronger demonstration of scalar value coding in the brain should show mathematical properties such as the additivity of value, or separation from preferences as these are changed without changing value, as suggested above. If the response of neurons to the novel sum of two stimuli each promising 0.1 ml of juice is similar to the response of those same neurons to a different stimulus associated with 0.2 ml of juice, and especially if this response did not change when preference for an option was induced by means such as the mere exposure effect  (Schonberg & Katz, 2020) , it would be harder to argue that this is due to a shared motor plan, or attentional capture. We note that research to date has shown that the activity of brain areas associated with value, in particular the OFC, does change when preferences are modified through methods that should, in principle, not change economic value  (Botvinik-Nezer et al., 2020) .
 An Alternative View: Direct Learning of Policies Although reinforcement-learning models have contributed to the assumption that the brain computes values, many reinforcementlearning algorithms do not learn or estimate values for different actions. Instead, they directly learn action policies. In fact, in the reinforcement-learning literature, the goal of an agent is to obtain as much reward as possible by executing optimal actions-calculating values is only one means to achieve that end. In explaining reinforcement-learning methods,  Dayan and Abbott's (2001)  textbook begins with the "direct actor"-an actor that learns actions without computing their values. The Actor-Critic model, a prominent algorithm for reinforcement learning that has been linked to the brain  (Barto, 1995; Joel et al., 2002; Maia, 2010; O'Doherty et al., 2004; Takahashi et al., 2008) , does exactly that. In this algorithm, a Critic module learns values of states in terms of expected future rewards (here, the state includes all available actions, averaging over choices and explicitly not computing the value of each possible choice), and uses these to compute reward prediction errors. These prediction errors are used to learn an action policy in the Actor module: the probability of actions that are followed by positive prediction errors is increased, and the probability of actions that are followed by negative prediction errors is decreased. Under some reasonable conditions, this model learns correct reward-maximizing policies  (Sutton & Barto, 2018) . However, the quantities learned by the Actor-tendencies to perform one action over the other-cannot be read out as action values. In particular, due to the way the algorithm learns, ties are broken between equally good actions such that eventually agents learn deterministic policies. To be clear, if four actions were to lead to 1, 2, 3, and 3 drops of juice, respectively, the model may learn to always choose the third option, or to always choose the fourth (both optimal policies). At the end of learning, action weights in the Actor may be 0, 0, 1, 0 respectively, losing all information about value, or relative value. Even before convergence to a deterministic policy, weights may be 0, 0.1, 0.95, 0.2 respectively (or any other combinationweights here do not have to sum up to 1, and these are just illustrative numbers). Importantly, if the basal ganglia indeed implement an Actor-Critic learning algorithm in the brain, there is no sense in which we can glean action values from the Actor or the Critic. The Actor-Critic model is only one of a class of reinforcementlearning algorithms that learn policies directly, that is, without calculating option values (e.g.,  Sutton et al., 2000; Williams, 1992) . In their general form, these algorithms maintain an action policy, use experience to evaluate a gradient direction for this policy (that is, what change in policy would increase the overall obtained reward), and change the policy in that direction. Another notable model, developed to explain behavioral patterns in choices, is the Experience-Weighted Attraction model of  Camerer and Ho (1999) , which interpolates between value calculation and direct policy learning. Recent findings suggest that both behavior and learning signals measured in humans during decision making are more in line with a policy-learning algorithm than with value estimation. For instance, This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. THE CASE AGAINST ECONOMIC VALUES  Li and Daw (2011)  had participants choose between two options that gave reward with different probabilities. After each choice, both the outcome of the chosen option and the counterfactual outcome of the unchosen option were displayed. Behavior, as well as neural signals corresponding to prediction errors in the basal ganglia, suggested that subjects were updating both options in opposite directions, learning relative choice propensities (a policy) rather than tracking the expected value of each option. In another task in which humans chose between pairs of options and were able to view both the outcome of their choice and the counterfactual outcome of the forgone option,  Palminteri et al. (2015)  had subjects learn which of two probabilistically rewarding options is better, and which of two probabilistically punishing options was better. At test, subjects were asked to choose between pairs from both the rewarding and the punishing contexts. Surprisingly, when choosing between the less rewarding option and the less punishing option, subjects tended to choose the less punishing option. This is consistent with policy learning, as that option had been the favored option in the punishment context, whereas the less rewarding option had been the disfavored one in the reward context. However, the value of a sometimes-rewarding option is clearly higher than that of a sometimes-punishing option, hence value learning cannot explain this fundamentally suboptimal choice pattern. Direct learning of policies is consistent with basic tenets of decision making, such as the fact that choices are stochastic even when no exploration is necessary or warranted (e.g., choosing between two gambles that are fully described;  Khaw, Li, et al., 2017) . Because policies are relative quantities (preference for one option implies unfavorability of another, as the probabilities of all choices have to sum to one), they also explain common violations of the independence axiom such as the effects of third-option "decoys" on choice  (Soltani et al., 2012) , and temporal and spatial contextual influences on choice  (Khaw, Glimcher, et al., 2017) , although these phenomena can also be explained by valuation models that involve range normalization.
 Conclusion The field of neuroeconomics started with the putative identification of pure value signals  (Platt & Glimcher, 1999) . The meaning of these signals was disputed early on  (Maunsell, 2004; Roesch et al., 2006) , but deep questions were set aside as researchers continued to identify brain areas with parts of the computational processes of choice, and in particular, identified the OFC with the seat of economic value. However, while those debates have subsided, the problems they raised have not been resolved. Progress on these issues will require additional data, but we stress that not every experiment that involves choices also implies valuation, and one must be careful in interpreting data not only because of potential confounds, but also because we must be wary of treating a hypothesis-that the brain computes value-as an axiom. We also suggest the need for additional philosophical work to define value in a way that is-at least in principle-dissociable from other factors that promote choices  (Juechems & Summerfield, 2019) . In other words, we argue for a return to the productive debates of the early days of the field, 20 years ago (for neuroeconomics), and earlier (for its psychological underpinnings). Bolstered by an additional 20 years of new data, such debates would surely benefit the field of neuroeconomics moving forward. They would also potentially help reconcile conflicting views on what the OFC might or might not be doing in decision making  (Stalnaker et al., 2015) . 			 We note here that, in neuroeconomics, value-based choices are generally thought to be part of explicit, aware, goal-directed, or model-based decision making that relies on frontal-cortex areas, in particular, the OFC.2 We consider subjective worth as revealed by preference, following neuroeconomic theory, but note the limiting focus on only the "wanting" not the "liking" side of value (Berridge, 1996) .
			 This document is copyrighted by the American Psychological Association or one of its allied publishers.This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Abstract: Much of traditional neuroeconomics proceeds from the hypothesis that value is reified in the brain, that is, that there are neurons or brain regions whose responses serve the discrete purpose of encoding value. This hypothesis is supported by the finding that the activity of many neurons covaries with subjective value as estimated in specific tasks, and has led to the idea that the primary function of the orbitofrontal cortex is to compute and signal economic value. Here we consider an alternative: That economic value, in the cardinal, common-currency sense, is not represented in the brain and used for choice by default. This idea is motivated by consideration of the economic concept of value, which places important epistemic constraints on our ability to identify its neural basis. It is also motivated by the behavioral economics literature, especially work on heuristics, which proposes value-free process models for much if not all of choice. Finally, it is buoyed by recent neural and behavioral findings regarding how animals and humans learn to choose between options. In light of our hypothesis, we critically reevaluate putative neural evidence for the representation of value and explore an alternative: direct learning of action policies. We delineate how this alternative can provide a robust account of behavior that concords with existing empirical data.


Toby Ord
Rafaela Hillerbrand
Anders Sandberg
Francis Rjrr_A_412799 Sgm

Probing the improbable: methodological challenges for risks with low probabilities and high stakes

Introduction Large asteroid impacts are highly unlikely events. 1 Nonetheless, governments spend large sums on assessing the associated risks. It is the high stakes that make these otherwise rare events worth examining. Assessing a risk involves consideration of both the stakes involved and the likelihood of the hazard occurring. If a risk threatens the lives of a great many people, it is not only rational but morally imperative to examine the risk in some detail and to see what we can do to reduce it. This paper focuses on low-probability high-stakes risks. In Section 2, we show that the probability estimates in scientific analysis cannot be equated with the likelihood of these events occurring. Instead of the probability of the event occurring, scientific analysis gives the event's probability conditioned on the given argument being sound. Though this is the case in all probability estimates, we show how it becomes crucial when the estimated probabilities are smaller than a certain threshold. To proceed, we need to know something about the reliability of the argument. To do so, risk analysis commonly falls back on the distinction between model and parameter uncertainty. We argue that this dichotomy is not well suited for incorporating information about the reliability of the theories involved in the risk assessment. Furthermore, the distinction does not account for mistakes made unknowingly. In Section 3, we therefore propose a three-fold distinction between an argument's theory, its model and its calculations. While explaining this distinction in more detail, we illustrate it with historic examples of errors in each of the three areas. We indicate how specific risk assessment can make use of the proposed theory-model-calculation distinction in order to evaluate the reliability of the given argument and thus improve the reliability of their probability estimate for rare events. Recently, concerns have been raised that high-energy experiments in particle physics, such as the Relativistic Heavy Ion Collider (RHIC) at Brookhaven National Laboratory or the Large Hadron Collider (LHC) at CERN, Geneva, may threaten humanity. If these fears are justified, these experiments pose a risk to humanity that can be avoided by simply not turning on the experiment. In Section 4, we use the methods of this paper to address the current debate on the safety of experiments within particle physics. We evaluate current reports in the light of our findings and give suggestions for future research. The final section brings the debate back to the general issue of assessing low-probability risk. We stress that the findings in this paper are not to be interpreted as an argument for anti-intellectualism, but rather as arguments for making the noisy and fallible nature of scientific and technical research subject to intellectual reasoning, especially in situations where the probabilities are very low and the stakes are very high.
 Probability estimates Suppose you read a report which examines a potentially catastrophic risk and concludes that the probability of catastrophe is one in a billion. What probability should you assign to the catastrophe occurring? We argue that direct use of the report's estimate of one in a billion is naive. This is because the report's authors are not infallible and their argument might have a hidden flaw. What the report has told us is not the probability of the catastrophe occurring, but the probability of the catastrophe occurring, given that the included argument is sound. Even if the argument looks watertight, the chance that it contains a critical flaw may well be much larger than one in a billion. After all, in a sample of a billion apparently watertight arguments you are likely to see many that have hidden flaws. Our best estimate of the probability of catastrophe may thus end up noticeably higher than the report's estimate.  2  Let us use the following notation: X, the catastrophe occurs; A, the argument is sound; P(X), the probability of X and P(X|A), the probability of X given A. While we are actually interested in P(X), the report provides us only with an estimate of P(X|A), since it cannot fully take into account the possibility that it is in error.  3, 4  From the axioms of probability theory, we know that P(X) is related to P(X|A) by the following formula: P X P X A P A P X A P A ( ) ( | ) ( ) ( | ) ( ). ( ) = + ¬ ¬ 1 To use this formula to derive P(X), we would require estimates for the probability that the argument is sound, P(A), and the probability of the catastrophe occurring, given that the argument is unsound, P(X|¬A). We are highly unlikely to be able to acquire accurate values for these probabilities in practice, but we shall see that even crude estimates are enough to change the way we look at certain risk calculations. A special case, which occurs quite frequently, is for reports to claim that X is completely impossible. However, this just tells us that X is impossible, given that all our current beliefs are correct, that is P(X|A) = 0. By Equation (1) we can see that this is entirely consistent with P(X) > 0, as the argument may be flawed. Figure  1  is a simple graphical representation of our main point. The square on the left represents the space of probabilities as described in the scientific report, where the black area represents the catastrophe occurring and the white area represents not occurring. The normalized vertical axis denotes the probabilities for the event occurring and not occurring. This representation ignores the possibility of the argument being unsound. To accommodate this possibility, we can revise it in the form of the square on the right. The black and white areas have shrunk in proportion to the probability that the argument is sound and a new grey area represents the possibility that the argument is unsound. Now, the horizontal axis is also normalized and represents the probability that the argument is sound. Figure  1 . The left panel depicts a report's view on the probability of an event occurring. The black area represents the chance of the event occurring, the white area represents it not occurring. The right-hand panel is the more comprehensive picture, taking into account the possibility that the argument is flawed and that we thus face a grey area containing an unknown amount of risk. To continue our example, let us suppose that the argument made in the report looks very solid, and that our best estimate of the probability that it is flawed is one in a thousand, (P(¬A) = 10 −3 ). The other unknown term in Equation (1), P(X|¬A), is generally even more difficult to evaluate, but for the purposes of the current example, let us suppose that we think it highly unlikely that the event will occur even if the argument is not sound and treat this probability as one in a thousand as well. Equation (1) tells us that the probability of catastrophe would then be just over one in a million -an estimate which is a thousand times higher than that in the report itself. This reflects the fact that if the catastrophe were to actually occur, it is much more likely that this was because there was a flaw in the report's argument than that a one in a billion event took place. Flawed arguments are not rare. One way to estimate the frequency of major flaws in academic papers is to look at the proportions which are formally retracted after publication. While some retractions are due to misconduct, most are due to unintentional errors.  5  Using the MEDLINE database,  7 Cokol et al. (2007)  found a raw Figure  1 . The left panel depicts a report's view on the probability of an event occurring. The black area represents the chance of the event occurring, the white area represents it not occurring. The right-hand panel is the more comprehensive picture, taking into account the possibility that the argument is flawed and that we thus face a grey area containing an unknown amount of risk. retraction rate of 6.3 × 10 −5 , but used a statistical model to estimate that the retraction rate would actually be between 0.001 and 0.01 if all journals received the same level of scrutiny as those in the top tier. This would suggest that P(¬A) > 0.001 making our earlier estimate rather optimistic. We must also remember that an argument can easily be flawed without warranting retraction. Retraction is only called for when the underlying flaws are not trivial and are immediately noticeable by the academic community. The retraction rate for a field would thus provide a lower bound for the rate of serious flaws. Of course, we must also keep in mind the possibility that different branches of science may have different retraction rates and different error rates: the hard sciences may be less prone to error than the more applied sciences. Finally, we can have more confidence in an article, the longer it has been open to public scrutiny without a flaw being detected. It is important to note the particular connection between the present analysis and high-stakes low-probability risks. While our analysis could be applied to any risk, it is much more useful for those in this category. For it is only when P(X|A) is very low that the grey area has a relatively large role to play. If P(X|A) is moderately high, then the small contribution of the error term is of little significance in the overall probability estimate, perhaps making the difference between 10 and 10.001% rather than the difference between 0.001 and 0.002%. The stakes must also be very high to warrant this additional analysis of the risk, for the adjustment to the estimated probability will typically be very small in absolute terms. While an additional one in a million chance of a billion deaths certainly warrants further consideration, an additional one in a million chance of a house fire may not. One might object to our approach on the grounds that we have shown only that the uncertainty is greater than previously acknowledged, but not that the probability of the event is greater than estimated: the additional uncertainty could just as well decrease the probability of the event occurring. When applying our approach to arbitrary examples, this objection would succeed; however in this paper, we are specifically looking at cases where there is an extremely low value of P(X|A), so practically any value of P(X|¬A) will be higher and thus drive the combined probability estimate upwards. The situation is symmetric with regard to extremely high estimates of P(X|A), where increased uncertainty about the argument will reduce the probability estimate, the symmetry is broken only by our focus on arguments which claim that an event is very unlikely. Another possible objection is that since there is always a non-zero probability of the argument being flawed, the situation is hopeless: any new argument will be unable to remove the grey area completely. It is true that the grey area can never be completely removed; however, if a new argument (A 2 ) is independent of the previous argument (A 1 ), then the grey area will shrink, for P(¬A 1 , ¬A 2 ) < P(¬A 1 ). This can allow for significant progress. A small remaining grey area can be acceptable if P(X|¬A) P(¬A) is estimated to be sufficiently small in comparison to the stakes.
 Theories, models and calculations The most common way to assess the reliability of an argument is to distinguish between model and parameter uncertainty and assign reliabilities to these choices. While this distinction has certainly been of use in many practical cases, it is unnecessarily crude for the present purpose, failing to account for potential errors in the paper's calculations or a failure of the background theory. In order to account for all possible mistakes in the argument, we look separately at its theory, its model and its calculations. The calculations evaluate a concrete model representing the processes under consideration, for example the formation of black holes in a particle collision, the response of certain climate parameters (such as mean temperature or precipitation rate) to changes in greenhouse gas concentrations or the response of economies to changes in the oil price. These models are mostly derived from more general theories. In what follows, we do not restrict the term 'theory' to well-established and mathematically elaborate theories like electrodynamics, quantum chromodynamics or relativity theory. Rather, theories are understood to include theoretical background knowledge, such as specific research paradigms or the generally accepted research practice within a field (see Figure  2 ). An example is the efficient market hypothesis which underlies many models within economics, such as the Black-Scholes model. We consider adequate models or theories rather than correct ones. For example, we wish to allow that Newtonian mechanics is an adequate theory in many situations, while recognizing that in some cases it is clearly inadequate (such as for calculating the electron orbitals). We thus call a representation of some system adequate if it is able to predict the relevant system features at the required precision. For example, if climate modellers wish to determine the implications our greenhouse gas emissions will have on the well-being of future generations, their model/theory will not be adequate unless it tells them the changes in the local temperature and precipitation. In contrast, a model might only need to tell them changes in global temperature and precipitation to be adequate for answering less sensitive questions. On a theoretical level, much more could be said about this distinction between adequacy and correctness, but for the purposes of evaluating the reliability of risk assessment, the explanation above should suffice. With the following notation: T, the involved theories are adequate; M, the derived model is adequate; and C, the calculations are correct, we can represent P(A) as P(T, M, C). We can then benefit from the theory-model-calculation distinction by expressing this as the product of three separate probabilities, each of which should be easier to estimate than P(A). Firstly, from the laws of conditional probability, it follows that P(T, M, C) = P(T) P(M|T) P(C|M, T). This can be simplified as we can assume C to be independent of M and T, since the correctness of a calculation is independent of whether the theoretical and model assumptions underpinning it were adequate. Given this independence P(C|M, T) = P(C), the above equation can be simplified to: Substituting this back into Equation (1), we obtain a more tractable formula for P(X). P A P T P M T P C ( ) ( ) ( | ) ( ). ( ) = 2 We have already made a rough attempt at estimating P(A) from the paper retraction rates. Estimating P(T), P(M|T) and P(C) is more accurate and somewhat easier, though still of significant difficulty. While estimating the various terms in Equation (  2 ) must ultimately be done on a case-by-case basis, the following elucidation of what we mean by theory, model and calculation will shed some light on how to pursue such an analysis. By incorporating our threefold distinction, it is straightforward to apply findings on the reliability of theories from philosophy of science -based, for example, on probabilistic verification methods (e.g.  Reichenbach 1938)  or falsifications as in  Hempel (1950)  or  Popper (1959) . Often, however, the best we can do is to put some bounds upon them based on the historical record. We thus review typical sources of error in the following three areas.
 Calculation: analytic and numeric Estimating the correctness of the calculation independently from the adequacy of the model and the theory seems important whenever the mathematics involved is non-trivial. Most cases where we are able to provide more than purely heuristic and hand-waving risk assessments are of this sort. Consider climate models evaluating runaway climate change and risk estimates for the LHC or for asteroid impacts. When calculations accumulate, even trivial mathematical procedures become error-prone. A particular difficulty arises due to the division of labour in the sciences: commonly in modern scientific practice, various steps in a calculation are done by different individuals who may be in different working groups in different countries. The Mars Climate Observer spacecraft was lost in 1999 because a piece of control software from Lockheed Martin used imperial units instead of the metric units the interfacing NASA software expected (NASA 1999). Calculation errors are distressingly common. There are no reliable statistics on the calculation errors made in risk assessment or, even more broadly, within scientific papers. However, there is research on errors made in some very simple calculations performed in hospitals. Dosing errors give an approximate estimate of how often mathematical slips occur. Errors in drug charts occur at a rate of 1.2-31% across different studies  (Prot et al. 2005; Stubbs, Haw, and Taylor 2006; Walsh et al. 2008) , with a median of roughly 5% of administrations. Of these errors, 15-40% were dose errors, giving an overall dose error rate of about 1-2%. What does this mean for error rates in risk estimation? Since the stakes are high when it comes to dosing errors, this data represents a serious attempt to get the right answer in a life or death circumstance. It is likely that the people doing risk estimation are more reliable at arithmetic than health professionals and have more time for error correction, but it appears unlikely that they would be more reliable in many orders of magnitude. Hence, a chance of 10 −3 for a mistake per simple calculation does not seem unreasonable. A random sample of papers from Nature and the British Medical Journal found that roughly 11% of statistical results were flawed, largely due to rounding and transcription errors (García-Berthou and Alcaraz 2004). Calculation errors include more than just the 'simple' slips which we know from school, such as confusing units, forgetting a negative square root or incorrectly transcribing from the line above. Instead, many mistakes arise here due to numerical implementation of the analytic mathematical equations. Computer-based simulations and numerical analysis are rarely straightforward. The history of computers contains a large number of spectacular failures due to small mistakes in hardware or software. The 4 June 1996 explosion of an Ariane 5 rocket was due to a leftover piece of code triggering a cascade of failures  (ESA 1996) . Audits of spreadsheets in real world use find error rates on the order of 88%  (Panko 1998) . The 1993 Intel Pentium floating point error affected 3-5 million processors, reducing their numeric reliability and hence our confidence in anything calculated with them  (Nicely 2008) . Programming errors can remain dormant for a long time even in apparently correct code, only to emerge under extreme conditions. An elementary and widely used binary search algorithm included in the standard libraries for Java was found after nine years to contain a bug that emerges only when searching very large lists  (Bloch 2006) . A mistake in data processing led to the retraction of five high-profile protein structure papers as the handedness of the molecules had become inverted  (Miller 2006) . In cases where computational methods are used in modelling, many mistakes cannot be avoided. Discrete approximations of the often continuous model equations are used, and in some cases we know that the discrete version is not a good proxy for the continuous model  (Morawetz and Walke 2003) . Moreover, numerical evaluations are often done on a discrete computational grid, with the values inside the meshes being approximated from the values computed at the grid points. Though we know that certain extrapolation schemes are more reliable in some cases than others, we are often unable to exclude the possibility of error, or to even quantify it.
 Ways of modelling and theorizing Our distinction between model and theory follows the typical use of the terms within mathematical sciences like physics or economics. Whereas theories are associated with broad applicability and higher confidence in the correctness of their description, models are closer to the phenomena. For example, when estimating the probability of a particular asteroid colliding with the Earth, one would use either Newtonian mechanics or general relativity as a theory for describing the role of gravity. One could then use this theory in conjunction with observations of the bodies' positions, velocities and masses to construct a model, and finally, one could perform a series of calculations based on this model to estimate the probability of impact. As this shows, the errors that can be introduced in settling for a specific model include and surpass those which are sometimes referred to as parameter uncertainty. As well as questions of the individual parameters (positions, velocities and masses), there are important questions of detail (Can we neglect the inner structure of the involved bodies?), and breadth (Can we focus on the Earth and asteroid only, or do we have to model other planets, or the Sun?).  7  As can be seen from this example, one way to distinguish theories from models is that theories are too general to be applied directly to the problem. For any given theory, there are many ways to apply it to the problem and these ways give rise to different models. Philosophers of science will note that our theory-model distinction is in accordance with the non-uniform notion used by  Giere (1999) ,  Morrison (1998) ,  Cartwright (1999) , and others, but differs from that of the semantic interpretation of theories  (Suppes 1957) . We should also note that it is quite possible for an argument to involve several theories or several models. This complicates the analysis and typically provides additional ways for the argument to be flawed.  8  For example, in estimating the risk of black hole formation at the LHC, we not only require quantum chromodynamics (the theory on which the LHC is built to test), but also relativity and Hawking's theory of black hole radiation. In addition to their other roles, modelling assumptions also have to explain how to glue such different theories together  (Hillerbrand and Ghil 2008) . In risk assessment, the systems involved are most often not as well understood as asteroid impacts. Often, various models exist simultaneously -all known to be incomplete or incorrect in some way, but difficult to improve upon. 9 Particularly in these cases, having an expected or desired outcome in mind while setting up a model makes one vulnerable to expectation bias: the tendency to reach the desired answer rather than the correct one. This bias has affected many of science's great names  (Jeng 2006) , and in the case of risk assessment, the desire for a 'positive' outcome (safety in the case of the advocate or danger in the case of the protestor) seems a likely cause of bias in modelling.
 Historical examples of model and theory failure A dramatic example of a model failure was the Castle Bravo nuclear test on 1 March 1954. The device achieved 15 megatons of yield instead of the predicted four to eight megatons. Fallout affected parts of the Marshal Islands and irradiated a Japanese fishing boat so badly that one fisherman died, causing an international incident (Nuclear Weapon Archive 2006). Though the designers at Los Alamos National Laboratories understood the involved theory of alpha decay, their model of the reactions involved in the explosion was too narrow, for it neglected the decay of one of the involved particles (lithium-7), which turned out to contribute the bulk of the explosion's energy. The Castle Bravo test is also notable for being an example of model failure in a very serious experiment conducted in the hard sciences and with known high stakes. The history of science contains numerous examples of how generally accepted theories have been overturned by new evidence or understanding, as well as a plethora of minor theories that persisted for a surprising length of time before being disproven. Classic examples for the former include the Ptolemaic system, phlogiston theory and caloric theory; an example for the latter is human chromosome number, which was systematically miscounted as 48 (rather than 46) and this error persisted for more than 30 years  (Gartler 2006 ). As a final example, consider Lord Kelvin's estimates of the age of the Earth  (Burchfield 1975) . They were based on information about the earth's temperature and heat conduction, estimating an age of the Earth between 20 and 40 million years. These estimates did not take into account radioactive heating, for radioactive decay was unknown at the time. Once it was shown to generate additional heat the models were quickly updated. While neglecting radioactivity today would count as a model failure, in Lord Kelvin's day it represented a largely unsuspected weakness in the physical understanding of the Earth and thus amounted to theory failure. This example makes it clear that the probabilities for the adequacy of model and theory are not independent of each other, and thus in the most general case we cannot further decompose Equation (2).
 Applying our analysis to the risks from particle physics research Particle physics is the study of the elementary constituents of matter and radiation, and the interactions between them. A major experimental method in particle physics involves the use of particle accelerators, such as the RHIC and LHC, to bring beams of particles near the speed of light and then collide them together. This focuses a large amount of energy in a very small region and breaks the particles down into their components, which are then detected. As particle accelerators have become larger, the energy densities achieved have become more extreme, prompting some concern about their safety. These safety concerns have focused on three possibilities: the formation of 'true vacuum', the transformation of the Earth into 'strange matter', and the destruction of the Earth through the creation of a black hole.
 True vacuum and strange matter formation The type of vacuum that exists in our universe might not be the lowest possible vacuum energy state. In this case, the vacuum could decay to the lowest energy state, either spontaneously, or if triggered by a sufficient disturbance. This would produce a bubble of 'true vacuum' expanding outwards at the speed of light, converting the universe into different state apparently inhospitable for any kind of life  (Turner and Wilczek 1982) . Our ordinary matter is composed of electrons and two types of quarks: up quarks and down quarks. Strange matter also contains a third type of quark: the 'strange' quark. It has been hypothesized that strange matter might be more stable than normal matter, and able to convert atomic nuclei into more strange matter  (Witten 1984) . It has also been hypothesized that particle accelerators could produce small negatively charged clumps of strange matter, known as strangelets. If both these hypotheses were correct and the strangelet also had a high enough chance of interacting with normal matter, it would grow inside the Earth, attracting nuclei at an ever higher rate until the entire planet was converted to strange matter -destroying all life in the process. Unfortunately, strange matter is complex and little understood, giving models with widely divergent predictions about its stability, charge and other properties  (Jaffe et al. 2000) . One way of bounding the risk from these sources is the cosmic ray argument: the same kind of high-energy particle collisions occur all the time in Earth's atmosphere, on the surface of the Moon and elsewhere in the universe. The fact that the Moon or observable stars have not been destroyed despite a vast number of past collisions (many at much higher energies than can be achieved in human experiments) suggests that the threat is negligible. This argument was first used against the possibility of vacuum decay  (Hut and Rees 1983)  but is quite general. An influential analysis of the risk from strange matter was carried out in  Dar, De Rujula, and Heinz (1999)  and formed a key part of the safety report for the RHIC. This analysis took into account the issue that any dangerous remnants from cosmic rays striking matter at rest would be moving at high relative velocity (and hence much less likely to interact) while head-on collisions in accelerators could produce remnants moving at much slower speeds. They used the rate of collisions of cosmic rays in free space to estimate strangelet production. These strangelets would then be slowed by galactic magnetic fields and eventually be absorbed during star formation. When combined with estimates of the supernova rate, this can be used to bound the probability of producing a dangerous strangelet in a particle accelerator. The resulting probability estimate was < 2 × 10 −9 per year of RHIC operation.  10  While using empirical bounds and experimentally tested physics reduces the probability of a theory error, the paper needs around 30 steps to reach its conclusion. For example, even if there was just a 10 −4 chance of a calculation or modelling error per step this would give a total P(¬A) ≈ 0.3%. This would easily overshadow the risk estimate. Indeed, even if just one step had a 10 −4 chance of error, this would overshadow the estimate. A subtle complication in the cosmic ray argument was noted in  Tegmark and Bostrom (2005) . The Earth's survival so far is not sufficient as evidence for safety, since we do not know if we live in a universe with 'safe' natural laws or a universe where planetary implosions or vacuum decay do occur, but we have just been exceedingly lucky so far. While this latter possibility might sound very unlikely, all observers in such a universe would find themselves to be in the rare cases where their planets and stars had survived, and would thus have much the same evidence as we do. Tegmark and Bostrom had thus found that in ignoring these anthropic effects, the previous model had been overly narrow. They corrected for this anthropic bias and, using analysis from  Jaffe et al. (2000) , concluded that the risk from accelerators was less than 10 −12 per year. This is an example of a demonstrated flaw in an important physics risk argument (one that was pivotal in the safety assessment of the RHIC). Moreover, it is significant that the RHIC had been running for five years on the strength of a flawed safety report, before Tegmark and Bostrom noticed and fixed this gap in the argument. Although this flaw was corrected immediately after being found, we should also note that the correction is dependent on both anthropic reasoning and on a complex model of the planetary formation rate  (Lineweaver, Fenner, and Gibson 2004) . If either of these or the basic Brookhaven analysis is flawed, the risk estimate is flawed.
 Black hole formation The LHC experiment at CERN was designed to explore the validity and limitations of the standard model of particle physics by colliding beams of high-energy protons. This will be the most energetic particle collision experiment ever done, which has made it the focus of a recent flurry of concerns. Due to the perceived strength of the previous arguments on vacuum decay and strangelet production, most of the concern about the LHC has focused on black hole production. None of the theory papers we have found appears to have considered the black holes to be a safety hazard, mainly because they all presuppose that any black holes would immediately evaporate due to Hawking radiation. However, it was suggested by  Dimopoulos and Landsberg (2001)  that if black holes form, particle accelerators could be used to test the theory of Hawking radiation. Thus, critics also began questioning whether we could simply assume that black holes would evaporate harmlessly. A new risk analysis of LHC black hole production (Giddings and Mangano 2008) provides a good example of how risks can be more effectively bounded through multiple sub-arguments. While never attempting to give a probability of disaster (rather concluding 'there is no risk of any significance whatsoever from such black holes'), it uses a multiple bounds argument. It first shows that rapid black hole decay is a robust consequence of several different physical theories (A 1 ). Second, it discusses the likely incompatibility between non-evaporating black holes and mechanisms for neutralizing black holes: in order for cosmic ray-produced stable black holes to be innocuous but accelerator-produced black holes to be dangerous, they have to be able to shed excess charge rapidly (A 2 ). Our current understanding of physics suggests both that black holes decay and that even if they did not, they would be unable to discharge themselves. Only if this understanding is flawed will the next section come into play. The third part, which is the bulk of the paper, models how multidimensional and ordinary black holes would interact with matter. This leads to the conclusion that if the size scale of multidimensional gravity is smaller than about 20 nm, then the time required for the black hole to consume the Earth would be larger than the natural lifetime of the planet. For scenarios where rapid Earth accretion is possible, the accretion time inside white dwarves and neutron stars would also be very short, yet production and capture of black holes from impinging cosmic rays would be so high that the lifespan of the stars would be far shorter than the observed lifespan (and would contradict white dwarf cooling rates) (A 3 ). The force of the total argument (A 1 , A 2 , A 3 ) is significantly stronger than any of its parts. Essentially, the paper acts as three sequential arguments, each partly filling in the grey area (see Figure  1 ) left by the previous. If the theories surrounding black hole decay fail, the argument about discharge comes into play; and if against all expectation black holes are stable and neutral, the third argument shows that astrophysics constrains them to a low accretion rate.
 Implications for the safety of the LHC What are the implications of our analysis for the safety assessment of the LHC? Firstly, let us consider the stakes in question. If one of the proposed disasters were to occur, it would mean the destruction of the Earth. This would involve the complete destruction of the environment, 6.5 billion human deaths and the loss of all future generations. It is worth noting that this loss of all future generations (and with it, all of humanity's potential) may well be the greatest of the three, but a comprehensive assessment of these stakes is outside the scope of this paper. For the present purposes, it suffices to observe that the destruction of the Earth is at least as bad as 6.5 billion human deaths. There is some controversy as to how one should combine probabilities and stakes into an overall assessment of a risk. Some hold that the simple approach of expected utility is the best, while others hold some form of risk aversion. However, we can sidestep this dispute by noting that in either case, the risk of some harm is at least as bad as the expected loss. Thus, a risk with probability p of causing a loss at least as bad as 6.5 billion deaths is at least as bad as a certain 6.5 × 10 9 p deaths. Now let us turn to the best estimate we can make of the probability of one of the above disasters occurring during the operation of the LHC. While the arguments for the safety of the LHC are commendable for their thoroughness, they are not infallible. Although the report considered several possible physical theories, it is eminently possible that these are all inadequate representations of the underlying physical reality. It is also possible that the models of processes in the LHC or the astronomical processes appealed to in the cosmic ray argument are flawed in an important way. Finally, it is possible that there is a calculation error in the report. In Equation (1), P(X) is the sum of two terms. The second of these represents the additional probability of disaster due to the argument being unsound. It is the product of the probability of argument failure and the probability of disaster given such a failure. Both terms are very difficult to estimate, but we can gain insight by showing the ranges they would have to lie within, for the risk presented by the LHC to be acceptable. If we let l denote the acceptable limit of expected deaths from the operation of the LHC, we get: 6.5 × 10 9 P(X) ≤ l. Since P(X) is at least as great as its second term, we obtain: This inequality puts a severe bound on the acceptable values for these probabilities. Since it is much easier to grasp this with an example, we shall provide some numbers for the purposes of illustration. Suppose, for example, that if the limits were set at 1000 expected deaths, then P(X|¬A) P(¬A) would have to be below 1.5 × 10 −7 for the risk to be worth bearing. This requires very low values for these probabilities. We have seen that for many arguments, P(¬A) is above 10 −3 . We have also seen that the argument for the safety of the RHIC turned out to have a significant flaw, which was unnoticed by the experts at the time. It would thus be very bold to suppose that the argument for the safety of the LHC was much lower than 10 −3 , but for the sake of argument, let us grant that it is as low as 10 −4 -that out of a sample of 10,000 independent arguments of similar apparent merit, only one would have any serious error. Even if the value of P(¬A) were as low as 10 −4 , P(X|¬A) would have to be below 0.15% for the risk to be worth taking. P(X|¬A) is the probability of disaster, given that the arguments of the safety report are flawed and is the most difficult component of Equation (  1 ) to estimate. Indeed, few would dispute that we really have very little idea of what value to put on P(X|¬A). It would thus seem overly bold to set this below 0.15% without some substantive argument. Perhaps such an argument could be provided, but until it is, such a low value for P(X|¬A) seems unwarranted. We stress that the above combination of numbers was purely for illustrative purposes, but we cannot find any plausible combination of the three numbers which meets the bound and which would not require significant argument to explain either the levels of confidence or the disregard for expected deaths. We would also like to stress that we are open to the possibility that additional supporting arguments and independent verification of the models and calculations could significantly reduce the current chance of a flaw in the argument. However, our analysis implies that the current safety report should not be the final word in the safety assessment of the LHC. To proceed with the LHC on the arguments of the most recent safety report alone, we would require further work on estimating P(¬A), P(X|¬A), the acceptable expected death toll, and the value of future generations and other life on Earth. Such work would require expertise beyond theoretical physics, and an interdisciplinary group would be essential. If the stakes were lower, then it might make sense for pragmatic concerns to sweep aside this extra level of risk analysis, but the stakes are astronomically large, and so further analysis is critical. Even if the LHC goes ahead without any further analysis, as is very likely, these lessons must be applied to the assessment of other high-stakes low-probability risks.
 Conclusions When estimating threat probabilities, it is not enough to make conservative estimates (using the most extreme values or model assumptions compatible with known data). Rather, we need robust estimates that can handle theory, model and calculation errors. The need for this becomes considerably more pronounced for low-probability high-stake events, though we do not say that low probabilities cannot be treated systematically. Some people have raised the concern that our argument might be too powerful, for it is impossible to disprove the risk of even something as trivial as dropping a pencil, P X A P A l ( | ) ( ) . . ( ) ¬ ¬ ≤ × − 1 5 10 3 then our argument might amount to prohibiting everything. It is true that we cannot completely rule out any probability that apparently inconsequential actions might have disastrous effects, but there are a number of reasons why we do not need to worry about universal prohibition. A major reason is that for events like the dropping of a pencil which have no plausible mechanism for destroying the world, it seems just as likely that the world would be destroyed by not dropping the pencil. The expected losses would thus balance out. It is also worth noting that our argument is simply an appeal to a weak form of decision theory to address an unusual concern: for our method to lead to incorrect conclusions, it would require a flaw in decision theory itself, which would be very big news. It will have occurred to some readers that our argument is fully applicable to this very paper: there is a chance that we have made an error in our own arguments. We entirely agree, but note that this possibility does not change our conclusions very much. Suppose, very pessimistically, that there is a 90% chance that our argument is sufficiently flawed that the correct approach is to take safety reports' probability estimates at face value. Even then, our argument would make a large difference to how we treat such values. Recall the example from Section 2, where a report concludes a probability of 10 −9 and we revise this to 10 −6 . If there is even a 10% chance that we are correct in doing so, then the overall probability estimate would be revised to 0.9 × 10 −9 + 0.1 × 10 −6 ≈ 10 −7 , which is still a very significant change from the report's own estimate. In short, even serious doubt about our methods should not move one's probability estimates more than an order of magnitude away from those our method produces. More modest doubts would have negligible effect. The basic message of this paper is that any scientific risk assessment is only able to give us the probability of a hazard occurring conditioned on the correctness of its main argument. The need to evaluate the reliability of the given argument in order to adequately address the risk was shown to be of particular relevance in low-probability high-stake events. We drew a three-fold distinction between theory, model and calculation, and showed how this can be more useful than the common dichotomy in risk assessment between model and parameter uncertainties. By providing historic examples for errors in the three fields, we clarified the three-fold distinction and showed where flaws in a risk assessment might occur. Our analysis was applied to the recent assessment of risks that might arise from experiments within particle physics. To conclude this paper, we now provide some very general remarks on how to avoid argument flaws when assessing risks with high stakes. Firstly, the testability of predictions can help discern flawed arguments. If a risk estimate produces a probability distribution for smaller, more common disasters, this can be used to judge whether the observed incidences are compatible with the theory. Secondly, reproducibility appears to be the most effective way of removing many of these errors. By having other people replicate the results of calculations independently, our confidence in them can be dramatically increased. By having other theories and models independently predict the same risk probability, our confidence in them can again be increased, as even if one of the arguments is wrong the others will remain. Finally, we can reduce the possibility of unconscious bias in risk assessment through the simple expedient of splitting the assessment into a 'blue' team of experts attempting to make an objective risk assessment and a 'red' team of devil's advocates attempting to demonstrate a risk, followed by repeated turns of mutual criticism and updates of the models and estimates  (Calogero 2000) . Application of such methods could in many cases reduce the probability of error by several orders of magnitude. Figure 2 . 2 Figure 2. Ways in which risk assessments can be flawed.
 Figure 2 . 2 Figure 2. Ways in which risk assessments can be flawed.
			 *Corresponding author. Email: toby.ord@philosophy.ox.ac.uk

Abstract: Some risks have extremely high stakes. For example, a worldwide pandemic or asteroid impact could potentially kill more than a billion people. Comfortingly, scientific calcultions often put very low probabilities on the occurrence of such catastrophes. In this paper, we argue that there are important new methodological problems which arise when assessing global catastrophic risks and we focus on a problem regarding probability estimation. When an expert provides a calculation of the probability of an outcome, they are really providing the probability of the outcome occurring, given that their argument is watertight. However, their argument may fail for a number of reasons, such as a flaw in the underlying theory, a flaw in the modelling of the problem or a mistake in the calculations. If the probability estimate given by an argument is dwarfed by the chance that the argument itself is flawed, then the estimate is suspect. We develop this idea formally, explaining how it differs from the related distinction between model and parameter uncertainty. Using the risk estimates from the Large Hadron Collider as a test case, we show how serious the problem can be when it comes to catastrophic risks and how best to address it.