Gillian K Hadfield
Dylan Hadfield-Menell

Pervasive Spurious Normativity

Introduction One of the attributes that differentiates human from other animal societies is the presence of norms and in particular norms over behaviors that do not appear to have a material impact on well-being apart from the fact that people will punish the failure to adhere to a norm. In all known human societies there is a rich normative landscape, which attaches normative valence to actions including many that have no immediate material implications for those who are expected to help enforce them through social sanctions. Most societies for example have rules about what it is appropriate to eat when, what tone of voice to use in what settings, how close to stand to someone else, how to behave when waiting with others to get into a venue or access to a resource, what information should be shared with whom and when, who can participate in particular trades or economic activities and so on. In many cases, particular norms are arbitrary, even though violation of them is treated as worthy of punishment by others. We call this spurious normativity.  Hadfield [1999] , for example, reviews the anthropological literature on the sexual division of labor across pre-industrial societies. While almost all societies categorize work as either women's work or men's work, the particular classifications vary substantially cross-culturally: what is women's work in one society is men's work in another. The classification is arbitrary.  1  Fessler and Navarrete  [2003]  call the process by which patterns of behavior are imbued with moral sentiments that motivate sanctioning of violations of the pattern normative moralization. They use as an example the normative moralization of handedness. Most people are right-handed but particularly in societies with few specialized tools, whether someone is right-or left-handed often has no material consequences for others. Nonetheless, many cultures treat using one's right hand as a morally approved category-denoting purity or politeness-and one's left hand as cause for opprobrium-revealing weakness or evil. Fessler  [1999]  hypothesizes that, driven forward by the emotions of shame and pride that are triggered by violation or compliance with cooperative norms, culture extends the set of actions that are subject to normative moralization as a way of extending the set of actions that can be used as information about cooperation beyond those actions that directly involve cooperation. A norm that says "it is wrong to fail to look where you are going" generates direct cooperative benefits, helping people to avoid crashing into one another. A norm that says "it is wrong for a man to walk down the street wearing shorts" does not generate cooperative benefitsunless people in this society treat conformity to this norm as informative about a person's likelihood of behaving in conformity with norms that do generate cooperative benefits. As Fessler  [1999]  (who identifies the the "watch where you are going" and "men don't wear shorts out in public" norms in an Indonesian village) puts this: Ego's inclusion in cooperative activity is dependent on her ability to meet Others' expectations, and those expectations are, in turn related to shared standards for behaviors which are relevant to cooperation. As a consequence, the significant adaptive advantage offered by participation in cooperative activities generated selective pressure for an increase in the attention paid to these standards. Late [evolving] second order emotions [emotions that arise in response to others' first order emotions] were the vehicle through which this increase in attention was achieved. Moreover, because late second order emotions entail a sensitivity to the reactions of all individuals, Ego must be concerned with her performance vis--vis shared standards when interacting with any other member of her group. It is only a small step from this situation to one in which the shared standards with which Ego is concerned are not limited to the question of cooperative activity-once Ego is concerned with how all Others evaluate her, it is not difficult for shared standards governing other types of behavior to become salient as well. This is because an Other may extrapolate from situations that do not involve cooperation to those that do-an Other may think "if that individual does not follow shared standards in this context, how can I be confident that he will do so if I invite her to engage in cooperative activity?"(pre-pub p. 34, emphasis added) Fessler's account focuses on the evolution of emotions in response to norm violation (in particular shame) to motivate voluntary conformance with norms. But as  Boyd and Richerson [1992]  and many others have emphasized, and Fessler's own account of shame as a second order response to Ego's actual or imagined experience of hostility or criticism from Other triggered by norm violation implies, effective third-party punishment plays a significant role in supporting norm compliance. Indeed,  Mathew et al. [2012]  argue that even small-scale cooperation among kin and close associates may require third-party punishment to achieve evolutionary stability. In this paper we analyze mathematically the potential benefits of extending normative moralization to behaviors that are from a material perspective irrelevant or at least of small consequence to most people, that is, the value of pervasive and spurious normativity. We ask: does a community generate higher payoffs for participants if it punishes violation of apparently meaningless rules or if it focuses more narrowly only on norms that are functional in the sense of generating direct benefits? Intuitively, one might expect that only functional norms that govern behaviors that matter for payoffs would emerge and stabilize in equilibrium: punishment is costly and why would a society expend punishment resources on ensuring conformity with rules that have no impact on material well-being? Most analyses of norms in the law and economics literature assume that norms coordinate outcomes that improve welfare. Sugden  [1986] ,  McAdams [2005]  and  Myerson [2004]  propose, for example, that property rules emerge because they solve the coordination problems that arise in costly contests over resources (Hawk-Dove games). We consider the impact of extending normativity to apparently arbitrary actions on the prospect for sustaining welfareimproving behavior in an equilibrium social order based on collective third-party punishment. We demonstrate that communities that extend normativity in this way can generate higher value for participants than those that restrict the range of normativity. The intuition of our result is as follows. Communities with pervasive spurious normativity provide agents with plentiful and cheap opportunities to observe punishment behavior by others. An individual's willingness to cooperate in a community-which requires foregoing safe non-cooperative options and exposing oneself to the risk of being exploiteddepends on that individual's beliefs about the likelihood that the community effectively punishes conduct that harms the individual. If you are going to risk exposing yourself to harm, you want to know if your community contains enough people who will punish the perpetrator to give you confidence that harm is reasonably deterred. Assume you are a newcomer to a community or that the community was recently handed a new set of norms. (Think here of rule-of-law building efforts in developing countries.) Assume that the community is in an equilibrium in terms of punishment behavior but you do not know the likelihood of effective punishment of norm violations. Assume also that the only way to learn about the likelihood of punishment is to observe punishment behavior. You can learn this information more cheaply if you are given abundant opportunities to observe what happens when there are violations if the violations that you have to expose yourself to don't really matter very much. You don't really care whether men walk down the street in shorts but by taking a walk yourself you can see how others react to shorts-wearing men and thus gain information about how they would react to violations you do care about-careless driving for example. Thus, if you could participate solely as an observer except when your own interests were directly at stake, you would prefer to live in a world with pervasive even if spurious normativity-abundant opportunities to observe reactions to norm violations-than one that was narrowly focused on punishing just the stuff you care about. This will still be true even if you are required to participate in the community-complying with and punishing spurious norms-so long as those costs, which increase with the pervasiveness of norms, are not too great. Our results have implications for the evolution and microfoundations of law.  Hadfield and Weingast [2012]  present a model that derives characteristic features of law-such as generality, stability, uniqueness and universality-as attributes necessary to support an equilibrium in which behavior is patterned on the classifications of behavior articulated by a centralized institution. Enforcement is assumed to come exclusively from decentralized collective punishment of conduct classified as punishable by the centralized institution; there is no centralized enforcement apparatus such as the state. This theory of the microfoundations of law proposes, contrary to most economic and positive political theories of law, which define law as a set of rules enforced by a centralized enforcement apparatus (see  Hadfield and Weingast [2014] ), that law is an innovation in the mechanism used to coordinate the same enforcement mechanism that supports other normative social orders-decentralized collective punishment. This has important implications for our understanding of how law developed and how it can be built in environments where it is currently lacking. The equilibrium legal order in Hadfield and Weingast  [2012]  is supported by a particular specification of beliefs. Their model posits that an agent (call the agent Ego) treats other agents' (Others') failure to punish behavior classified by the institution as punishable, including behavior that has no impact on Ego's (or perhaps any agent's) payoff, as informative of the likelihood that Others will also fail to punish wrongful behavior that does have an impact on Ego's payoff. This belief structure creates an incentive for individuals to participate in collective punishment, potentially mitigating the free-rider problem in collective punishment. Ego has no incentive to participate in collective punishment (the model assumes standard preferences) except to maintain an equilibrium in which behavior that reduces Ego's longterm payoff is deterred by the threat of collective punishment. Effectively, an agent's participation in punishment is treated as information about that agent's continued assessment that an equilibrium in which collective punishment is coordinated by the classifications articulated by the central institution will benefit that agent and hence that the agent will be willing to incur costs to support the equilibrium. The classification institution in this model is serving as what  Hadfield and Weingast [2012]  call an authoritative steward of a very simple binary partition of behaviors into those that are punishable and those that are not. This can be interpreted as the construction of a sparse labeling system: lawful and not lawful.  2  We can interpret the beliefs underlying legal order as beliefs about whether people are punishers or not of behavior to which the constructed (that is, not natural) label "unlawful" is attached. Our model here provides a basis for understanding how a simple binary classification scheme that is comprehensivecovering a wide variety of conduct-can emerge in a setting in which equilibrium depends on 1) voluntary participation in punishment and 2) a belief structure in which punishment of an action labeled punishable is considered informative about the likelihood of punishment of other actions labeled punishable, even when the assignment of the label is potentially arbitrary. We suggest that understanding how such a classification scheme and belief structure can emerge is critical for understanding the emergence of law. The strategy of our paper is as follows. We first give an overview of the model and basic notation in Section 2, together with some technical background from the analysis of multi-armed bandit games and partially observed Markov decision processes. To build intuition, we then present in Section 3 analytical results for the limiting case in which Ego bears no cost of complying with spurious norms or punishing their violation. Because the games we analyze quickly becomes analytically complex but relatively easy to compute once we introduce a positive cost of complying with norms and punishing their violation, we turn to computational results in Section 4. Section 5 relates our results to conjectures about the likely growth and stability of communities in which norms are more or less pervasive and in which a legal institution that reduces ambiguity about norm violations and increases the informativeness of punishing behavior (by linking disparate rules into a code such that punishment of one rule is informative about the likelihood of punishment of another) exists.
 Overview of Model The basic idea of the model is as follows. Consider an infinitely repeated game setting in which an agent Ego is faced with the choice in each period of participating or sitting out. If choosing to sit out, Ego receives a payoff normalized to 0. If Ego chooses to participate, she plays a randomly selected game g with two randomly selected agents drawn from a population (Others). We model these games in reduced form. In each game, one of the Others is randomly selected and presented with an opportunity to choose between two actions, one which is classified by a classification institution L as "rule violation" and another which is classified as "not rule violation." If Other chooses "rule violation" the remaining Other and Ego each independently choose either to punish or not punish. Rule violations are deterred by collective punishment, that is, punishment that requires more than one agent to punish. For example, in Hadfield and Weingast  [2012]  two buyers and a seller engage in repeated contract and performance games. Actions for the seller are drawn from a set of possible contract performances, some of which are classified as breach and others which are not breach. A decision by a buyer not to purchase from a third-party seller in one period constitutes punishment, specifically a boycott. Breach is deterred when the seller expects both buyers to boycott in response to breach. Games are distinguished by the rule that may be violated. For example, there could be a game in which rule "watch where you are going" may be violated and one in which "men should not wear shorts in the street" may be violated. We assume, and Ego believes, that the community of Others is playing an equilibrium in the super game that consists of the sequence of repeated games. Others are of two types, t: punishers (t = 1), who punish anyone who chooses an action classified by L as a rule violation, and non-punishers (t = 0), who never punish anyone. Let θ be the true proportion of punishers in the equilibrium. We assume that an Other's type is observable by the other participants in any particular game, that is, only in the context of the opportunity for rule violation. We focus on sub-game perfect equilibria in which the knowledge that two punishers are present deters rule violation. 3 (That is, on the off-equilibrium path where a violation does occur in the presence of an Other of type t, punishment is carried out that imposes costs on the violator that exceed the present value of benefits from violation.) We do not model how this equilibrium is generated or supported but we observe that the equilibrium is not destabilized by the presence of non-punishers. We assume, however, that Ego plays as a punisher, bearing an expected cost c in each round. c can be thought of as the cost to Ego of signaling that she is a punisher. For simplicity we assume that Ego is never presented with an opportunity for rule violation.  4  Ego's participation in the game is assumed to be on the margin, with no impact on the equilibrium played by the Others. Ego is able to observe rule violations, the types of Others and punishments in games in which she participates.
 Formal Model Specification Before providing a formalization of our model we provide a brief overview of the theory of Markov decision processes and multi-armed bandits.
 Technical Background and Notation We define a Markov decision process (MDP), M , is a tuple: M = S, A, P, R, δ 5 . S is a set of states. A is a set of actions. P : S × A × S → [0, 1] is a function that assigns probability to state transitions for each state-action pair. If Ego is in state, s, and selects action a the probability of transitioning to s is given by P (s, a, s ). R is a (bounded) reward function that maps states, to an interval of R, w.l.o.g., R : S → [0, 1]. δ ∈ [0, 1) is a discount factor that expresses Ego's preference for current versus future rewards. A solution to M is a policy, π, that maps states to actions, π : S → A. The value of a state, s, under π is the sum of expected discounted rewards received by starting in s and selecting actions according to π: V π (s) = E ∞ t=0 δ t R(s t )|s 0 = s, π . The optimal policy, π * , maximizes this value and we will write V = V π * for the optimal value function. Standard results show that a unique optimal value function exists  [Puterman, 1994] . In a partially observed Markov decision process (POMDP), we additionally define a distribution over observations (O) for a each state. A policy now maps a history of observations to an action, as the agent does not know the true state of the world. A POMDP can always be converted into a (continuous state) MDP where each state is a distribution over states of the world and the transition distribution is defined by Bayesian inference. An interesting class of POMDPs are multi-armed bandits (MAB). In a multi-armed bandit, an agent is given access to several distributions. At each time step, that agent must selects a distribution to sample from, and receives a reward that is equal to the value of the sample. A multi-armed bandit provides an analytically and computationally tractable model of exploration-exploitation tradeoffs that occur with practical agents. In particular, success in this class of problems requires explicit reasoning about the impact of information on future decision making quality. Recent applicability to online banner advertising has led to rapid progress in both theoretical and computational methods for MABs  [Auer et al., 2002] . The optimal full information policy (which knows the distributions of the arms) will always select the arm with the highest mean. A key result from Lai and Robbins  [1985]  lower bounds the number of times an optimal (partial information) policy selects a suboptimal arm in expectation. This result holds for the class of consistent policies: policies where the probability that the optimal arm is chosen at time t approaches 1 as t → ∞. We additionally require a smoothness constraint on the MAB distributions. Proposition 1.  [Lai and Robbins, 1985]  Let Θ be a class of MAB arms (i.e., a class of distributions) with parameter θ. Let µ(θ) be the corresponding mean. If ∀θ ∈ Θ, ∀δ > 0, ∃θ = θ such that µ(θ) ≤ µ(θ ) ≤ µ(θ) + δ then, for any consistent policy the expected number of times a suboptimal arm is selected in the first n rounds is Ω(log n) 6 .
 Model Description We define our super game as a tuple: G, T θ , Π, U, δ, c where G is a distribution over games and T θ is a distribution over punishment types t in the population of Others. We will abuse notation somewhat as use T and G to refer to the support of the corresponding distributions where the meaning is obvious. Π is Ego's prior distribution over the parameters of T θ , and U : G × T θ → R is a mapping from types and games to immediate payoffs for Ego. The understanding is that this mapping represents the results of the Others playing their role in the equilibrium. δ is Ego's discount parameter for future rewards. c expresses a participation cost. This can be understood as the expected cost of to Ego of signaling to an Other that she is a punisher. Ego begins in period 1 with perfect knowledge of how actions are classified by L, all payoffs and the distribution of games. Ego does not know the distribution of types of Others but holds a prior which we will specify shortly. Ego updates her beliefs about the distribution of types using Bayes' rule. The super game is defined as follows: An initial game g 0 ∼ G is drawn. Then, for each period j: 1. Ego chooses whether to participate or not. If she opts out, then she collects 0 payoff and the next round starts. 2. If Ego opts in, she incurs the cost of signaling that she is a punisher c and a type, t j ∼ T θ is for the remaining Other is drawn. If a non-punisher is drawn, a rule violation occurs; if a punisher is drawn, no rule violation occurs. 3. The agent observes whether a punisher is present and whether a rule violation occurs and collects payoff U (g j , t j ). 4. A game g j+1 ∼ G is drawn for the next round. We assume that in equilibrium and given the ruleset created by L, there are two types of games from Ego's perspective: those to which Ego is indifferent and those that Ego cares about. Games to which Ego is indifferent always generate a reward of 0 for Ego. Suppose, for example, that a game involves a rule requiring genuflecting by an Other. We assume Ego realizes no costs or benefits from the Other's choice about whether to genuflect or not, other than the cost of signaling that she is a punisher. We will thus use represent the type of Other with an indicator variable that idicates if she is a punisher for this game. Games Ego cares about are ones in which Ego receives a positive reward, R if there is another punisher present in the game and a negative reward −R if there is not. We call these important games. We formalize the set of important games as follows: G = {g ∈ G|U (g, •) = 0} U (g, t|g ∈ G ) = (2t − 1)R − c. We will use EU = E g,t [U (g, t)|g ∈ G ] to denote the expected utility of an important game. We let s denote the sparsity of the process generating games: the probability that a game is unimportant. s = 1 − P (g ∈ G ); g ∼ G. Critically, we assume that the sparsity of games does not alter the (expected) rate at which important games are presented to Ego. To be concrete, we assume the expected discounted reward obtained from important games is independent of s. This condition can be attained through a suitably modification of δ as a function of s: Proposition 2. Setting δ s = 1 − (1 − s)(1 − δ) ensures that the expected sum of discounted rewards from important games is independent of s: ∀s, ∈ [0, 1) E gj ,tj   ∞ j=0 δ j U (g j , t j ) g j ∈ G   = E gj ,tj   ∞ j=0 I[g j ∈ G ]δ j s U (g j , t j ) s   7 . Proof. We first show that it is sufficient to ensure that the expected value of δ j s is the same given that j is a round with an important game: E gj ,tj   ∞ j=0 I[g j ∈ G ]δ j s U (g j , t j ) s   = ∞ j=0 E gj ,tj I[g j ∈ G ]δ j s U (g j , t j ) s = ∞ j=0 δ j s E gj ,tj [U (g j , t j )|s, g j ∈ G ]E gj [I[g j ∈ G ]|s] = (1 − s)EU ∞ j=0 δ j s Where the first line holds by the linearity of expectation, and the fact that g j , t j are independent iid draws from a stationary distribution. Substituting the form of the infinite geometric series, we see that EU 1 − δ = (1 − s)EU 1 − δ s (1) is sufficient to acheive our goal. Substituting the form for δ s in the theorem statement and reducing shows that this condition is satisfied. It can be easily shows that this model describes a class of MABs. If the parameters that describe equilibrium were known, the decision problem would be trivial. The safe option corresponds to a constant arm, which is a degenerate distribution. Optimal policies for bandits with constant arms exhibit clear structure: is that if it is optimal to choose a known option in round j, it will be optimal to choose the known option in round j + 1 as well  (Bradt et al., 1956) . The argument is straighforward: if Ego reaches a point at which her estimate of the tradeoff between risking a negative payoff and learning so as to improve future decisions leads her optimally to choose not to participate, then her information state can never change and so her optimal choice can never be any different than the current opt-out decision. Thus, in our game, if Ego ever retires in a round, then she will never participate again. We refer to the decision not to participate at any point, then, as a decision to retire. Futhermore, an MAB is an instance of a POMDP, so it the optimal policy maps a distribution over states, a belief state, to decision between retirement and particpation. We give our agent a Beta prior over this parameter so that the belief space for our agent is a two dimensional lattice equivalent to Z 2 + . Initially, the belief state is (α 0 , β 0 ) and can be understood as the state an agent would be in if she had seen α 0 punishers and β 0 non-punishers. The conditional probability that a punisher is present in the first game is p αβ = α α + β . Once the games begin, Ego updates the prior beliefs using Bayes' rule, which in the Beta distribution means adding the counts of punishers and non-punishers observed to the prior values. In the following, we will use α i (β i ) to represent the number of observed punishers (non-punishers) prior to round i. A second useful result from the theory of multi-armed bandits is that the optimal policy is a function that maps a sequence of observations to a decision about retirement. If we restrict to two types, punishers and non-punishers, this problem is a partially observed version of a Markov decision process where the state is the probability, p, of drawing a punisher. From the theory of partially observable Markov decision processes (POMDPs), this optimal policy can also be represented as a mapping from a distribution over p to an action about retirement  [Puterman, 1994] . This reduces a partially observed process to a fully observed deterministic process in belief space.
 The Value of Sparsity Consider first the case in which the participation cost, c, is zero. In this case, Ego only faces a risky choice in periods in which she is presented with an important game. In all other periods, the per-period the expected payoff of playing the risky arm is a constant 0. Thus we can also conclude that if Ego retires, she will retire in a period in which she is playing an important game. In order to maintain the structure of a multi-armed bandit problem, we specify that if Ego chooses not to participate in an important game then the next game in the sequence is also an important game. We can think of this as a suspension of the game.  8  This rules out the possibility that Ego can simply choose not to play an important game and then reenter to play unimportant games in the hope of learning more before the next important game comes along. A decision not to participate is a decision to retire assuming a rational agent. We let i = 1 represent the state in which there is a punisher present in the game. The value of a state is then characterized by the following recursion. To simplify notation we let p αi,βi be the probability that a punisher will be present in the game in the belief state (α i , β i )and we abuse notation somewhat by letting V ((α i , β i ); s, δ) represent the discounted expected value of the super-game with sparsity s and discount factor δ s in the state (α i , β i ). We now show a property of the value of perfect information (VPI) in our super-game. The VPI for a state in a decision processis a measurement of the improvement in decision making as a function of information gathering actions [?]. It is defined as the amount a rational agent is willing to pay to remove all uncertainty associated with a particular random variable. Proposition 3. If the participation cost, c, is 0, then, for any belief state, (α i , β i ), and discount rate δ, the corresponding VPI goes to zero as the sparsity ratio goes to 1. That is lim s→1 V P I((α i , β i ); s, δ) = 0 (2) Proof. Given θ, it is easy to compute the value of participation: V (θ) = EU ∞ t=0 δ t = (2θ − 1) ∞ t=0 δ t = 2θ − 1 1 − δ . (3) The optimal full information policy π 0 will retire whenever V (θ) < 0. We use V + (θ) = max{V (θ), 0} denote the value of π 0 as a function of θ. The VPI is computed as the difference between the expected value of V + and the value of the optimal policy that only uses the history of observations: V P I((α i , β i ); s, δ) = E θ [ V + (θ)| (α i , β i )] − V * ((α i , β i ); s, δ) (4) We proceed by lower bounding V . V is the value of the optimal policy so it is weakly lower bounded by any arbitrary policy. A useful candidate is one-step greedy policy, π g , that always participates for unimportant games and retires in important games if the expected value of participation is negative (disregarding the benefit of new information). We let τ be the random number of games played before an important game is drawn. τ is geometrically distributed with success parameter 1 − s. We let n p be the random number of punishment actions observed prior to drawing an important game. The distribution over n p will be a binomial distribution conditioned on τ . Thus, the value of executing this policy ca be written as a joint expectation under τ and n p : V πg ((α i , β i ); s, δ) = E τ,np [max {E θ [V (θ )|(α i + n p , β i + τ − n p )] , 0} |(α i , β i ), s] . (5) We will be interested in the limit of this value, as s → 1. Before proceeding with that, we note that, from the law of large numbers, we have E θ [V (θ )|a + n p , b + τ − n p ] will concentrate about V (θ). Thus, E θ [ V + (θ )| (α i , β i )] = lim τ →∞ E np [max {E θ [ V (θ )| (α i + n, β i + τ − n p )] , 0} |τ, (α i , β i )] . (6) Note that E[V (θ)|(α i , β i )] only depends on the ratio of (α i , β i ), so the difference between the left and righthand sides of 6 is caused by the fact that the maximum must be taken at finitely many ratios (for finite τ ). Furthermore these ratios are evenly spaced out, so the lefthand side can only increase as τ increases. We can use this to lower bound the limit of V πg : lim s→1 V πg ((α i , β i ); s, δ) = lim s→1 E τ,np [max {E θ [V (θ )|(α i + n p , β i + τ − n p )] , 0} |s] (7) ≥ lim s→1 P (τ ≥ c(s)) min τ ≥c(s) E np [max {E θ [V (θ )|(α i + n p , β i + τ − n p )] , 0} |s, (α i , β i ), τ ] + P (τ ≤ c(s)) min τ <c(s) E np [max {E θ [V (θ )|(α i + n p , β i + τ − n p )] , 0} |s, (α i , β i ), τ ] (8) ≥ lim s→1 P (τ ≥ c(s))E np [max {E θ [ V (θ )| α i + n p , β i + c(s) − n p ] , 0} |c(s), (α i , β i )] (9) Using the form of the cumulative distribution of a geometric variable, P (τ ≥ c(s)) = 1 − P (τ < c(s)) = s c(s) . We set c(s) = − log 1 − s so that lim s→1 c(s) = ∞, and lim s→1 s c(s) = 1. Combining 6 and 9 with these facts allows us to deduce the following: lim s→1 V πgi ((α i , β i ); s, δ) ≥ E θ [V + (θ)|(α i , β i )] (10) Thus, lim s→1 V P I((α i , β i ); s, δ) ≤ 0. However, we have that, for any s, V P I((α i , β i ); s, δ) ≥ 0 by standard properties of VPI. This shows the result. Proposition 4. If the participation cost, c, is zero, then for any (α i , β i ) such that V + ( αi αi+βi ) > 0, VPI is strictly positive for s = 0. Proof. From Lai and Robbins  [1985] , we have that, for consistent and assymptotically efficient policies, the expected number of pulls of a suboptimal arm after n rounds is lower bounded by c log n, where c is a positive constant that measures the similarity of the reward distributions for the arms. This class contains the optimal policy. Thus, we have that for any finite s, V P I((α i , β i ); s, δ) > 0. (11) The combination of these two propositions shows that for any (α i , β i ), the corresponding value will eventually increase as s goes to 1. Thus, in the case where participation costs can be neglected, Ego will prefer an equilibrium with a higher s. shows low confidence initial beliefs. In these information states, Ego has seen very little data and so her belief is very spread out. In these scenarios, we expect sparsity to be helpful because the gap between full information and partial information is large. Conversly, (b) shows belief distributions after Ego has seen 50 effective samples. The corresponding distribution is more concentrated, so we expect less positive impact from sparsity.
 The Cost of Pervasive Normativity: Computational Results Our results above show that an environment with lots of rules that an agent cares nothing about intrinsically is more valuable for an agent contemplating participation than one in which the only rules are ones that matter on the meritsaltering Ego's payoff directly. We assumed, however, that increasing the number of spurious rules is costless to Ego and of course this is not generally likely to be true. If Ego is going to participate in a community with lots of spurious rules, Ego is also likely to bear costs, specifically the cost of participating in collective punishment and the cost of complying with spurious rules. In this section, we relax the assumption that c = 0. Doing so, however, increases the analytical complexity. We therefore turn to computational methods to explore environments in which Ego enjoys both costs and benefits from an increase in the number of spurious rules. To illustrate the effect of participation costs, we select six initial belief states and computed values as a function of c. Our initial beliefs vary the expected value of θ and the variance of the belief about its mean. We selected initial states to cover scenarios where the expected values of an important game is negative, positive and equal to zero. In this work, we chose E[θ] ∈ {.4, .5, .6}. We varied Ego's confidence in her current estimate by varying the effective number of samples (α + β) in the initial belief. Figure  1  shows the corresponding beta distributions for our selection of initial states. The optimal policy is invariant to scaling of rewards, so we fix P = 1 and let c P be the independent variable in our computations. We compute these values with a variant of value iteration that takes advantage of the structure of the state space. A python script to generate these plots is included as appendix A. We set the parameters of our computation to allow for at most 10 −8 of error in the computation. Figure  2 : Plots of V ((α i , β i ); s, .9) for select initial states. The probability density functions that correspond to these beliefs are shown in Fig.  1 . The one step value of participating in an equilibrium with enforcement is 1 (P = 1).The increase in value for larer sparsity for low values of c P shows confirmation of the results in Prop. 3. In comparing vertically, we can see that sparsity has a larger effects on states with low mean and low confidence: (f) is essentially unimpacted by sparsity for low c P while for (a) and (c) participation is suboptimal unless sparsity is positive. The cost of this sparsity is increased sensitivity to participation costs. Figure  2  shows value as a function of c P for several pairs of sparsity and initial states. We can clearly see the costs of pervasive normativity: value functions at higher sparsity decreases more quickly and for lower setting of the participation costs. This occurs because the number of rounds per important game increases so more participation costs are paid. In essence, increased participation costs force Ego to pay more for information and so she may prefer an equilibrium with less sparsity. As we might expect, the net gain in value from sparsity is larger in belief states with that are very uncertain. For example, the value function for initial state (30, 20) is essentially invariant to sparsity for low c P . Similarly, gains from increased sparsity are more pronouced with low means. Essentially all value in those states is due to improved decision making abilities from information.
 Implications for the Microfoundations of Law Our results have surprisingly powerful implications for the structure of normativity in human communities. [LETheory readers: we are only sketching these implications for this draft. We anticipate that at least some of them will be capable of more formal analytic or computational demonstration.]
 Gossip, Silly Rules and Durability The value of participating in a super-game depends on the expected cost of punishing rule violations relative to the reward Ego expects if violations of rules she cares about are deterred. The computations in Section 4 indicate that when expected costs relative to rewards are sufficiently low, ceteris paribus, Ego enjoys higher value in environments with more rules that she does not care about. This provides an interesting explanation for the observation from ethnographic studies that simple societies are characterized both by pervasive and apparently spurious rules that are effectively enforced by low-cost collective punishments.  Wiessner [2005] , for example documents the use of gossip, group criticism and mocking as the principal means by which norms are enforced among the Ju/'hoansi Bushmen of northwest Botswana. In her observations, violations of norms were punished by escalating criticism and rarely got to the point of physical violence. Assuming that the rewards generated for individuals aggregate to raise group well-being, our model thus can be read to predict that communities that succeed in securing an equilibrium with many rules that impose low compliance costs and which are enforced by low-cost means will outperform communities with fewer rules and more costly forms of punishment. 9
 Birds of a Feather The rewards Ego enjoys when joining a community depend on the rules of that community. In comparing across communities with comparable forms of punishment, and comparable numbers of rules, Ego will prefer a community with rules for important games that, when effectively enforced, generate higher rewards, R. We have not modeled the source of rules in a community but if we suppose that rules emerge that reflect the interests of the members of a community, this suggests that Ego is more likely to find rules that achieve higher rewards in communities with a number of agents with similar preferences, that is in more homogeneous communities.
 The Emergence of Law As communities grow more heterogeneous-which they naturally do as they generate value and support greater specialization through the division of labor-we expect ambiguity about rule violations to increase. Increasing ambiguity increases Ego's expected cost of participating in a punishment regime if errors in punishment-failing to punish when a violation is perceived by an Other and punishing when the Other does not perceive a violation-are themselves treated as rule violations which produce punishment. This is a feature of the punishment schemes of many communities: members of the Flemish cloth merchants guild in the 13th Century, for example, included in their rules the provision that any member who failed to observe a boycott of a buyer who had cheated another guild member was barred from trading and no guild member was to "house the goods" or "keep company" with the non-punisher. Increasing ambiguity also reduces the expected rewards Ego enjoys in important games because with some probability a violation perceived by Ego is not perceived as such by Others and hence the probability of a violation increases. Thus the cost of punishing relative to rewards decreases with decreasing ambiguity.  Hadfield and Weingast [2012]  argue that the central function of law is to serve as a unique classification institution, capable of resolving ambiguity about what counts as a rule violation. Moreover, they show that for a classification institution to effectively secure an equilibrium of legal order around a given set of rules enforced only by decentralized collective punishment, the institution must possess legal attributes such as neutrality, openness, clarity, consistency and stability. Our analysis here predicts that communities that introduce law in the form of a classification institution with legal attributes that reduces ambiguity will enjoy higher value and greater durability.
 Hammurabi's Code One of the key assumptions of our model is that people who punish any rule violation are expected to punish all rule violations. This is a distinctive feature of the labeling system generated by a legal regime: people are "law breakers" or not; they are "law-abiding" or not.  Cooter [1998]  proposes that "law" is a meaningful category and that people have preferences over behaviors solely on the basis of whether they are labeled "lawful" or not. Our model captures this idea by treating observation of punishment behavior in the context of any rule as informative of the probability of punishment in important games. The model can be interpreted as representing, for example, a community in which legal order is coordinated around a single legal code. Hammurabi's Code from ancient Babylon, for example, consisted of 247 individual rules, such as "If any one hire an ox or an ass, and a lion kill it in the field, the loss is upon its owner" (Rule 244) and "If any one open his ditches to water his crop, but is careless, and the water flood the field of his neighbor, then he shall pay his neighbor corn for his loss" (Rule 55). These rules likely emerged individually over time. We can imagine that knowing whether someone punished Rule 244 may or may not have helped to predict whether they would also punish Rule 55. But when Hammurabi placed all 247 together on a stone pillar and named the collection as his Code, he created the possibility for the emergence of two types of people: those who punished violations of the Code and those that did not. Our analysis suggests that the creation of collections of rules, rather than disparate rules, can generate value. Suppose, for example, that Ego cares about five rules, enjoying rewards when violations of each of them is deterred. Our model treats these five rules as integrated into a single super game in which the observation of punishment behavior in any game is informative, and equally so, of the likelihood of deterrence of violations in any of the five games Ego cares about. But suppose instead that these rules are not connected in this way. Suppose that punishment behavior in each game Ego cares about is only predicted by punishment behavior in non-overlapping subsets of unimportant games. We could then decompose our single super-game into five distinct super-games, each one of which would be considerably less sparse than our original game. Our results predict that Ego's value in a community with distinct super-games-with disconnected rules and a belief structure about punishment that limits the informativeness of observing punishment of any individual rule-will be lower than the value enjoyed in a community with a comprehensive code.
 Legal Pluralism and the Nation State Finally, we can combine some of the above observations to shed light on the phenomena of legal pluralism and the emergence of the nation state. Modern advanced legal systems are characterized by comprehensive systems of rules, with the label 'law' attributed to any rule generated by a government organized within the constitutional framework of a state. Indeed, law is frequently equated with the rules generated by a government, as distinguished from those generated from other entities such as corporations or schools or that emerge organically from social interaction  [Ellickson, 1991] . But as  Hadfield and Weingast [2013]  emphasize, prior to the emergence of the nation state, there were many institutions that coordinated legal order in different spheres, often in competition. Medieval Europe, for example, was characterized by multiple legal orders, with rules generated by merchant guilds, towns, churches, local rulers and more. Many societies, particularly those seen as struggling to establish the rule of law, are characterized by multiple legal orders-some governing family relations, others governing commercial dealings for example. Our model suggests a way of thinking about the tradeoffs Ego will face between participating in multiple legal communities, each of which coordinates punishment over some subset of rules and participating in a comprehensive legal community. On the one hand, rewards may be higher when a legal community is comprised of a relatively homogeneous group with shared interests-such as a community of traders-who can select rules that serve Ego's interests. But such a community will also have lower sparsity. On the other hand a system with a single system of comprehensive rules may achieve less alignment with Ego's interests but may also, because of its higher sparsity, provide Ego with more information about the value of continuing to participate.  Figure 1 : 1 Figure 1: Plots of a selection of initial belief distributions. Each curve corresponds to a different value of E[θ]. (a)
 fig = plt.figure() ax = fig.add_subplot(1, 1, 1) ax.set_xscale('log') ax.set_xlabel('c/P') ax.set_ylabel('Value')
			 Hadfield [1999]  shows that there is functionality to an arbitrary classification scheme of enforced norms: it coordinates investments in specialized learning by gender which raises the value of heterosexual matching in the marriage market.
			 Cooter [1998]  also suggests that the effectiveness of law can be understood as deriving from the classification of behavior as lawful or not. Cooter however presumes the existence of preferences based on this simple binary classification, that is, that at least some people inherently prefer to avoid actions labeled unlawful. This presumes that a category "law", which extends to potentially arbitrary actions, exists.
			 For an example of such a game, see Boyd et al. [2010] . They present an evolutionary game model in which punishment is a heritable strategy and deterrence requires multiple punishers. A population with a fraction of punishers can be stable in equilibrium when punishers can signal that they are punishers at low cost and thus avoid the costs of punishment if there are too few punishers present.4  It is straightforward to generalize our interpretation of c as the cost of complying with spurious rules to signal Ego's support for the equilibrium rules.5  In standard treatments, T is typically used for the transition distribution, we use P here to avoid confusion with our model specification.
			 If the function g(n) is Ω(f (n)), then there is a positive constant c such that g(n) ≥ cf (n)∀n
			 In any multi-armed bandit game, the decision to stop suspends the game: deciding to return to the game implies making the risky pull that was rejected previously.
			 Our model takes into account Ego's willingness to bear the higher cost of punishment in more sparse environments. We assume, but have not shown, that the willingness of Others to punish is not reduced with sparsity-that is, that Others have incentives comparable to Ego's.

Abstract: This paper proposes a mathematical model for a simplified version of the game defined in Hadfield and  Weingast [2012]  which proposes that legal order can be described as an equilibrium in thirdparty decentralized enforcement coordinated by a centralized classification institution. We explore the attractiveness of joining a new group (which is assumed to have settled on an enforcement equilibrium already) where groups differ in terms of the frequency of interactions in which norm violation is possible (normative interactions) and thus punishment is called for. We show that groups in which normative interactions are frequent but involve relatively unimportant rules may achieve higher value for participants.


Learning What to Value

Agents and Implementations Traditional agents  [2] [3] interact with their environments cyclically: in cycle k, an agent acts with action y k , then perceives observation x k . The interaction history of an agent with lifespan m is a string y 1 x 1 y 2 x 2 ...y m x m , also written yx 1:m or yx ≤m . Beyond these interactions, a traditional agent is isolated from its environment, so an agent can be formalized as an agent function from an interaction history yx <k to an action y k . Since we are concerned not with agents in the abstract, but with very powerful agents in the real world, we introduce the concept of an agent implementation. An agent implementation is a physical structure that, in the absence of interference from its environment, implements an agent function. In cycle k, an unaltered agent implementation executes its agent function on its recalled interaction history yx <k , sends the resulting y k into the environment as output, then receives and records an observation x k . An agent implementation's behavior is only guaranteed to match its implemented function so long as effects from the environment do not destroy the agent or alter its functionality. In keeping with this realism, an agent implementation's environment is considered to be the real world in which we live. We may engineer some parts of the world to meet our specifications, but (breaking with some traditional agent formulations) we do not consider the environment to be completely under our control, to be defined as we wish. Why would one want to study agent implementations? For narrowlyintelligent agents, the distinction between traditional agents and agent implementations may not be worth making. For ultraintelligent agents, the distinction is quite important: agent implementations offer us better predictions about how powerful agents will affect their environments and their own machinery, and are the basis for understanding real-world agents that model, defend, maintain, and improve themselves.
 Reinforcement Learning Reinforcement learning adds to the agent formalism the concept of reward, encoded as a scalar r k in each observation x k . The reinforcement learning problem is to define an agent that maximizes its total received rewards over the course of its interactions with its environment  [3] . In order to think clearly about ultraintelligent reinforcement learning agent implementations, we make use of Hutter's AIXI  [3][4] , an optimality notion that solves the reinforcement learning problem in a very general sense. AIXI's optimality means that the best an agent can do is to approximate a full search of all possible future interaction histories yx k:m , find the probability of each history, and take the action with the highest expected total reward. A simplified version of this optimality notion, adapted for use with agent implementations 1 , is given by y k = arg max y k x k yx k:m (r k + . . . + r m )P (yx ≤m |yx <k y k ) . ( The Trouble with Reinforcement Learning. The trouble with reinforcement learning is that, in the real world, it can only be used to define agents whose goal is to maximize observed rewards. Consider a hypothetical agent implementation AI-RL that approximates (1). It is appealing to think that AI-RL "has no goal" and will learn its goal from its environment, but this is not strictly true. AI-RL may in some sense learn instrumental goals, but its final goal is to maximize expected rewards in any way it can. Since human goals are not naturally instrumental to maximized rewards, the burden is on us to engineer the environment to prevent AI-RL from receiving rewards except when human goals are fulfilled. An AGI whose goals do not match ours is not desirable because it will work at cross-purposes to us in many cases. For example, AI-RL could benefit by altering its environment to give rewards regardless of whether 1 AIXI makes its optimality claims using the knowledge that it will continue to maximize expected rewards in the future, incorporating a rigid self-model (an expectimax tree). An agent implementation has no such knowledge, as environmental effects may interfere with future decisions. The optimality notion given here models itself as part of the world, using induction to predict its own future decisions; thanks to Peter de Blanc for this idea. We have also omitted detail we will not require by simplifying AIXI's Solomonoff prior ξ to P , some "appropriate distribution" over interaction histories. human goals are achieved  2  . This provides a strong incentive for AI-RL to free its rewards from their artificial dependence on fulfillment of human goals, which in turn creates a conflict of interest for us: increasing AI-RL's intelligence makes AI-RL more effective at achieving our goals, but it may allow AI-RL to devise a way around its restrictions. Worse, increasing intelligence could trigger an intelligence explosion  [1]   [8]  in which AI-RL repeatedly self-improves until it far surpasses our ability to contain it. Reinforcement learning is therefore not appropriate for a real-world AGI; the more intelligent a reinforcement learner becomes, the harder it is to use to achieve human goals, because no amount of careful design can yield a reinforcement learner that works towards human goals when it is not forced to.
 Learning What to Value In the following sections, we construct an optimality notion for implemented agents that can be designed to pursue any goal, and which can therefore be designed to treat human goals as final rather than instrumental goals. These agents are called value learners because, like reinforcement learners, they are flexible enough to be used even when a detailed specification of desired behavior is not known. The trouble with the reinforcement learning notion (  1 ) is that it can only prefer or disprefer future interaction histories on the basis of the rewards they contain. Reinforcement learning has no language in which to express alternative final goals, discarding all non-reward information contained in an interaction history. To solve this problem, our more expressive optimality notion replaces the sum of future rewards (r 1 + • • • + r m ) with some other evaluation of future interaction histories. First we will consider a fixed utility function, then we will generalize this notion to learn what to value. Observation-Utility Maximizers. Our first candidate for a reward replacement is inspired by Nick Hay's work  [2] , and is called an observation-utility function. Let U be a function from an interaction history yx ≤m to a scalar utility.
 U calculates expected utility given an interaction history. Observation-utility functions deserve a brief explanation. A properly-designed U uses all of the information in the interaction history yx ≤m to calculate the probabilities of different outcomes in the real world, then finds an expected utility by performing a probability-weighted average over the utilities of these outcomes. U must take into account the reliability of its sensors and be able to use local 2 Self-rewarding has been compared to a human stimulating their own pleasure center, e.g. using drugs  [3] . This metaphor is imperfect: while in humans, pleasure induces satiation and reduces activity, an agent governed by (1) that "hacks" its own rewards will not stop acting to maximize expected future rewards. It will continue to maintain and protect itself by acquiring free energy, space, time, and freedom from interference (as in  [5] ) in order to ensure that it will not be stopped from self-rewarding. Thus, an ultraintelligent self-rewarder could be highly detrimental to human interests. observations to infer distant events; it must also be able to distinguish between any outcomes with different values to humans, and assign proportional utilities to each. Putting U (yx ≤m ) in place of the sum of rewards (r 1 + . . . + r m ) produces an optimality notion that chooses actions so as to maximize the expected utility given its future interaction history: y k = arg max y k x k yx k:m U (yx ≤m )P (yx ≤m |yx <k y k ) . ( Unlike reinforcement learning, expected observation-utility maximization can be used to define agents with many different final goals  3  . Whereas reinforcement learners universally act to bring about interaction histories containing high rewards, an agent implementing (2) acts to bring about different futures depending upon its U . If U is designed appropriately, an expected utility maximizer could act so as to bring about any set of human goals. Unless we later decide that we don't want the goals specified by U to be fulfilled, we will not work at cross-purposes to such an agent, and increasing its intelligence will be in our best interest. Value-Learning Agents. Though an observed-utility maximizer can in principle have any goal, it requires a detailed observation-utility function U up front. This is not ideal; a major benefit of reinforcement learning was that it seemed to allow us to apply an intelligent agent to a problem without clearly defining its goal up front. Can this idea of learning to maximize an initially unknown utility function be recovered? To address this, we propose uncertainty over utility functions. Instead of providing an agent one utility function up front, we provide an agent with a pool of possible utility functions and a probability distribution P such that each utility function can be assigned probability P (U |yx ≤m ) given a particular interaction history. An agent can then calculate an expected value over possible utility functions given a particular interaction history: U U (yx ≤m )P (U |yx ≤m ). This recovers the kind of learning what to value that was desired in reinforcement learning agents. In designing P , we specify what kinds of interactions constitute evidence about goals; unlike rewards from reinforcement learning, this evidence is not elevated to an end in and of itself, and so does not lead the agent to seek evidence of easy goals 4 instead of acting to fulfill the goals it has learned. Replacing the reinforcement learner's sum of rewards with an expected utility over a pool of possible utility functions, we have an optimality notion for a value-learning agent: y k = arg max y k x k yx k:m P 1 (yx ≤m |yx <k y k ) U U (yx ≤m )P 2 (U |yx ≤m ) (3) A value-learning agent approximates a full search over all possible future interaction histories yx k:m , finds the probability of each future interaction history, and takes the action with the highest expected value, calculated by a weighted average over the agent's pool of possible utility functions.
 Conclusion Hutter's introduction to AIXI  [3]  offers a compelling statement of the goals of AGI: Most, if not all known facets of intelligence can be formulated as goaldriven or, more precisely, as maximizing some utility function. It is, therefore, sucient to study goal-driven AI... The goal of AI systems should be to be useful to humans. The problem is that, except for special cases, we know neither the utility function nor the environment in which the agent will operate in advance. Reinforcement learning, we have argued, is not an adequate real-world solution to the problem of maximizing an initially unknown utility function. Reinforcement learners, by definition, act to maximize their expected observed rewards; they may learn that human goals are in some cases instrumentally useful to high rewards, but this dynamic is not tenable for agents of human or higher intelligence, especially considering the possibility of an intelligence explosion. Value learning, on the other hand, is an example framework expressive enough to be used in agents with goals other than reward maximization. This framework is not a full design for a safe, ultraintelligent agent; at very least, the design of probability distributions and model pools for utility functions is crucial and nontrivial, and still better frameworks for ultraintelligent agents likely exist. Value learners do not solve all problems of ultraintelligent agent design, but do give a direction for future work on this topic. 			 It is tempting to think that an observation-utility maximizer (let us call it AI-OUM ) would be motivated, as AI-RL was, to take control of its own utility function U . This is a misunderstanding of how AI-OUM makes its decisions. According to (2) , actions are chosen to maximize the expected utility given its future interaction history according the current utility function U , not according to whatever utility function it may have in the future. Though it could modify its future utility function, this modification is not likely to maximize U , and so will not be chosen. By similar argument, AI-OUM will not "fool" its future self by modifying its memories.Slightly trickier is the idea that AI-OUM could act to modify its sensors to report favorable observations inaccurately. As noted above, a properly designed U takes into account the reliability of its sensors in providing information about the real world. If AI-OUM tampers with its own sensors, evidence of this tampering will appear in the interaction history, leading U to consider observations unreliable with respect to outcomes in the real world; therefore, tampering with sensors will not produce high expected-utility interaction histories.
			 As long as P obeys the axioms of probability, an agent cannot purposefully increase or decrease the probability of any possible utility function through its actions.

Abstract: I. J. Good's intelligence explosion theory predicts that ultraintelligent agents will undergo a process of repeated self-improvement; in the wake of such an event, how well our values are fulfilled would depend on the goals of these ultraintelligent agents. With this motivation, we examine ultraintelligent reinforcement learning agents. Reinforcement learning can only be used in the real world to define agents whose goal is to maximize expected rewards, and since this goal does not match with human goals, AGIs based on reinforcement learning will often work at cross-purposes to us. To solve this problem, we define value learners, agents that can be designed to learn and maximize any initially unknown utility function so long as we provide them with an idea of what constitutes evidence about that utility function.