3c-pomdp.md

---
layout: chapter
title: "Environments with hidden state: POMDPs"
description: Mathematical formalism for POMDPs, Bandit and Restaurant Choice examples.
---

## Introduction: Learning about the world from observation

The previous chapters made two strong assumptions that often fail in practice. First, we assumed the environment was an MDP, where the state is fully observed by the agent at all times. Second, we assumed that the agent starts off with *full knowledge* of the MDP -- rather than having to learn its parameters from experience. This chapter relaxes the first assumption by introducing POMDPs. The next [chapter](/chapters/3d-reinforcement-learning.html) introduces **reinforcement learning**, an approach to learning MDPs from experience.

## POMDP Agent Model

### Informal overview

In an MDP the agent observes the full state of the environment at each timestep. In Gridworld, for instance, the agent always knows their precise position and is uncertain only about their future position. Yet in real-world problems, the agent often does not observe the full state every timestep. For example, suppose you are sailing at night without any navigation instruments. You might be very uncertain about your precise position and you only learn about it indirectly, by waiting to observe certain landmarks in the distance. For environments where the state is only observed partially and indirectly, we use Partially Observed Markov Decision Processes (POMDPs).

In a Partially Observed Markov Decision Process (POMDP), the agent knows the transition function of the environment. This distinguishes POMDPs from [Reinforcement Learning](/chapters/3d-reinforcement-learning.html) problems. However, the agent starts each episode uncertain about the precise state of the environment. For example, if the agent is choosing where to eat on holiday, they may be uncertain about their own location and uncertain about which restaurants are open.

The agent learns about the state indirectly via *observations*. At each timestep, they receive an observation that depends on the true state and their previous action (according to a fixed *observation function*). They update a probability distribution on the current state and then choose an action. The action causes a state transition just like in an MDP but the agent only receives indirect evidence about the new state.

As an example, consider the Restaurant Choice Problem. Suppose Bob doesn't know whether the Noodle Shop is open. Previously, the agent's state consisted of Bob's location on the grid as well as the remaining time. In the POMDP case, the state also represents whether or not the Noodle Shop is open, which determines whether Bob can enter the Noodle Shop. When Bob gets close enough to the Noodle Shop, he will observe whether or not it's open. Bob's planning should take this into account: if the Noodle Shop is closed then Bob will observe this can simply head to a different restaurant.

### Formal model

We first define the class of decision probems (POMDPs) and then define an agent model for optimally solving these problems. Our definitions are based on reft:kaelbling1998planning.

A Partially Observable Markov Decision Process (POMDP) is a tuple $$ \left\langle S,A(s),T(s,a),U(s,a),\Omega,O \right\rangle$$, where:

- $$S$$ (state space), $$A$$ (action space), $$T$$ (transition function), $$U$$ (utility or reward function) form an MDP as defined in [chapter 3.1](/chapters/3a-mdp.html), with $$U$$ assumed to be deterministic[^utility].

- $$\Omega$$ is the finite space of observations the agent can receive.

- $$O$$ is a function  $$ O\colon S \times A \to \Delta \Omega $$. This is the *observation function*, which maps an action $$a$$ and the state $$s'$$ resulting from taking $$a$$ to an observation $$o \in \Omega$$ drawn from $$O(s',a)$$.

[^utility]: In the RL literature, the utility or reward function is often allowed to be *stochastic*. Our agent models assume that the agent's utility function is deterministic. To represent environments with stochastic "rewards", we treat the reward as a stochastic part of the environment (i.e. the world state). So in a Bandit problem, instead of the agent receiving a (stochastic) reward $$R$$, they transition to a state to which they assign a fixed utility $$R$$. (Why do we avoid stochastic utilities? One focus of this tutorial is inferring an agent's preferences. The preferences are fixed over time and non-stochastic. We want to identify the agent's utility function with their preferences).

So at each timestep, the agent transitions from state $$s$$ to state $$s' \sim T(s,a)$$ (where $$s$$ and $$s'$$ are generally unknown to the agent) having performed action $$a$$. On entering $$s'$$ the agent receives an observation $$o \sim O(s',a)$$ and a utility $$U(s,a)$$.

To characterize the behavior of an expected-utility maximizing agent, we need to formalize the belief-updating process. Let $$b$$, the current belief function, be a probability distribution over the agent's current state. Then the agent's succesor belief function $$b'$$ over their next state is the result of a Bayesian update on the observation $$o \sim O(s',a)$$ where $$a$$ is the agent's action in $$s$$. That is:

**Belief-update formula:**

$$
b'(s') \propto O(s',a,o)\sum_{s \in S}{T(s,a,s')b(s)}
$$

Intuitively, the probability that $$s'$$ is the new state depends on the marginal probability of transitioning to $$s'$$ (given $$b$$) and the probability of the observation $$o$$ occurring in $$s'$$. The relation between the variables in a POMDP is summarized in Figure 1 (below).

>**Figure 1:** The dependency structure between variables in a POMDP.

The ordering of events in Figure 1 is as follows:

>(1). The agent chooses an action $$a$$ based on belief distribution $$b$$ over their current state (which is actually $$s$$).

>(2). The agent gets utility $$u = U(s,a)$$ when leaving state $$s$$ having taken $$a$$.

>(3). The agent transitions to state $$s' \sim T(s,a)$$, where it gets observation $$o \sim O(s',a)$$ and updates its belief to $$b'$$ by updating $$b$$ on the observation $$o$$.

In our previous agent model for MDPs, we defined the expected utility of an action $$a$$ in a state $$s$$ recursively in terms of the expected utility of the resulting pair of state $$s'$$ and action $$a'$$. This same recursive characterization of expected utility still holds. The important difference is that the agent's action $$a'$$ in $$s'$$ depends on their updated belief $$b'(s')$$. Hence the expected utility of $$a$$ in $$s$$ depends on the agent's belief $$b$$ over the state $$s$$. We call the following the **POMDP Expected Utility of State Recursion**. This recursion defines the function $$EU_{b}$$, which is analogous to the *value function*, $$V_{b}$$, in reft:kaelbling1998planning.

**POMDP Expected Utility of State Recursion:**

$$
EU_{b}[s,a] = U(s,a) + \mathbb{E}_{s',o,a'}(EU_{b'}[s',a'_{b'}])
$$

where:

- we have $$s' \sim T(s,a)$$ and $$o \sim O(s',a)$$

- $$b'$$ is the updated belief function $$b$$ on observation $$o$$, as defined above

- $$a'_{b'}$$ is the softmax action the agent takes given belief $$b'$$

The agent cannot use this definition to directly compute the best action, since the agent doesn't know the state. Instead the agent takes an expectation over their belief distribution, picking the action $$a$$ that maximizes the following:

$$
EU[b,a] = \mathbb{E}_{s \sim b}(EU_{b}[s,a])
$$

We can also represent the expected utility of action $$a$$ given belief $$b$$ in terms of a recursion on the successor belief state. We call this the **Expected Utility of Belief Recursion**, which is closely related to the Bellman Equations for POMDPs:

$$
EU[b,a] = \mathbb{E}_{s \sim b}( U(s,a) + \mathbb{E}_{s',o,a'}(EU[b',a']) )
$$

where $$s'$$, $$o$$, $$a'$$ and $$b'$$ are distributed as in the Expected Utility of State Recursion.

Unfortunately, finding the optimal policy for POMDPs is intractable. Even in the special case where observations are deterministic and the horizon is finite, determining whether the optimal policy has expected utility greater than some constant is PSPACE-complete refp:papadimitriou1987complexity.

### Implementation of the Model

As with the agent model for MDPs, we provide a direct translation of the equations above into an agent model for solving POMDPs. The variables `nextState`, `nextObservation`, `nextBelief`, and `nextAction` correspond to $$s'$$,  $$o$$, $$b'$$ and $$a'$$ respectively, and we use the Expected Utility of Belief Recursion.

~~~~

var updateBelief = function(belief, observation, action){
    return Infer({ model() {
      var state = sample(belief);
      var predictedNextState = transition(state, action);
      var predictedObservation = observe(predictedNextState);
      condition(_.isEqual(predictedObservation, observation));
      return predictedNextState;
    }});
};

var act = function(belief) {
  return Infer({ model() {
    var action = uniformDraw(actions);
    var eu = expectedUtility(belief, action);
    factor(alpha * eu);
    return action;
  }});
};

var expectedUtility = function(belief, action) {
  return expectation(
    Infer({ model() {
      var state = sample(belief);
      var u = utility(state, action);
      if (state.terminateAfterAction) {
        return u;
      } else {
        var nextState = transition(state, action);
        var nextObservation = observe(nextState);
        var nextBelief = updateBelief(belief, nextObservation, action);
        var nextAction = sample(act(nextBelief));
        return u + expectedUtility(nextBelief, nextAction);
      }
    }}));
};

// To simulate the agent, we need to transition
// the state, sample an observation, then
// compute agent's action (after agent has updated belief).

// *startState* is agent's actual startState (unknown to agent)
// *priorBelief* is agent's initial belief function

var simulate = function(startState, priorBelief) {

  var sampleSequence = function(state, priorBelief, action) {
    var observation = observe(state);
    var belief = updateBelief(priorBelief, observation, action);
    var action = sample(act(belief));
    var output = [ [state, action] ];

    if (state.terminateAfterAction){
      return output;
    } else {
      var nextState = transition(state, action);
      return output.concat(sampleSequence(nextState, belief, action));
    }
  };
  return sampleSequence(startState, priorBelief, 'noAction');
};
~~~~

## Applying the POMDP agent model

### Multi-arm Bandits

[Multi-armed Bandits](https://en.wikipedia.org/wiki/Multi-armed_bandit) are an especially simple class of sequential decision problem. A Bandit problem has a single state and multiple actions ("arms"), where each arm has a distribution on rewards/utilities that is initially unknown. The agent has a finite time horizon and must balance exploration (i.e. learn about the reward distribution) with exploitation (obtain reward).

Bandits can be modeled as Reinforcement Learning problems, where the agent learns a good policy for an initially unknown MDP. This is the practical way to solve Bandits and the next [chapter](/chapters/3d-reinforcement-learning.html) illustrates this approach. Here we model Bandits as POMDPs and use the code above to find the optimal policy for some toy Bandit problems[^optimal]. (We choose a Bandit example to demonstrate the difficulty of exactly solving even the simplest POMDPs.)

[^optimal]: In the standard Bandit problem, there is a single unknown MDP characterized by the reward distribution of each arm. In a more challenging generalization, the agent faces a sequence of random Bandit problems that are drawn from some prior. If we treat a standard Bandit as a POMDP, we compute the Bayes optimal policy for the single Bandit and by doing so, we implicitly compute the optimal policy for a sequence of Bandits drawn from the same prior. This is analogous to finding the optimal policy for an MDP: the optimal policy covers every possible state, including those occurring with tiny probability. Model-free RL, by contrast, will focus on the states that are actually encountered in practice.

In our examples, the arms are labeled with integers and arm $$i$$ has Bernoulli distributed rewards with parameter $$\theta_i$$. In the first codebox (below), the true reward distribution,  $$(\theta_0,\theta_1)$$, is $$(0.7,0.8)$$ but the agent's prior is uniform over $$(0.7,0.8)$$ and $$(0.7,0.2)$$. So the agent's only uncertainty is over $$\theta_1$$.

Rather than implement everything in the codebox, we use the library [webppl-agents](https://github.com/agentmodels/webppl-agents). This includes functions for constructing a Bandit environment (`makeBanditPOMDP`), for constructing a POMDP agent (`makePOMDPAgent`) and for running the agent on the environment (`simulatePOMDP`). This [chapter](/chapters/guide-library.html) explains how to use webppl-agents. The Appendix includes a codebox with a full implementation of a POMDP agent on a Bandit problem.

~~~~
///fold: displayTrajectory

// Takes a trajectory containing states and actions and returns one containing
// locs and actions, getting rid of 'start' and the final meaningless action.
var displayTrajectory = function(trajectory) {
  var getPrizeAction = function(stateAction) {
    var state = stateAction[0];
    var action = stateAction[1];
    return [state.manifestState.loc, action];
  };

  var prizesActions = map(getPrizeAction, trajectory);
  var flatPrizesActions = _.flatten(prizesActions);
  var actionsPrizes = flatPrizesActions.slice(1, flatPrizesActions.length - 1);

  var printOut = function(n) {
    print('\n Arm: ' + actionsPrizes[2*n] + ' -- Prize: '
          + actionsPrizes[2*n + 1]);
  };
  return map(printOut, _.range((actionsPrizes.length)*0.5));
};
///

// 1. Construct Bandit POMDP

// Reward distributions are Bernoulli
var getRewardDist = function(theta){
  return Categorical({ vs:[0,1], ps: [1-theta, theta]});
}

// True reward distributions are [.7,.8].
var armToRewardDist = {
  0: getRewardDist(.7),
  1: getRewardDist(.8)
};

// But the agent's prior is uniform over [.7,.8] and [.7,.2].
var alternateArmToRewardDist = {
  0: getRewardDist(.7),
  1: getRewardDist(.2)
}

// Options for library function for Bandits. Number of trials = horizon.
var banditOptions = {
  numberOfArms: 2,
  armToPrizeDist: armToRewardDist,
  numberOfTrials: 11,
  numericalPrizes: true
};

var bandit = makeBanditPOMDP(banditOptions);
var startState = bandit.startState;
var world = bandit.world;

// 2. Construct POMDP agent

// Prior as described above and *latentState* is an implementation detail
// for the libraries implementation of POMDPs
var priorBelief = Infer({ model() {
  var armToRewardDist = uniformDraw([armToRewardDist,
                                    alternateArmToRewardDist]);
  return extend(startState, { latentState: armToRewardDist });
}});

var utility = function(state, action) {
  var reward = state.manifestState.loc;
  return reward === 'start' ? 0 : reward;
};

var params = {
  priorBelief,
  utility,
  alpha: 1000
};

var agent = makePOMDPAgent(params, bandit.world);

// 3. Simulate agent and return state-action pairs

var trajectory = simulatePOMDP(startState, world, agent, 'stateAction');
displayTrajectory(trajectory);

~~~~

Solving Bandit problems using the simple dynamic approach of our POMDP agent does not blows up as the horizon ("number of trials") and number of arms increase. The codebox below shows how runtime scales as a function of the number of trials. (This takes approximately 20 seconds to run.)

~~~~
///fold: Construct world and agent priorBelief as above

var getRewardDist = function(theta){
  return Categorical({ vs:[0,1], ps: [1-theta, theta]});
}

// True reward distributions are [.7,.8].
var armToRewardDist = {
  0: getRewardDist(.7),
  1: getRewardDist(.8)
};

// But the agent's prior is uniform over [.7,.8] and [.7,.2].
var alternateArmToRewardDist = {
  0: getRewardDist(.7),
  1: getRewardDist(.2)
}

var makeBanditWithNumberOfTrials = function(numberOfTrials) {
  return makeBanditPOMDP({
    numberOfTrials,
	numberOfArms: 2,
	armToPrizeDist: armToRewardDist,
	numericalPrizes: true
  });
};

var getPriorBelief = function(numberOfTrials){
  return Infer({ model() {
    var armToPrizeDist = uniformDraw([armToRewardDist,
                                      alternateArmToRewardDist]);
    return makeBanditStartState(numberOfTrials, armToPrizeDist);
  }})
};

var baseParams = { alpha: 1000 };
///

// Simulate agent for a given number of Bandit trials
var getRuntime = function(numberOfTrials) {
  var bandit = makeBanditWithNumberOfTrials(numberOfTrials);
  var world = bandit.world;
  var startState = bandit.startState;
  var priorBelief = getPriorBelief(numberOfTrials)
  var params = extend(baseParams, { priorBelief });
  var agent = makeBanditAgent(params, bandit, 'belief');

  var f = function() {
    return simulatePOMDP(startState, world, agent, 'stateAction');
  };

  return timeit(f).runtimeInMilliseconds.toPrecision(3) * 0.001;
};

// Runtime as a function of number of trials
var numberOfTrialsList = _.range(15).slice(2);
var runtimes = map(getRuntime, numberOfTrialsList);
viz.line(numberOfTrialsList, runtimes);

~~~~

Scaling is much worse in the number of arms. The following may take over a minute to run:

~~~~
///fold:

var getRewardDist = function(theta){
  return Categorical({ vs:[0,1], ps: [1-theta, theta]});
}

var makeArmToRewardDist = function(numberOfArms) {
  return map(function(x) { return getRewardDist(0.8); }, _.range(numberOfArms));
};

var armToRewardDistSampler = function(numberOfArms) {
  return map(function(x) { return uniformDraw([getRewardDist(0.2),
                                               getRewardDist(0.8)]); },
             _.range(numberOfArms));
};

var getPriorBelief = function(numberOfTrials, numberOfArms) {
  return Infer({ model() {
    var armToRewardDist = armToRewardDistSampler(numberOfArms);
    return makeBanditStartState(numberOfTrials, armToRewardDist);
  }});
};

var baseParams = {alpha: 1000};
///

var getRuntime = function(numberOfArms) {
  var armToRewardDist = makeArmToRewardDist(numberOfArms);
  var options = {
    numberOfTrials: 5,
	armToPrizeDist: armToRewardDist,
	numberOfArms,
	numericalPrizes: true
  };
  var numberOfTrials = options.numberOfTrials;
  var bandit = makeBanditPOMDP(options);
  var world = bandit.world;
  var startState = bandit.startState;
  var priorBelief = getPriorBelief(numberOfTrials, numberOfArms);
  var params = extend(baseParams, { priorBelief });
  var agent = makeBanditAgent(params, bandit, 'belief');

  var f = function() {
    return simulatePOMDP(startState, world, agent, 'stateAction');
  };

  return timeit(f).runtimeInMilliseconds.toPrecision(3) * 0.001;
};

// Runtime as a function of number of arms
var numberOfArmsList = [1, 2, 3];
var runtimes = map(getRuntime, numberOfArmsList);
viz.line(numberOfArmsList, runtimes);
~~~~

### Gridworld with observations

A person looking for a place to eat will not be *fully* informed about all local restaurants. This section extends the [Restaurant Choice problem](/chapters/3a-mdp.html) to represent an agent with uncertainty about which restaurants are open. The agent *observes* whether a restaurant is open by moving to one of the grid locations adjacent to the restaurant. If the restaurant is open, the agent can enter and receive utility.

The POMDP version of Restaurant Choice is built from the MDP version. States now have the form:

>`{manifestState: { ... }, latentState: { ... }}`

The `manifestState` contains the features of the world that the agent always observes directly (and so always knows). This includes the remaining time and the agent's location in the grid. The `latentState` contains features that are only observable in certain states. In our examples, `latentState` specifies whether each restaurant is open or closed. The transition function for the POMDP is the same as the MDP except that if a restaurant is closed the agent cannot transition to it.

The next two codeboxes use the same POMDP, where all restaurants are open but for Noodle. The first agent prefers the Donut Store and believes (falsely) that Donut South is likely closed. The second agent prefers Noodle and believes (falsely) that Noodle is likely open.

~~~~
///fold:
var getPriorBelief = function(startManifestState, latentStateSampler){
  return Infer({ model() {
    return {
      manifestState: startManifestState,
      latentState: latentStateSampler()};
  }});
};

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var pomdp = makeGridWorldPOMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});
///

var utilityTable = {
  'Donut N': 5,
  'Donut S': 5,
  'Veg': 1,
  'Noodle': 1,
  'timeCost': -0.1
};
var utility = function(state, action) {
  var feature = pomdp.feature;
  var name = feature(state.manifestState).name;
  if (name) {
    return utilityTable[name];
  } else {
    return utilityTable.timeCost;
  }
};

var latent = {
  'Donut N': true,
  'Donut S': true,
  'Veg': true,
  'Noodle': false
};
var alternativeLatent = extend(latent, {
  'Donut S': false,
  'Noodle': true
});

var startState = {
  manifestState: {
    loc: [3, 1],
    terminateAfterAction: false,
    timeLeft: 11
  },
  latentState: latent
};

var latentStateSampler = function() {
  return categorical([0.8, 0.2], [alternativeLatent, latent]);
};

var priorBelief = getPriorBelief(startState.manifestState, latentStateSampler);
var agent = makePOMDPAgent({ utility, priorBelief, alpha: 100 }, pomdp);
var trajectory = simulatePOMDP(startState, pomdp, agent, 'states');
var manifestStates = _.map(trajectory, _.property('manifestState'));

viz.gridworld(pomdp.MDPWorld, { trajectory: manifestStates });
~~~~

Here is the agent that prefers Noodle and falsely belives that it is open:

~~~~
///fold: Same world, prior, start state, and latent state as previous codebox
var getPriorBelief = function(startManifestState, latentStateSampler){
  return Infer({ model() {
    return {
      manifestState: startManifestState,
      latentState: latentStateSampler()
    };
  }});
};

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var pomdp = makeGridWorldPOMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var latent = {
  'Donut N': true,
  'Donut S': true,
  'Veg': true,
  'Noodle': false
};
var alternativeLatent = extend(latent, {
  'Donut S': false,
  'Noodle': true
});

var startState = {
  manifestState: {
    loc: [3, 1],
    terminateAfterAction: false,
    timeLeft: 11
  },
  latentState: latent
};

var latentSampler = function() {
  return categorical([0.8, 0.2], [alternativeLatent, latent]);
};

var priorBelief = getPriorBelief(startState.manifestState, latentSampler);
///

var utilityTable = {
  'Donut N': 1,
  'Donut S': 1,
  'Veg': 3,
  'Noodle': 5,
  'timeCost': -0.1
};
var utility = function(state, action) {
  var feature = pomdp.feature;
  var name = feature(state.manifestState).name;
  if (name) {
    return utilityTable[name];
  } else {
    return utilityTable.timeCost;
  }
};
var agent = makePOMDPAgent({ utility, priorBelief, alpha: 100 }, pomdp);
var trajectory = simulatePOMDP(startState, pomdp, agent, 'states');
var manifestStates = _.map(trajectory, _.property('manifestState'));

viz.gridworld(pomdp.MDPWorld, { trajectory: manifestStates });
~~~~

When does it make sense to treat this Restaurant Choice problem as a POMDP? As with Bandits, if the problem we face is a fixed (but initially unknown) MDP, and we get many  episodes in which to learn by trial and error, then Reinforcement Learning is a simple and scalable approach. If the MDP varies with every episode (e.g. the hidden state of whether a restaurant is open varies from day to day), then POMDP methods may work better. (Even in the case where the MDP is fixed, if the stakes are very high, it will be best to solve for the optimal POMDP policy.) Finally, if our goal is to model human planning, then POMDP models are worth considering as they are more sample efficient than RL techniques (and humans can often solve planning problems in very few tries).

The next [chapter](/chapters/3d-reinforcement-learning.html) is on reinforcement learning, an approach which *learns* to solve an initially unknown MDP.

### Appendix: Complete Implementation of POMDP agent for Bandits

We apply the POMDP agent to a simplified variant of the Multi-arm Bandit Problem. In this variant, pulling an arm produces a *prize* deterministically. The agent begins with uncertainty about the mapping from arms to prizes and learns by trying the arms. In our example, there are only two arms. The first arm is known to have the prize "chocolate" and the second arm either has "champagne" or has no prize at all ("nothing"). See Figure 2 (below) for details.

>**Figure 2:** Diagram for deterministic Bandit problem used in the codebox below. The boxes represent possible deterministic mappings from arms to prizes. Each prize has a reward/utility $$u$$. On the right are the agent's initial beliefs about the probability of each mapping. The true mapping (i.e. true *latent state*) has a solid outline.

In our implementation of this problem, the two arms are labeled "0" and "1" respectively. The *action* of pulling `Arm0` is also labeled "0" (and likewise for `Arm1`). After taking action `0`, the agent transitions to a state corresponding to the prize for `Arm0` and the gets to observe this prize. States are Javascript objects that contain a property for counting down the time (as in the MDP case) as well as a `prize` property. States also contain the *latent* mapping from arms to prizes (called `armToPrize`) that determines how an agent transitions on pulling an arm.

~~~~
// Pull arm0 or arm1
var actions = [0, 1];

// Use latent "armToPrize" mapping in state to
// determine which prize agent gets
var transition = function(state, action){
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    prize: state.armToPrize[action],
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft == 1
  });
};

// After pulling an arm, agent observes associated prize
var observe = function(state){
  return state.prize;
};

// Starting state specifies the latent state that agent tries to learn
// (In order that *prize* is defined, we set it to 'start', which
// has zero utilty for the agent).
var startState = {
  prize: 'start',
  timeLeft: 3,
  terminateAfterAction:false,
  armToPrize: { 0: 'chocolate', 1: 'champagne' }
};
~~~~

Having illustrated our implementation of the POMDP agent and the Bandit problem, we put the pieces together and simulate the agent's behavior. The `makeAgent` function is a simplified version of the library function `makeBeliefAgent` used throughout the rest of this tutorial[^makeBelief].

The Belief-Update Formula is implemented by `updateBelief`. Instead of hand-coding a Bayesian belief update, we simply use WebPPL's built in inference primitives. This approach means our POMDP agent can do any kind of inference that WebPPL itself can do. For this tutorial, we use the inference function `Enumerate`, which captures exact inference over discrete belief spaces. By changing the inference function, we get a POMDP agent that does approximate inference and simulates their future selves as doing approximate inference. This inference could be over discrete or continuous belief spaces. (WebPPL includes Particle Filters, MCMC, and Hamiltonian Monte Carlo for differentiable models).

[^makeBelief]: One difference between the functions is that `makeAgent` uses the global variables `transition` and `observation`, instead of having a `world` parameter.

~~~~
///fold: Bandit problem is defined as above

// Pull arm0 or arm1
var actions = [0, 1];

// Use latent "armToPrize" mapping in state to
// determine which prize agent gets
var transition = function(state, action){
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    prize: state.armToPrize[action],
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft == 1
  });
};

// After pulling an arm, agent observes associated prize
var observe = function(state){
  return state.prize;
};

// Starting state specifies the latent state that agent tries to learn
// (In order that *prize* is defined, we set it to 'start', which
// has zero utilty for the agent).
var startState = {
  prize: 'start',
  timeLeft: 3,
  terminateAfterAction:false,
  armToPrize: {0:'chocolate', 1:'champagne'}
};
///

// Defining the POMDP agent

// Agent params include utility function and initial belief (*priorBelief*)

var makeAgent = function(params) {
  var utility = params.utility;

  // Implements *Belief-update formula* in text
  var updateBelief = function(belief, observation, action){
    return Infer({ model() {
      var state = sample(belief);
      var predictedNextState = transition(state, action);
      var predictedObservation = observe(predictedNextState);
      condition(_.isEqual(predictedObservation, observation));
      return predictedNextState;
    }});
  };

  var act = dp.cache(
    function(belief) {
      return Infer({ model() {
        var action = uniformDraw(actions);
        var eu = expectedUtility(belief, action);
        factor(1000 * eu);
        return action;
      }});
    });

  var expectedUtility = dp.cache(
    function(belief, action) {
      return expectation(
        Infer({ model() {
          var state = sample(belief);
          var u = utility(state, action);
          if (state.terminateAfterAction) {
            return u;
          } else {
            var nextState = transition(state, action);
            var nextObservation = observe(nextState);
            var nextBelief = updateBelief(belief, nextObservation, action);
            var nextAction = sample(act(nextBelief));
            return u + expectedUtility(nextBelief, nextAction);
          }
        }}));
    });

  return { params, act, expectedUtility, updateBelief };
};

var simulate = function(startState, agent) {
  var act = agent.act;
  var updateBelief = agent.updateBelief;
  var priorBelief = agent.params.priorBelief;

  var sampleSequence = function(state, priorBelief, action) {
    var observation = observe(state);
    var belief = ((action === 'noAction') ? priorBelief :
                  updateBelief(priorBelief, observation, action));
    var action = sample(act(belief));
    var output = [[state, action]];

    if (state.terminateAfterAction){
      return output;
    } else {
      var nextState = transition(state, action);
      return output.concat(sampleSequence(nextState, belief, action));
    }
  };
  // Start with agent's prior and a special "null" action
  return sampleSequence(startState, priorBelief, 'noAction');
};

//-----------
// Construct the agent

var prizeToUtility = {
  chocolate: 1,
  nothing: 0,
  champagne: 1.5,
  start: 0
};

var utility = function(state, action) {
  return prizeToUtility[state.prize];
};

// Define true startState (including true *armToPrize*) and
// alternate possibility for startState (see Figure 2)

var numberTrials = 1;
var startState = {
  prize: 'start',
  timeLeft: numberTrials + 1,
  terminateAfterAction: false,
  armToPrize: { 0: 'chocolate', 1: 'champagne' }
};

var alternateStartState = extend(startState, {
  armToPrize: { 0: 'chocolate', 1: 'nothing' }
});

// Agent's prior
var priorBelief = Categorical({
  ps: [.5, .5],
  vs: [startState, alternateStartState]
});

var params = { utility: utility, priorBelief: priorBelief };
var agent = makeAgent(params);
var trajectory = simulate(startState, agent);

print('Number of trials: ' + numberTrials);
print('Arms pulled: ' +  map(second, trajectory));
~~~~

You can change the agent's behavior by varying `numberTrials`, `armToPrize` in `startState` or the agent's prior. Note that the agent's final arm pull is random because the agent only gets utility when *leaving* a state.

### Footnotes


Modeling Agents with Probabilistic Programs

Owain Evans
Andreas Stuhlmüller
John Salvatier
Daniel Filan

5a-time-inconsistency.md

---
layout: chapter
title: "Time inconsistency I"
description: Exponential vs. hyperbolic discounting, Naive vs. Sophisticated planning.

---

### Introduction

Time inconsistency is part of everyday human experience. In the night you wish to rise early; in the morning you prefer to sleep in. There is an inconsistency between what you prefer your future self to do and what your future self prefers to do. Forseeing this inconsistency, you take actions in the night to bind your future self to get up. These range from setting an alarm clock to arranging for someone drag you out of bed.

This pattern is not limited to attempts to rise early. People make failed resolutions to attend a gym regularly. Students procrastinate on writing papers, planning to start early but delaying until the last minute. Empirical studies have highlighted the practical import of time inconsistency both to completing online courses refp:patterson2015can and to watching highbrow movies refp:milkman2009highbrow. Time inconsistency has been used to explain not just quotidian laziness but also addiction, procrastination, and impulsive behavior, as well an array of "pre-commitment" behaviors refp:ainslie2001breakdown.

Lab experiments of time inconsistency often use simple quantitative questions such as:

>**Question**: Would you prefer to get $100 after 30 days or $110 after 31 days?

Most people prefer the $110. But a significant proportion of people reverse their earlier preference once the 30th day comes around and they contemplate getting $100 immediately. How can this time consistency be captured by a formal model?

### Time inconsistency due to hyperbolic discounting

This chapter models time inconsistency as resulting from *hyperbolic discounting*. The idea is that humans prefer receiving the same rewards sooner rather than later and the *discount function* describing this quantitatively is a hyperbola. Before describing the hyperbolic model, we provide some background on time discounting and incorporate it into our previous agent models.

#### Exponential discounting for optimal agents

The examples of decision problems in previous chapters have a *known*, *finite* time horizon. Yet there are practical decision problems that are better modeled as having an *unbounded* or *infinite* time horizon. For example, if someone tries to travel home after a vacation, there is no obvious time limit for their task. The same holds for a person saving or investing for the long-term.

Generalizing the previous agent models to the unbounded case faces a difficulty. The *infinite* summed expected utility of an action will (generally) not converge. The standard solution is to model the agent as maximizing the *discounted* expected utility, where the discount function is exponential. This makes the infinite sums converge and results in an agent model that is analytically and computationally tractable. Aside from mathematical convenience, exponential discounting might also be an accurate model of the "time preference" of certain rational agents[^justification]. Exponential discounting represents a (consistent) preference for good things happening sooner rather than later[^exponential].

[^justification]: People care about a range of things: e.g. the food they eat daily, their careers, their families, the progress of science, the preservation of the earth's environment. Many have argued that humans have a time preference. So models that infer human preferences from behavior should be able to represent this time preference.

[^exponential]: There are arguments that exponential discounting is the uniquely rational mode of discounting for agents with time preference. The seminal paper by refp:strotz1955myopia proves that, "in the continuous time setting, the only discount function such that the optimal policy doesn't vary in time is exponential discounting". In the discrete-time setting, refp:lattimore2014general prove the same result, as well as discussing optimal strategies for sophisticated time-inconsistent agents.

What are the effects of exponential discounting? We return to the deterministic Bandit problem from Chapter III.3 (see Figure 1). Suppose a person decides every year where to go on a skiing vacation. There is a fixed set of options {Tahoe, Chile, Switzerland} and a finite time horizon[^bandit]. The person discounts exponentially and so they prefer a good vacation now to an even better one in the future. This means they are less likely to *explore*, since exploration takes time to pay off.

>**Figure 1**: Deterministic Bandit problem. The agent tries different arms/destinations and receives rewards. The reward for Tahoe is known but Chile and Switzerland are both unknown. The actual best option is Tahoe.

[^bandit]: As noted above, exponential discounting is usually combined with an *unbounded* time horizon. However, if a human makes a series of decisions over a long time scale, then it makes sense to include their time preference. For this particular example, imagine the person is looking for the best skiing or sports facilities and doesn't care about variety. There could be a known finite time horizon because at some age they are too old for adventurous skiing.

~~~~
///fold:
var baseParams = {
  noDelays: false,
  discount: 0,
  sophisticatedOrNaive: 'naive'
};

var armToPlace = function(arm){
  return {
    0: "Tahoe",
    1: "Chile",
    2: "Switzerland"
  }[arm];
};

var display = function(trajectory) {
  return map(armToPlace, most(trajectory));
};
///

// Arms are skiing destinations:
// 0: "Tahoe", 1: "Chile", 2: "Switzerland"

// Actual utility for each destination
var trueArmToPrizeDist = {
  0: Delta({ v: 1 }),
  1: Delta({ v: 0 }),
  2: Delta({ v: 0.5 })
};

// Constuct Bandit world
var numberOfTrials = 10;
var bandit = makeBanditPOMDP({
  numberOfArms: 3,
  armToPrizeDist: trueArmToPrizeDist,
  numberOfTrials,
  numericalPrizes: true
});

var world = bandit.world;
var start = bandit.startState;

// Agent prior for utility of each destination
var priorBelief = Infer({ model() {
  var armToPrizeDist = {
    // Tahoe has known utility 1:
    0: Delta({ v: 1 }),
    // Chile has high variance:
    1: categorical([0.9, 0.1],
                   [Delta({ v: 0 }), Delta({ v: 5 })]),
    // Switzerland has high expected value:
    2: uniformDraw([Delta({ v: 0.5 }), Delta({ v: 1.5 })])
  };
  return makeBanditStartState(numberOfTrials, armToPrizeDist);
}});

var discountFunction = function(delay) {
  return Math.pow(0.5, delay);
};

var exponentialParams = extend(baseParams, { discountFunction, priorBelief });
var exponentialAgent = makeBanditAgent(exponentialParams, bandit,
                                       'beliefDelay');
var exponentialTrajectory = simulatePOMDP(start, world, exponentialAgent, 'actions');

var optimalParams = extend(baseParams, { priorBelief });
var optimalAgent = makeBanditAgent(optimalParams, bandit, 'belief');
var optimalTrajectory = simulatePOMDP(start, world, optimalAgent, 'actions');

print('exponential discounting trajectory: ' + display(exponentialTrajectory));
print('\noptimal trajectory: ' + display(optimalTrajectory));
~~~~

#### Discounting and time inconsistency

Exponential discounting is typically thought of as a *relative* time preference. A fixed reward will be discounted by a factor of $$\delta^{-30}$$ if received on Day 30 rather than Day 0. On Day 30, the same reward is discounted by $$\delta^{-30}$$ if received on Day 60 and not at all if received on Day 30. This relative time preference is "inconsistent" in a superficial sense. With $$\delta=0.95$$ per day (and linear utility in money), $100 after 30 days is worth $21 and $110 at 31 days is worth $22. Yet when the 30th day arrives, they are worth $100 and $105 respectively[^inconsistent]! The key point is that whereas these *magnitudes* have changed, the *ratios* stay fixed. Indeed, the ratio between a pair of outcomes stays fixed regardless of when the exponential discounter evaluates them. In summary: while a discounting agent evaluates two prospects in the future as worth little compared to similar near-term prospects, the agent agrees with their future self about which of the two future prospects is better.

[^inconsistent]: One can think of exponential discounting in a non-relative way by choosing a fixed staring time in the past (e.g. the agent's birth) and discounting everything relative to that. This results in an agent with a preference to travel back in time to get higher rewards!

Any smooth discount function other than an exponential will result in preferences that reverse over time refp:strotz1955myopia. So it's not so suprising that untutored humans should be subject to such reversals[^reversal]. Various functional forms for human discounting have been explored in the literature. We describe the *hyperbolic discounting* model refp:ainslie2001breakdown because it is simple and well-studied. Other functional form can be substituted into our models.

[^reversal]: Without computational aids, human representations of discrete and continuous quantities (including durations in time and dollar values) are systematically inaccurate. See refp:dehaene2011number.

Hyperbolic and exponential discounting curves are illustrated in Figure 2. We plot the discount factor $$D$$ as a function of time $$t$$ in days, with constants $$\delta$$ and $$k$$ controlling the slope of the function. In this example, each constant is set to 2. The exponential is:

$$
D=\frac{1}{\delta^t}
$$

The hyperbolic function is:

$$
D=\frac{1}{1+kt}
$$

The crucial difference between the curves is that the hyperbola is initially steep and then becomes almost flat, while the exponential continues to be steep. This means that exponential discounting is time consistent and hyperbolic discounting is not.

~~~~
var delays = _.range(7);
var expDiscount = function(delay) {
  return Math.pow(0.5, delay);
};
var hypDiscount = function(delay) {
  return 1.0 / (1 + 2*delay);
};
var makeExpDatum = function(delay){
  return {
    delay,
    discountFactor: expDiscount(delay),
    discountType: 'Exponential discounting: 1/2^t'
  };
};
var makeHypDatum = function(delay){
  return {
    delay,
    discountFactor: hypDiscount(delay),
    discountType: 'Hyperbolic discounting: 1/(1 + 2t)'
  };
};
var expData = map(makeExpDatum, delays);
var hypData = map(makeHypDatum, delays);
viz.line(expData.concat(hypData), { groupBy: 'discountType' });
~~~~

>**Figure 2:** Graph comparing exponential and hyperbolic discount curves.

>**Exercise:** We return to our running example but with slightly different numbers. The agent chooses between receiving $100 after 4 days or $110 after 5 days. The goal is to compute the preferences over each option for both exponential and hyperbolic discounters, using the discount curves shown in Figure 2. Compute the following:

> 1. The discounted utility of the $100 and $110 rewards relative to Day 0 (i.e. how much the agent values each option when the rewards are 4 or 5 days away).
>2. The discounted utility of the $100 and $110 rewards relative to Day 4 (i.e. how much each option is valued when the rewards are 0 or 1 day away).

### Time inconsistency and sequential decision problems

We have shown that hyperbolic discounters have different preferences over the $100 and $110 depending on when they make the evaluation. This conflict in preferences leads to complexities in planning that don't occur in the optimal (PO)MDP agents which either discount exponentially or do not discount at all.

Consider the example in the exercise above and imagine you have time inconsistent preferences. On Day 0, you write down your preference but on Day 4 you'll be free to change your mind. If you know your future self would choose the $100 immediately, you'd pay a small cost now to *pre-commit* your future self. However, if you believe your future self will share your current preferences, you won't pay this cost (and so you'll end up taking the $100). This illustrates a key distinction. Time inconsistent agents can be "Naive" or "Sophisticated":

- **Naive agent**: assumes his future self shares his current time preference. For example, a Naive hyperbolic discounter assumes his far future self has a nearly flat discount curve (rather than the "steep then flat" discount curve he actually has).

- **Sophisticated agent**: has the correct model of his future self's time preference. A Sophisticated hyperbolic discounter has a nearly flat discount curve for the far future but is aware that his future self does not share this discount curve.

Both kinds of agents evaluate rewards differently at different times. To distinguish a hyperbolic discounter's current and future selves, we refer to the agent acting at time $$t_i$$ as the $$t_i$$-agent. A Sophisticated agent, unlike a Naive agent, has an accurate model of his future selves. The Sophisticated $$t_0$$-agent predicts the actions of the $$t$$-agents (for $$t>t_0$$) that would conflict with his preferences. To prevent these actions, the $$t_0$$-agent tries to take actions that *pre-commit* the future agents to outcomes the $$t_0$$-agent prefers[^sophisticated].

[^sophisticated]: As has been pointed out previously, there is a kind of "inter-generational" conflict between agent's future selves. If pre-commitment actions are available at time $$t_0$$, the $$t_0$$-agent does better in expectation if it is Sophisticated rather than Naive. Equivalently, the $$t_0$$-agent's future selves will do better if the agent is Naive.

### Naive and Sophisticated Agents: Gridworld Example

Before describing our formal model and implementation of Naive and Sophisticated hyperbolic discounters, we illustrate their contrasting behavior using the Restaurant Choice example. We use the MDP version, where the agent has full knowledge of the locations of restaurants and of which restaurants are open. Recall the problem setup:

>**Restaurant Choice**: Bob is looking for a place to eat. His decision problem is to take a sequence of actions such that (a) he eats at a restaurant he likes and (b) he does not spend too much time walking. The restaurant options are: the Donut Store, the Vegetarian Salad Bar, and the Noodle Shop. The Donut Store is a chain with two local branches. We assume each branch has identical utility for Bob. We abbreviate the restaurant names as "Donut South", "Donut North", "Veg" and "Noodle".

The only difference from previous versions of Restaurant Choice is that restaurants now have *two* utilities. On entering a restaurant, the agent first receives the *immediate reward* (i.e. how good the food tastes) and at the next timestep receives the *delayed reward* (i.e. how good the person feels after eating it).

**Exercise:** Run the codebox immediately below. Think of ways in which Naive and Sophisticated hyperbolic discounters with identical preferences (i.e. identical utilities for each restaurant) might differ for this decision problem.

~~~~
///fold: restaurant choice MDP
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});
///
viz.gridworld(mdp.world, { trajectory: [mdp.startState] });
~~~~

The next two codeboxes show the behavior of two hyperbolic discounters. Each agent has the same preferences and discount function. They differ only in that the first is Naive and the second is Sophisticated.

~~~~
///fold: restaurant choice MDP, naiveTrajectory
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var naiveTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"l"],
  [{"loc":[2,5],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5],"timeAtRestaurant":0},"l"],
  [{"loc":[2,5],"terminateAfterAction":true,"timeLeft":6,"previousLoc":[2,5],"timeAtRestaurant":1},"l"]
];
///
viz.gridworld(mdp.world, { trajectory: naiveTrajectory });
~~~~

~~~~
///fold: restaurant choice MDP, sophisticatedTrajectory
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var sophisticatedTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"r"],
  [{"loc":[4,3],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"r"],
  [{"loc":[5,3],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[4,3]},"u"],
  [{"loc":[5,4],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[5,3]},"u"],
  [{"loc":[5,5],"terminateAfterAction":false,"timeLeft":5,"previousLoc":[5,4]},"u"],
  [{"loc":[5,6],"terminateAfterAction":false,"timeLeft":4,"previousLoc":[5,5]},"l"],
  [{"loc":[4,6],"terminateAfterAction":false,"timeLeft":3,"previousLoc":[5,6]},"u"],
  [{"loc":[4,7],"terminateAfterAction":false,"timeLeft":2,"previousLoc":[4,6],"timeAtRestaurant":0},"l"],
  [{"loc":[4,7],"terminateAfterAction":true,"timeLeft":2,"previousLoc":[4,7],"timeAtRestaurant":1},"l"]
];
///
viz.gridworld(mdp.world, { trajectory: sophisticatedTrajectory });
~~~~

>**Exercise:** (Try this exercise *before* reading further). Your goal is to do preference inference from the observed actions in the codeboxes above (using only a pen and paper). The discount function is the hyperbola $$D=1/(1+kt)$$, where $$t$$ is the time from the present, $$D$$ is the discount factor (to be multiplied by the utility) and $$k$$ is a positive constant. Find a single setting for the utilities and discount function that produce the behavior in both the codeboxes above. This includes utilities for the restaurants (both *immediate* and *delayed*) and for the `timeCost` (the negative utility for each additional step walked), as well as the discount constant $$k$$. Assume there is no softmax noise.

------

The Naive agent goes to Donut North, even though Donut South (which has identical utility) is closer to the agent's starting point. One possible explanation is that the Naive agent has a higher utility for Veg but gets "tempted" by Donut North on their way to Veg[^naive_path].

[^naive_path]: At the start, no restaurants can be reached quickly and so the agent's discount function is nearly flat when evaluating each one of them. This makes Veg look most attractive (given its higher overall utility). But going to Veg means getting closer to Donut North, which becomes more attractive than Veg once the agent is close to it (because of the discount function). Taking an inefficient path -- one that is dominated by another path -- is typical of time-inconsistent agents.

The Sophisticated agent can accurately model what it *would* do if it ended up in location [3,5] (adjacent to Donut North). So it avoids temptation by taking the long, inefficient route to Veg.

In this simple example, the Naive and Sophisticated agents each take paths that optimal time-consistent MDP agents (without softmax noise) would never take. So this is an example where a bias leads to a *systematic* deviation from optimality and behavior that is not predicted by an optimal model. In Chapter 5.3 we explore inference of preferences for time inconsistent agents.

Next chapter: [Time inconsistency II](/chapters/5b-time-inconsistency.html)

### Footnotes


6b-inference-sampling.md

---
layout: chapter
title: Sampling-based methods
description: A brief review of sampling-based methods. Discuss MCMC over enumeration with caching for inverse planning.
status: stub
is_section: false
hidden: true
---


5d-joint-inference.md

---
layout: chapter
title: Joint inference of biases and preferences I
description: Assuming agent optimality leads to mistakes in inference. Procrastination and Bandit Examples.

---

### Introduction

Techniques for inferring human preferences and beliefs from their behavior on a task usually assume that humans solve the task (softmax) optimally. When this assumptions fails, inference often fails too. This chapter explores how incorporating time inconsistency and myopic planning into models of human behavior can improve inference.

### Formalization of Joint Inference

 We formalize joint inference over beliefs, preferences and biases by extending the approach developed in Chapter IV, "[Reasoning about Agents](/chapters/4-reasoning-about-agents)", where an agent was formally defined by parameters $$  \left\langle U, \alpha, b_0 \right\rangle$$. To include the possibility of time-inconsistency and myopia, an agent $$\theta$$ is now characterized by a tuple of parameters as follows:

$$
\theta = \left\langle U, \alpha, b_0, k, \nu, C \right\rangle
$$

where:

- $$U$$ is the utilty function

- $$\alpha$$ is the softmax noise parameter

- $$b_0$$ is the agent's belief (or prior) over the initial state

- $$k \geq 0$$ is the constant for hyperbolic discounting function $$1/(1+kd)$$

- $$\nu$$ is an indicator for Naive or Sophisticated hyperbolic discounting

- $$C \in [1,\infty]$$ is the integer cutoff or bound for Reward-myopic or Update-myopic Agents[^bound]

As in Equation (2) of Chapter IV, we condition on state-action-observation triples:

$$
P(\theta \vert (s,o,a)_{0:n}) \propto P( (s,o,a)_{0:n} \vert \theta)P(\theta)
$$

We obtain a factorized form in exactly the same way as in Equation (2), i.e. we generate the sequence $$b_i$$ from $$i=0$$ to $$i=n$$ of agent beliefs:

$$
P(\theta \vert (s,o,a)_{0:n}) \propto
P(\theta) \prod_{i=0}^n P( a_i \vert s_i, b_i, U, \alpha, k, \nu, C )
$$

The likelihood term on the right-hand side of this equation is simply the softmax probability that the agent with given parameters chooses $$a_i$$ in state $$s_i$$. This equation for inference does not make use of the *delay* indices used by time-inconsistent and Myopic agents. This is because the delays figure only in their internal simulations. In order to compute the likelihood the agent takes an action, we don't need to keep track of delay values.

[^bound]: To simplify the presentation, we assume here that one does inference either about whether the agent is Update-myopic or about whether the agent is Reward-myopic (but not both). It's actually straightforward to include both kinds of agents in the hypothesis space and infer both $$C_m$$ and $$C_g$$.

## Learning from Procrastinators

The Procrastination Problem (Figure 1 below) illustrates how agents with identical preferences can deviate *systematically* in their behavior due to time inconsistency. Suppose two agents care equally about finishing the task and assign the same cost to doing the hard work. The optimal agent will complete the task immediately. The Naive hyperbolic discounter will delay every day until the deadline, which could be thirty days away!

>**Figure 1:** Transition graph for Procrastination Problem. States are represented by nodes. Edges are state-transitions and are labeled with the action name and the utility of the state-action pair. Terminal nodes have a bold border and their utility is labeled below.

This kind of systematic deviation between agents is also significant for inferring preferences. We consider the problem of *online* inference, where we observe the agent's behavior each day and produce an estimate of their preferences. Suppose the agent has a deadline $$T$$ days into the future and leaves the work till the last day. This is typical human behavior -- and so is a good test for a model of inference.

We compare the online inferences of two models. The *Optimal Model* assumes the agent is time-consistent with softmax parameter $$\alpha$$. The *Possibly Discounting* model includes both optimal and Naive hyperbolic discounting agents in its prior. (The Possibly Discounting model includes the Optimal Model as a special case; this allows us to place a uniform prior on the models and exploit [Bayesian Model Selection](http://alumni.media.mit.edu/~tpminka/statlearn/demo/).)

For each model, we compute posteriors for the agent's parameters after observing the agent's choice at each timestep. We set $$T=10$$. So the observed actions are:

>`["wait", "wait", "wait", ... , "work"]`

where `"work"` is the final action. We fix the utilities for doing the work (the `workCost` or $$-w$$) and for delaying the work (the `waitCost` or $$-\epsilon$$). We infer the following parameters:

- The reward received after completing the work: $$R$$ or `reward`
- The agent's softmax parameter: $$\alpha$$
- The agent's discount rate (for the Possibly Discounting model): $$k$$ or `discount`

Note that we are not just inferring *whether* the agent is biased; we also infer how biased they are. For each parameter, we plot a time-series showing the posterior expectation of the parameter on each day. We also plot the model's posterior predictive probability that the agent does the work on the last day (assuming the agent gets to the last day without having done the work).

~~~~
// infer_procrastination

///fold: makeProcrastinationMDP, makeProcrastinationUtility, displayTimeSeries, ...
var makeProcrastinationMDP = function(deadlineTime) {
  var stateLocs = ["wait_state", "reward_state"];
  var actions = ["wait", "work", "relax"];

  var stateToActions = function(state) {
    return (state.loc === "wait_state" ?
            ["wait", "work"] :
            ["relax"]);
  };

  var advanceTime = function (state) {
    var newTimeLeft = state.timeLeft - 1;
    var terminateAfterAction = (newTimeLeft === 1 ||
                                state.loc === "reward_state");
    return extend(state, {
      timeLeft: newTimeLeft,
      terminateAfterAction: terminateAfterAction
    });
  };

  var transition = function(state, action) {
    assert.ok(_.includes(stateLocs, state.loc) && _.includes(actions, action),
              'procrastinate transition:' + [state.loc,action]);

    if (state.loc === "reward_state") {
      return advanceTime(state);
    } else if (action === "wait") {
      var waitSteps = state.waitSteps + 1;
      return extend(advanceTime(state), { waitSteps });
    } else {
      var newState = extend(state, { loc: "reward_state" });
      return advanceTime(newState);
    }
  };

  var feature = function(state) {
    return state.loc;
  };

  var startState = {
    loc: "wait_state",
    waitSteps: 0,
    timeLeft: deadlineTime,
    terminateAfterAction: false
  };

  return {
    actions,
    stateToActions,
    transition,
    feature,
    startState
  };
};

var makeProcrastinationUtility = function(utilityTable) {
  assert.ok(hasProperties(utilityTable, ['waitCost', 'workCost', 'reward']),
            'makeProcrastinationUtility args');
  var waitCost = utilityTable.waitCost;
  var workCost = utilityTable.workCost;
  var reward = utilityTable.reward;

  // NB: you receive the *workCost* when you leave the *wait_state*
  // You then receive the reward when leaving the *reward_state* state
  return function(state, action) {
    if (state.loc === "reward_state") {
      return reward + state.waitSteps * waitCost;
    } else if (action === "work") {
      return workCost;
    } else {
      return 0;
    }
  };
};

var getMarginal = function(dist, key){
  return Infer({ model() {
    return sample(dist)[key];
  }});
};

var displayTimeSeries = function(observedStateAction, getPosterior) {
  var features = ['reward', 'predictWorkLastMinute', 'alpha', 'discount'];

  // dist on {a:1, b:3, ...} -> [E('a'), E('b') ... ]
  var distToMarginalExpectations = function(dist, keys) {
    return map(function(key) {
      return expectation(getMarginal(dist, key));
    }, keys);
  };
  // condition observations up to *timeIndex* and take expectations
  var inferUpToTimeIndex = function(timeIndex, useOptimalModel) {
    var observations = observedStateAction.slice(0, timeIndex);
    return distToMarginalExpectations(getPosterior(observations, useOptimalModel), features);
  };

  var getTimeSeries = function(useOptimalModel) {

    var inferAllTimeIndexes = map(function(index) {
      return inferUpToTimeIndex(index, useOptimalModel);
    }, _.range(observedStateAction.length));

    return map(function(i) {
      // get full time series of online inferences for each feature
      return map(function(infer){return infer[i];}, inferAllTimeIndexes);
    }, _.range(features.length));
  };

  var displayOptimalAndPossiblyDiscountingSeries = function(index) {
    print('\n\nfeature: ' + features[index]);
    var optimalSeries = getTimeSeries(true)[index];
    var possiblyDiscountingSeries = getTimeSeries(false)[index];
    var plotOptimal = map(
      function(pair){
        return {
          t: pair[0],
          expectation: pair[1],
          agentModel: 'Optimal'
        };
      },
      zip(_.range(observedStateAction.length), optimalSeries));
    var plotPossiblyDiscounting = map(
      function(pair){
        return {
          t: pair[0],
          expectation: pair[1],
          agentModel: 'Possibly Discounting'
        };
      },
      zip(_.range(observedStateAction.length),
          possiblyDiscountingSeries));
    viz.line(plotOptimal.concat(plotPossiblyDiscounting),
             { groupBy: 'agentModel' });
  };

  print('Posterior expectation on feature after observing ' +
        '"wait" for t timesteps and "work" when t=9');
  map(displayOptimalAndPossiblyDiscountingSeries, _.range(features.length));
  return '';
};

var procrastinationData = [[{"loc":"wait_state","waitSteps":0,"timeLeft":10,"terminateAfterAction":false},"wait"],[{"loc":"wait_state","waitSteps":1,"timeLeft":9,"terminateAfterAction":false},"wait"],[{"loc":"wait_state","waitSteps":2,"timeLeft":8,"terminateAfterAction":false},"wait"],[{"loc":"wait_state","waitSteps":3,"timeLeft":7,"terminateAfterAction":false},"wait"],[{"loc":"wait_state","waitSteps":4,"timeLeft":6,"terminateAfterAction":false},"wait"],[{"loc":"wait_state","waitSteps":5,"timeLeft":5,"terminateAfterAction":false},"wait"],[{"loc":"wait_state","waitSteps":6,"timeLeft":4,"terminateAfterAction":false},"wait"],[{"loc":"wait_state","waitSteps":7,"timeLeft":3,"terminateAfterAction":false},"wait"],[{"loc":"wait_state","waitSteps":8,"timeLeft":2,"terminateAfterAction":false},"work"],[{"loc":"reward_state","waitSteps":8,"timeLeft":1,"terminateAfterAction":true},"relax"]];
///

var getPosterior = function(observedStateAction, useOptimalModel) {
  var world = makeProcrastinationMDP();
  var lastChanceState = secondLast(procrastinationData)[0];

  return Infer({ model() {

    var utilityTable = {
      reward: uniformDraw([0.5, 2, 3, 4, 5, 6, 7, 8]),
      waitCost: -0.1,
      workCost: -1
    };
    var params = {
      utility: makeProcrastinationUtility(utilityTable),
      alpha: categorical([0.1, 0.2, 0.2, 0.2, 0.3], [0.1, 1, 10, 100, 1000]),
      discount: useOptimalModel ? 0 : uniformDraw([0, .5, 1, 2, 4]),
      sophisticatedOrNaive: 'naive'
    };

    var agent = makeMDPAgent(params, world);
    var act = agent.act;

    map(function(stateAction) {
      var state = stateAction[0];
      var action = stateAction[1];
      observe(act(state, 0), action);
    }, observedStateAction);

    return {
      reward: utilityTable.reward,
      alpha: params.alpha,
      discount: params.discount,
      predictWorkLastMinute: sample(act(lastChanceState, 0)) === 'work'
    };
  }});
};

displayTimeSeries(procrastinationData, getPosterior);
~~~~

The optimal model makes inferences that clash with everyday intuition. Suppose someone has still not done a task with only two days left. Would you confidently rule out them doing it at the last minute?

With two days left, the Optimal model has almost complete confidence that the agent doesn't care about the task enough to do the work (`reward < workCost = 1`). Hence it assigns probability $$0.005$$ to the agent doing the task at the last minute (`predictWorkLastMinute`). By contrast, the Possibly Discounting model predicts the agent will do the task with probability around $$0.2$$. The predictive probability is no higher than $$0.2$$ because the Possibly Discounting model allows the agent to be optimal (`discount==0`) and because a sub-optimal agent might be too lazy to do the work even at the last minute (i.e. `discount` is high enough to overwhelm `reward`).

Suppose someone completes the task on the final day. What do you infer about them? The Optimal Model has to explain the action by massively revising its inference about `reward` and $$\alpha$$. It suddenly infers that the agent is extremely noisy and that `reward > workCost` by a big margin. The extreme noise is needed to explain why the agent would miss a good option nine out of ten times. By contrast, the Possibly Discounting Model does not change its inference about the agent's noise level very much at all (in terms of pratical significance). It infers a much higher value for `reward`, which is plausible in this context.

----------

## Learning from Reward-myopic Agents in Bandits

Chapter V.2. "[Bounded Agents](/chapters/5c-myopia)" explained that Reward-myopic agents explore less than optimal agents. The Reward-myopic agent plans each action as if time runs out in $$C_g$$ steps, where $$C_g$$ is the *bound* or "look ahead". If exploration only pays off in the long-run (after the bound) then the agent won't explore[^bandit1]. This means there are two possible explanations for an agent not exploring: either the agent is greedy or the agent has a low prior on the utility of the unknown options.

[^bandit1]: If there's no noise in transitions or in selection of actions, the Reward-myopic agent will *never* explore and will do poorly.

We return to the deterministic bandit-style problem from earlier. At each trial, the agent chooses between two arms with the following properties:

- `arm0`: yields chocolate

- `arm1`: yields either champagne or no prize at all (agent's prior is $$0.7$$ for nothing)

The inference problem is to infer the agent's preference over chocolate. While having only two deterministic arms may seem overly simple, the same structure is shared by realistic problems. For example, we can imagine observing people choosing between different cuisines, restaurants or menu options. Usually people know about some options well but are uncertain about others. When inferring their preferences, we distinguish between options chosen for exploration vs. exploitation. The same applies to people choosing media sources: someone might try out a channel to learn whether it shows their favorite genre.

As with the Procrastination example above, we compare the inferences of two models. The *Optimal Model* assumes the agent solves the POMDP optimally. The *Possibly Reward-myopic Model* includes both the optimal agent and Reward-myopic agents with different values for the bound $$C_g$$. The models know the agent's utility for champagne and his prior about how likely champagne is from `arm1`. The models have a fixed prior on the agent's utility for chocolate. We vary the agent's time horizon between 2 and 10 timesteps and plot posterior expectations for the utility of chocolate. For the Possibly Reward-myopic model, we also plot the expectation for $$C_g$$.

~~~~
// helper function to assemble and display inferred values
///fold:
var timeHorizonValues = _.range(10).slice(2);
var features = ['Utility of arm 0 (chocolate)', 'Greediness bound'];

var displayExpectations = function(getPosterior) {

  var getExpectations = function(useOptimalModel) {

    var inferAllTimeHorizons = map(function(horizon) {
      return getPosterior(horizon, useOptimalModel);
    }, timeHorizonValues);

    return map(
      function(i) {
        return map(function(infer){return infer[i];}, inferAllTimeHorizons);
      },
      _.range(features.length));
  };

  var displayOptimalAndPossiblyRewardMyopicSeries = function(index) {
    print('\n\nfeature: ' + features[index]);
    var optimalSeries = getExpectations(true)[index];
    var possiblyRewardMyopicSeries = getExpectations(false)[index];
    var plotOptimal = map(
      function(pair) {
        return {
          horizon: pair[0],
          expectation: pair[1],
          agentModel: 'Optimal'
        };
      },
      zip(timeHorizonValues, optimalSeries));
    var plotPossiblyRewardMyopic = map(
      function(pair){
        return {
          horizon: pair[0],
          expectation: pair[1],
          agentModel: 'Possibly RewardMyopic'
        };
      },
      zip(timeHorizonValues, possiblyRewardMyopicSeries));
    viz.line(plotOptimal.concat(plotPossiblyRewardMyopic),
             { groupBy: 'agentModel' });
  };

  print('Posterior expectation on feature after observing no exploration');
  map(displayOptimalAndPossiblyRewardMyopicSeries, _.range(features.length));
  return '';
};

var getMarginal = function(dist, key){
  return Infer({ model() {
    return sample(dist)[key];
  }});
};
///

var getPosterior = function(numberOfTrials, useOptimalModel) {
  var trueArmToPrizeDist = {
    0: Delta({ v: 'chocolate' }),
    1: Delta({ v: 'nothing' })
  };
  var bandit = makeBanditPOMDP({
    numberOfArms: 2,
    armToPrizeDist: trueArmToPrizeDist,
    numberOfTrials: numberOfTrials
  });

  var startState = bandit.startState;
  var alternativeArmToPrizeDist = extend(trueArmToPrizeDist,
                                         { 1: Delta({ v: 'champagne' }) });
  var alternativeStartState = makeBanditStartState(numberOfTrials,
                                                   alternativeArmToPrizeDist);

  var priorAgentPrior = Delta({
    v: Categorical({
      vs: [startState, alternativeStartState],
      ps: [0.7, 0.3]
    })
  });

  var priorPrizeToUtility = Infer({ model() {
    return {
      chocolate: uniformDraw(_.range(20).concat(25)),
      nothing: 0,
      champagne: 20
    };
  }});

  var priorMyopia =  (
    useOptimalModel ?
    Delta({ v: { on: false, bound:0 }}) :
    Infer({ model() {
      return {
        bound: categorical([.4, .2, .1, .1, .1, .1],
                           [1, 2, 3, 4, 6, 10])
      };
    }}));

  var prior = { priorAgentPrior, priorPrizeToUtility, priorMyopia };

  var baseAgentParams = {
    alpha: 1000,
    sophisticatedOrNaive: 'naive',
    discount: 0,
    noDelays: useOptimalModel
  };

  var observations = [[startState, 0]];

  var outputDist = inferBandit(bandit, baseAgentParams, prior, observations,
                               'offPolicy', 0, 'beliefDelay');

  var marginalChocolate = Infer({ model() {
    return sample(outputDist).prizeToUtility.chocolate;
  }});

  return [
    expectation(marginalChocolate),
    expectation(getMarginal(outputDist, 'myopiaBound'))
  ];
};

print('Prior expected utility for arm0 (chocolate): ' +
      listMean(_.range(20).concat(25)) );

displayExpectations(getPosterior);
~~~~

The graphs show that as the agent's time horizon increases the inferences of the two models diverge. For the Optimal agent, the longer time horizon makes exploration more valuable. So the Optimal model infers a higher utility for the known option as the time horizon increases. By contrast, the Possibly Reward-myopic model can explain away the lack of exploration by the agent being Reward-myopic. This latter model infers slightly lower values for $$C_g$$ as the horizon increases.

>**Exercise**: Suppose that instead of allowing the agent to be greedy, we allowed the agent to be a hyperbolic discounter. Think about how this would affect inferences from the observations above and for other sequences of observation. Change the code above to test out your predictions.

Next chapter: [Joint inference of biases and preferences II](/chapters/5e-joint-inference.html)

### Footnotes


8-guide-library.md

---
layout: chapter
title: Quick-start guide to the webppl-agents library
description: Create your own MDPs and POMDPs. Create gridworlds and k-armed bandits. Use agents from the library and create your own.
is_section: true
---

### Contents

1. Introduction

2. Creating MDPs

3. Creating Gridworld MDPs

4. Creating your own agents

5. Creating POMDPs

6. Creating k-armed bandits

### Introduction

This is a quick-start guide to using the `webppl-agents` library. For a comprehensive explanation of the ideas behind the library (e.g. MDPs, POMDPs, hyperbolic discounting) and diverse examples of its use, go to the online textbook [agentmodels.org](http://agentmodels.org).

The webppl-agents library is built around two basic entities: *agents* and *environments*. These entities are combined by *simulating* an agent interacting with a particular environment. The library includes two standard RL environments as examples (Gridworld and Multi-armed Bandits). Four kinds of agent are included. Many combinations of environment and agent are possible. In addition, it's easy to add your own environments and agents -- as we illustrate below.

Not all environments and agents can be combined. Among environments, we distinguish MDPs (Markov Decision Processes) and POMDPs (Partially Observable Markov Decision Processes). For a POMDP environment, the agent must be a "POMDP agent", which means they maintain a belief distribution on the state[^separation].

[^separation]: This separation of POMDPs and MDPs is not necessary from a theoretical perspective, since POMDPs generalize MDPs. However, the separation is convenient in practice; it allows the MDP code to be short and perspicuous and it provides performance advantages.

### Creating your own MDP environment

We begin by creating a very simple MDP environment and running two agents from the library on that environment.

MDPs are defined [here](http://agentmodels.org/chapters/3a-mdp.html). For use in the library, MDP environments are Javascript objects with the following methods:

>`{transition: ...,  stateToActions: ...}`

The `transition` method is a function from state-action pairs to states (as in the function $$T$$ in the MDP definition). The `stateToAction` method is a mapping from states to the actions that are allowed in that state. (This is often a constant function).

To run an agent on an MDP, the agent object must have a `utility` method defined on the MDP's state-action space. This method is the agent's "reward" or "utility" function (we use the terms interchangeably).

#### Creating the Line MDP environment
Our first MDP environment is a discrete line (or one-dimensional gridworld) where the agent can move left or right (starting from the origin). More precisely, the Line MDP is as follows:

- **States:** Points on the integer line (e.g ..., -1, 0, 1, 2, ...).

- **Actions/transitions:** Actions “left”, “right” and “stay” move the agent deterministically along the line in either direction. We represent the actions as $$[-1,0,1]$$ in the code below.

In our examples, the agent's `startState` is the origin. The utility is 1 at the origin, 3 at the third state right of the origin ("state 3"), and 0 otherwise.

The transition function must also decrement the time. States are objects with a `terminateAfterAction` property. In the example below, `terminateAfterAction` is set to `true` when the state's `timeLeft` attribute gets down to 1; this causes the MDP to terminate. Here is an example state for the Line MDP (it's also the `startState`):

>`{terminateAfterAction: false, timeLeft:5, loc:0}`

~~~~
// helper function that decrements time and triggers termination when
// time elapsed
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

// constructuor for the "line" MDP environment:
// argument *totalTime* is the time horizon
var makeLineMDP = function(totalTime) {

  var stateToActions = function(state) {
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var world = { stateToActions, transition };

  var startState = {
    timeLeft: totalTime,
    terminateAfterAction: false,
    loc: 0
  };

  var utility = function(state, action){
    var table = { 0: 1, 3: 3 };
    return table[state.loc] ? table[state.loc] : 0;
  };

  return { world, startState, utility };
};

// save the MDP constructor for use in other codeboxes
wpEditor.put('makeLineMDP', makeLineMDP);
~~~~

To run an agent on this MDP, we use a `makeAgent` constructor and the library function `simulateMDP`. The constructor for MDP agents is `makeMDPAgent`:

>`makeMDPAgent(params, world)`

Agent constructors always have these same two arguments. The `world` argument is required for the agent's internal simulations of possible transitions. The `params` argument specifies the agent's parameters and whether the agent is optimal or biased.

For an optimal agent, the parameters are:

>`{utility: <utility_function>,  alpha: <softmax_alpha>}`

An environment (or "world") and agent are combined with the `simulateMDP` function:

>`simulateMDP(startState, world, agent, outputType)`

Given the utility function defined above, the highest utility state is at location 3 (three steps to the right from the origin). So an optimal agent (who doesn't hyperbolically discount) will move to this location and stay there.

~~~~
///fold: helper function that decrements time and triggers termination when time elapsed
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

// constructuor for the "line" MDP environment:
// argument *totalTime* is the time horizon
var makeLineMDP = function(totalTime) {

  var stateToActions = function(state) {
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var world = { stateToActions, transition };

  var startState = {
    timeLeft: totalTime,
    terminateAfterAction: false,
    loc: 0
  };

  var utility = function(state, action){
    var table = { 0: 1, 3: 3 };
    return table[state.loc] ? table[state.loc] : 0;
  };

  return { world, startState, utility };
};
///

// Construct line MDP environment
var totalTime = 5;
var lineMDP = makeLineMDP(totalTime);
var world = lineMDP.world;

// The lineMDP object also includes a utility function and startState
var utility = lineMDP.utility;
var startState = lineMDP.startState;

// Construct MDP agent
var params = { alpha: 1000, utility };
var agent = makeMDPAgent(params, world);

// Simulate the agent on the lineMDP with *outputType* set to *states*
var trajectory = simulateMDP(startState, world, agent, 'states');

// Display start state
print(trajectory);
~~~~

We described the agent above as "optimal" because it does not hyperbolically discount and it is not myopic. However, we can adjust its "soft-max" noise by modifying the parameter `alpha` and induce sub-optimal behavior. Moreover, we can change the agent's behavior on this MDP by over-writing the utility method in `params`.

To construct a time-inconsistent, hyperbolically-discounting MDP agent, we include additional attributes in the `params` argument:

>`{ discount:<discount_parameter>, sophisticatedOrNaive: <boolean> }`

These attributes are explained in the [chapter](/chapters/5a-time-inconsistency.html) on hyperbolic discounting. The discounting agent stays at the origin because it isn't willing to "delay gratification" in order to get a larger total reward at location 3.

~~~~
///fold: helper function that decrements time and triggers termination when time elapsed
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

// constructuor for the "line" MDP environment:
// argument *totalTime* is the time horizon
var makeLineMDP = function(totalTime) {

  var stateToActions = function(state) {
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var world = { stateToActions, transition };

  var startState = {
    timeLeft: totalTime,
    terminateAfterAction: false,
    loc: 0
  };

  var utility = function(state, action){
    var table = { 0: 1, 3: 3 };
    return table[state.loc] ? table[state.loc] : 0;
  };

  return { world, startState, utility };
};
///

// Construct line MDP environment
var totalTime = 5;
var lineMDP = makeLineMDP(totalTime);
var world = lineMDP.world;

// The lineMDP object also includes a utility function and startState
var utility = lineMDP.utility;
var startState = lineMDP.startState;

// Construct hyperbolic agent
var params = {
  alpha: 1000,
  utility,
  discount: 2,
  sophisticatedOrNaive: 'naive'
};
var agent = makeMDPAgent(params, world);
var trajectory = simulateMDP(startState, world, agent, 'states');
print(trajectory);
~~~~

We've shown how to create your own MDP and then run different agents on that MDP. You can also create your own MDP agent, as we illustrate below.

>**Exercise:** Try some variations of the Line MDP by modifying the `transition` method in the `makeLineMDP` constructor above. For example, change the underlying graph structure from a line into a loop.

-----------

### Creating Gridworld MDPs

Gridworld is a standard toy environment for reinforcement learning problems. The library contains a constructor for making a gridworld with your choice of dimensions and reward function. There is also a function for displaying gridworlds in the browser.

We begin by creating a simple gridworld environment (using `makeGridWorldMDP`) and display it using `viz.gridworld`.

~~~~
// Create a constructor for our gridworld
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  var grid = [
    [___, ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0, 0],
  timeLeft: 10,
  terminateAfterAction: false
};

viz.gridworld(world, {trajectory: [startState]});
~~~~

Gridworld states have a `loc` attribute for the agent's location (using discrete Cartesian coordinates). The agent is able to move up, down, left and right but is not able to stay put.

Having created a gridworld, we construct a utility function (where utility depends only on the agent's grid location) and simulate an optimal MDP agent.

~~~~
///fold: Create a constructor for our gridworld
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  var grid = [
    [___, ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0,0],
  timeLeft: 10,
  terminateAfterAction: false
};
///

// `isEqual` is in *underscore* (included in webppl-agents)
var utility = function(state, action) {
  return _.isEqual(state.loc, [0, 3]) ? 1 : 0;
};

var params = { utility, alpha: 1000 };
var agent = makeMDPAgent(params, world);
var trajectory = simulateMDP(startState, world, agent);
viz.gridworld(world, {trajectory: trajectory});
~~~~

You can create terminal gridworld states by using features with a name. These named-features can also be used to create a utility function without specifying grid coordinates.

~~~~
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  // named features are terminal
  var G = { name: 'gold' };
  var S = { name: 'silver' };

  var grid = [
    [ G , ___, ___],
    [ S , ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0, 0],
  timeLeft: 10,
  terminateAfterAction: false
};

// The *makeUtilityFunction* method allows you to define
// a utility function in terms of named features
var makeUtilityFunction = simpleGridWorld.makeUtilityFunction;
var table = {
  gold: 2,
  silver: 1.8,
  timeCost: -0.5
};
var utility = makeUtilityFunction(table);

var params = { utility, alpha: 1000 };
var agent = makeMDPAgent(params, world);
var trajectory = simulateMDP(startState, world, agent);
viz.gridworld(world, { trajectory });
~~~~

There are many examples using gridworld in agentmodels.org, starting from this [chapter](/chapters/3b-mdp-gridworld.html).

-------

### Creating your own agents

As well as creating your own environments, it is straightfoward to create your own agents for MDPs and POMDPs. Much of agentmodels.org is a tutorial on creating agents (e.g. optimal agents, myopic agents, etc.). Rather than recapitulate agentmodels.org, this section is brief and focuses on the basic interface that agents need to present.

We begin by creating an agent that chooses actions uniformly at random. To run on agent on an environment using the `simulateMDP` function, an agent object must have an `act` method and a `params` attribute. The `act` method is a function from states to a distribution on the available actions. The `params` attribute indicates whether or not the agent is an MDP or POMDP agent.

We use the simple gridworld environment from the codebox above.

~~~~
///fold: Build gridworld environment
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  // named features are terminal
  var G = { name: 'gold' };
  var S = { name: 'silver' };

  var grid = [
    [ G , ___, ___],
    [ S , ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0, 0],
  timeLeft: 10,
  terminateAfterAction: false
};

// The *makeUtilityFunction* method allows you to define
// a utility function in terms of named features
var makeUtilityFunction = simpleGridWorld.makeUtilityFunction;
var table = {
  gold: 2,
  silver: 1.8,
  timeCost: -0.5
};
var utility = makeUtilityFunction(table);
///

var actions = ['u', 'd', 'l', 'r'];

var act = function(state){
  return Infer({ model(){ return uniformDraw(actions); }});
};

var randomAgent = { act, params: {} };
var trajectory = simulateMDP(startState, world, randomAgent);
viz.gridworld(world, { trajectory });
~~~~

In gridworld the same actions are available in each state. When the actions available depend on the state, the agent's `act` function needs access to the environment's `stateToActions` method.

~~~~
///fold: Create a constructor for our gridworld
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  // named features are terminal
  var G = { name: 'gold' };
  var S = { name: 'silver' };

  var grid = [
    [ G , ___, ___],
    [ S , ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0, 0],
  timeLeft: 10,
  terminateAfterAction: false
};

// The *makeUtilityFunction* method allows you to define
// a utility function in terms of named features
var makeUtilityFunction = simpleGridWorld.makeUtilityFunction;
var table = {
  gold: 2,
  silver: 1.8,
  timeCost: -0.5
};
var utility = makeUtilityFunction(table);
///

var makeRandomAgent = function(world) {
  var stateToActions = world.stateToActions;
  var act = function(state) {
    return Infer({ model() {
      return uniformDraw(stateToActions(state));
    }});
  };
  return { act, params: {} };
};

var randomAgent = makeRandomAgent(world);
var trajectory = simulateMDP(startState, world, randomAgent);

viz.gridworld(world, { trajectory });
~~~~

In the example above, the agent constructor `makeRandomAgent` takes the environment (`world`) as an argument in order to access `stateToActions`. Agent constructors will typically also use the environment's `transition` method to internally simulate state transitions.

>**Exercise:** Implement an agent who takes the action with highest expected utility under the random policy. (You can do this by making use of the codebox above. Use the `makeRandomAgent` and `simulateMDP` function within a new agent constructor.)

In addition to writing agents from scratch, you can build on the agents available in the library.

>**Exercise:** Start with the optimal MDP agent found [here](https://github.com/agentmodels/webppl-agents/blob/master/src/agents/makeMDPAgent.wppl#L3). Create a variant of this optimal agent that takes "epsilon-greedy" random actions instead of softmax random actions.

--------

### Creating POMDPs

POMDPs are introduced in agentmodels.org in this [chapter](/chapters/3c-pomdp.html). This section explains how to create your own POMDPs for use in the library.

As we explained above, MDPs in webppl-agents are objects with a `transition` method and a `stateToActions` method. POMDPs also have a `transition` method. Instead of `stateToActions`, they have a `beliefToActions` method, which maps a belief distribution over states to a set of available actions. POMDPs also have an `observe` method, which maps states to observations (typically represented as strings).

Here is a simple POMDP based on the "Line MDP" example above. The agent moves along the integer line as before. This time the agent is uncertain whether or not there is high reward at location 3. The agent can only find out by moving to location 3 and receiving an observation.

~~~~
// States have the same structure as in MDPs:
// the transition method needs to decrement
// the state's *timeLeft* attribute until termination

var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

var makeLinePOMDP = function() {

  var beliefToActions = function(belief){
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var observe = function(state) {
    if (state.loc == 3) {
      return state.treasureAt3 ? 'treasure' : 'no treasure';
    }
    return 'noObservation';
  };

  return { beliefToActions, transition, observe };

};
~~~~

To simulate an agent on this POMDP, we need to create a "POMDP" agent. POMDP agents have an `act` method which maps *beliefs* (rather than *states*) to distributions on actions. They also have an `updateBelief` method, mapping beliefs and observations to an updated belief.

This example uses the optimal POMDP agent. To construct a POMDP agent, we need to specify the agent's starting belief distribution on states. Here we assume the agent has a uniform distribution on whether or not there is "treasure" at location 3.

~~~~
///fold:
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

var makeLinePOMDP = function() {

  var beliefToActions = function(belief){
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var observe = function(state) {
    if (state.loc == 3) {
      return state.treasureAt3 ? 'treasure' : 'no treasure';
    }
    return 'noObservation';
  };

  return { beliefToActions, transition, observe };

};
///

var utility = function(state, action) {
  if (state.loc==3 && state.treasureAt3){ return 5; }
  if (state.loc==0){ return 1; }
  return 0;
};

var trueStartState = {
  timeLeft: 7,
  terminateAfterAction: false,
  loc: 0,
  treasureAt3: false
};

var alternativeStartState = extend(trueStartState, {treasureAt3: true});
var possibleStates = [trueStartState, alternativeStartState];

var priorBelief = Categorical({
  vs: possibleStates,
  ps: [.5, .5]
});

var params = {
  alpha: 1000,
  utility,
  priorBelief,
  optimal: true
};

var world = makeLinePOMDP();
var agent = makePOMDPAgent(params, world);
var trajectory = simulatePOMDP(trueStartState, world, agent, 'states');
print(trajectory);
~~~~

In POMDPs the agent does not directly observe their current state. However, in the Line POMDP (above) the "location" part of the agent's state is always known by the agent. The part of the state that is unknown is whether `treasureAt3` is true. So we could factor the state into attributes that are always known ("manifest") and parts that are not ("latent"). This factoring of the state can speed up the POMDP agent's belief-updating and is used for the POMDP environments in the library. The following codebox shows a factored version of the Line POMDP:

~~~~
///fold:
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};
///

var makeLinePOMDP = function() {
  var manifestStateToActions = function(manifestState){
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.manifestState.loc + action;
    var manifestStateNewLoc = extend(state.manifestState,{loc: newLoc});
    var newManifestState = advanceStateTime(manifestStateNewLoc);
    return {
      manifestState: newManifestState,
      latentState: state.latentState
    };
  };

  var observe = function(state) {
    if (state.manifestState.loc == 3){
      return state.latentState.treasureAt3 ? 'treasure' : 'no treasure';
    }
    return 'noObservation';
  };

  return { manifestStateToActions, transition, observe};
};

var utility = function(state, action) {
  if (state.manifestState.loc==3 && state.latentState.treasureAt3){ return 5; }
  if (state.manifestState.loc==0){ return 1; }
  return 0;
};

var trueStartState = {
  manifestState: {
    timeLeft: 7,
    terminateAfterAction: false,
    loc: 0
  },
  latentState: {
    treasureAt3: false
  }
};

var alternativeStartState = extend(trueStartState, {
  latentState: { treasureAt3: true }
});
var possibleStates = [trueStartState, alternativeStartState];

var priorBelief = Categorical({
  vs: possibleStates,
  ps: [.5, .5]
});

var params = {
  alpha: 1000,
  utility,
  priorBelief,
  optimal: true
};

var world = makeLinePOMDP();
var agent = makePOMDPAgent(params, world);
var trajectory = simulatePOMDP(trueStartState, world, agent, 'states');
print(trajectory);
~~~~

---------

### Footnotes


6-efficient-inference.md

---
layout: chapter
title: Efficient inference
description: Difficulty of inference, in particular for POMDPs and inverse planning. Outline of inference strategies.
status: stub
is_section: true
hidden: true
---