8-guide-library.md

---
layout: chapter
title: Quick-start guide to the webppl-agents library
description: Create your own MDPs and POMDPs. Create gridworlds and k-armed bandits. Use agents from the library and create your own.
is_section: true
---

### Contents

1. Introduction

2. Creating MDPs

3. Creating Gridworld MDPs

4. Creating your own agents

5. Creating POMDPs

6. Creating k-armed bandits

### Introduction

This is a quick-start guide to using the `webppl-agents` library. For a comprehensive explanation of the ideas behind the library (e.g. MDPs, POMDPs, hyperbolic discounting) and diverse examples of its use, go to the online textbook [agentmodels.org](http://agentmodels.org).

The webppl-agents library is built around two basic entities: *agents* and *environments*. These entities are combined by *simulating* an agent interacting with a particular environment. The library includes two standard RL environments as examples (Gridworld and Multi-armed Bandits). Four kinds of agent are included. Many combinations of environment and agent are possible. In addition, it's easy to add your own environments and agents -- as we illustrate below.

Not all environments and agents can be combined. Among environments, we distinguish MDPs (Markov Decision Processes) and POMDPs (Partially Observable Markov Decision Processes). For a POMDP environment, the agent must be a "POMDP agent", which means they maintain a belief distribution on the state[^separation].

[^separation]: This separation of POMDPs and MDPs is not necessary from a theoretical perspective, since POMDPs generalize MDPs. However, the separation is convenient in practice; it allows the MDP code to be short and perspicuous and it provides performance advantages.

### Creating your own MDP environment

We begin by creating a very simple MDP environment and running two agents from the library on that environment.

MDPs are defined [here](http://agentmodels.org/chapters/3a-mdp.html). For use in the library, MDP environments are Javascript objects with the following methods:

>`{transition: ...,  stateToActions: ...}`

The `transition` method is a function from state-action pairs to states (as in the function $$T$$ in the MDP definition). The `stateToAction` method is a mapping from states to the actions that are allowed in that state. (This is often a constant function).

To run an agent on an MDP, the agent object must have a `utility` method defined on the MDP's state-action space. This method is the agent's "reward" or "utility" function (we use the terms interchangeably).

#### Creating the Line MDP environment
Our first MDP environment is a discrete line (or one-dimensional gridworld) where the agent can move left or right (starting from the origin). More precisely, the Line MDP is as follows:

- **States:** Points on the integer line (e.g ..., -1, 0, 1, 2, ...).

- **Actions/transitions:** Actions “left”, “right” and “stay” move the agent deterministically along the line in either direction. We represent the actions as $$[-1,0,1]$$ in the code below.

In our examples, the agent's `startState` is the origin. The utility is 1 at the origin, 3 at the third state right of the origin ("state 3"), and 0 otherwise.

The transition function must also decrement the time. States are objects with a `terminateAfterAction` property. In the example below, `terminateAfterAction` is set to `true` when the state's `timeLeft` attribute gets down to 1; this causes the MDP to terminate. Here is an example state for the Line MDP (it's also the `startState`):

>`{terminateAfterAction: false, timeLeft:5, loc:0}`

~~~~
// helper function that decrements time and triggers termination when
// time elapsed
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

// constructuor for the "line" MDP environment:
// argument *totalTime* is the time horizon
var makeLineMDP = function(totalTime) {

  var stateToActions = function(state) {
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var world = { stateToActions, transition };

  var startState = {
    timeLeft: totalTime,
    terminateAfterAction: false,
    loc: 0
  };

  var utility = function(state, action){
    var table = { 0: 1, 3: 3 };
    return table[state.loc] ? table[state.loc] : 0;
  };

  return { world, startState, utility };
};

// save the MDP constructor for use in other codeboxes
wpEditor.put('makeLineMDP', makeLineMDP);
~~~~

To run an agent on this MDP, we use a `makeAgent` constructor and the library function `simulateMDP`. The constructor for MDP agents is `makeMDPAgent`:

>`makeMDPAgent(params, world)`

Agent constructors always have these same two arguments. The `world` argument is required for the agent's internal simulations of possible transitions. The `params` argument specifies the agent's parameters and whether the agent is optimal or biased.

For an optimal agent, the parameters are:

>`{utility: <utility_function>,  alpha: <softmax_alpha>}`

An environment (or "world") and agent are combined with the `simulateMDP` function:

>`simulateMDP(startState, world, agent, outputType)`

Given the utility function defined above, the highest utility state is at location 3 (three steps to the right from the origin). So an optimal agent (who doesn't hyperbolically discount) will move to this location and stay there.

~~~~
///fold: helper function that decrements time and triggers termination when time elapsed
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

// constructuor for the "line" MDP environment:
// argument *totalTime* is the time horizon
var makeLineMDP = function(totalTime) {

  var stateToActions = function(state) {
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var world = { stateToActions, transition };

  var startState = {
    timeLeft: totalTime,
    terminateAfterAction: false,
    loc: 0
  };

  var utility = function(state, action){
    var table = { 0: 1, 3: 3 };
    return table[state.loc] ? table[state.loc] : 0;
  };

  return { world, startState, utility };
};
///

// Construct line MDP environment
var totalTime = 5;
var lineMDP = makeLineMDP(totalTime);
var world = lineMDP.world;

// The lineMDP object also includes a utility function and startState
var utility = lineMDP.utility;
var startState = lineMDP.startState;

// Construct MDP agent
var params = { alpha: 1000, utility };
var agent = makeMDPAgent(params, world);

// Simulate the agent on the lineMDP with *outputType* set to *states*
var trajectory = simulateMDP(startState, world, agent, 'states');

// Display start state
print(trajectory);
~~~~

We described the agent above as "optimal" because it does not hyperbolically discount and it is not myopic. However, we can adjust its "soft-max" noise by modifying the parameter `alpha` and induce sub-optimal behavior. Moreover, we can change the agent's behavior on this MDP by over-writing the utility method in `params`.

To construct a time-inconsistent, hyperbolically-discounting MDP agent, we include additional attributes in the `params` argument:

>`{ discount:<discount_parameter>, sophisticatedOrNaive: <boolean> }`

These attributes are explained in the [chapter](/chapters/5a-time-inconsistency.html) on hyperbolic discounting. The discounting agent stays at the origin because it isn't willing to "delay gratification" in order to get a larger total reward at location 3.

~~~~
///fold: helper function that decrements time and triggers termination when time elapsed
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

// constructuor for the "line" MDP environment:
// argument *totalTime* is the time horizon
var makeLineMDP = function(totalTime) {

  var stateToActions = function(state) {
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var world = { stateToActions, transition };

  var startState = {
    timeLeft: totalTime,
    terminateAfterAction: false,
    loc: 0
  };

  var utility = function(state, action){
    var table = { 0: 1, 3: 3 };
    return table[state.loc] ? table[state.loc] : 0;
  };

  return { world, startState, utility };
};
///

// Construct line MDP environment
var totalTime = 5;
var lineMDP = makeLineMDP(totalTime);
var world = lineMDP.world;

// The lineMDP object also includes a utility function and startState
var utility = lineMDP.utility;
var startState = lineMDP.startState;

// Construct hyperbolic agent
var params = {
  alpha: 1000,
  utility,
  discount: 2,
  sophisticatedOrNaive: 'naive'
};
var agent = makeMDPAgent(params, world);
var trajectory = simulateMDP(startState, world, agent, 'states');
print(trajectory);
~~~~

We've shown how to create your own MDP and then run different agents on that MDP. You can also create your own MDP agent, as we illustrate below.

>**Exercise:** Try some variations of the Line MDP by modifying the `transition` method in the `makeLineMDP` constructor above. For example, change the underlying graph structure from a line into a loop.

-----------

### Creating Gridworld MDPs

Gridworld is a standard toy environment for reinforcement learning problems. The library contains a constructor for making a gridworld with your choice of dimensions and reward function. There is also a function for displaying gridworlds in the browser.

We begin by creating a simple gridworld environment (using `makeGridWorldMDP`) and display it using `viz.gridworld`.

~~~~
// Create a constructor for our gridworld
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  var grid = [
    [___, ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0, 0],
  timeLeft: 10,
  terminateAfterAction: false
};

viz.gridworld(world, {trajectory: [startState]});
~~~~

Gridworld states have a `loc` attribute for the agent's location (using discrete Cartesian coordinates). The agent is able to move up, down, left and right but is not able to stay put.

Having created a gridworld, we construct a utility function (where utility depends only on the agent's grid location) and simulate an optimal MDP agent.

~~~~
///fold: Create a constructor for our gridworld
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  var grid = [
    [___, ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0,0],
  timeLeft: 10,
  terminateAfterAction: false
};
///

// `isEqual` is in *underscore* (included in webppl-agents)
var utility = function(state, action) {
  return _.isEqual(state.loc, [0, 3]) ? 1 : 0;
};

var params = { utility, alpha: 1000 };
var agent = makeMDPAgent(params, world);
var trajectory = simulateMDP(startState, world, agent);
viz.gridworld(world, {trajectory: trajectory});
~~~~

You can create terminal gridworld states by using features with a name. These named-features can also be used to create a utility function without specifying grid coordinates.

~~~~
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  // named features are terminal
  var G = { name: 'gold' };
  var S = { name: 'silver' };

  var grid = [
    [ G , ___, ___],
    [ S , ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0, 0],
  timeLeft: 10,
  terminateAfterAction: false
};

// The *makeUtilityFunction* method allows you to define
// a utility function in terms of named features
var makeUtilityFunction = simpleGridWorld.makeUtilityFunction;
var table = {
  gold: 2,
  silver: 1.8,
  timeCost: -0.5
};
var utility = makeUtilityFunction(table);

var params = { utility, alpha: 1000 };
var agent = makeMDPAgent(params, world);
var trajectory = simulateMDP(startState, world, agent);
viz.gridworld(world, { trajectory });
~~~~

There are many examples using gridworld in agentmodels.org, starting from this [chapter](/chapters/3b-mdp-gridworld.html).

-------

### Creating your own agents

As well as creating your own environments, it is straightfoward to create your own agents for MDPs and POMDPs. Much of agentmodels.org is a tutorial on creating agents (e.g. optimal agents, myopic agents, etc.). Rather than recapitulate agentmodels.org, this section is brief and focuses on the basic interface that agents need to present.

We begin by creating an agent that chooses actions uniformly at random. To run on agent on an environment using the `simulateMDP` function, an agent object must have an `act` method and a `params` attribute. The `act` method is a function from states to a distribution on the available actions. The `params` attribute indicates whether or not the agent is an MDP or POMDP agent.

We use the simple gridworld environment from the codebox above.

~~~~
///fold: Build gridworld environment
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  // named features are terminal
  var G = { name: 'gold' };
  var S = { name: 'silver' };

  var grid = [
    [ G , ___, ___],
    [ S , ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0, 0],
  timeLeft: 10,
  terminateAfterAction: false
};

// The *makeUtilityFunction* method allows you to define
// a utility function in terms of named features
var makeUtilityFunction = simpleGridWorld.makeUtilityFunction;
var table = {
  gold: 2,
  silver: 1.8,
  timeCost: -0.5
};
var utility = makeUtilityFunction(table);
///

var actions = ['u', 'd', 'l', 'r'];

var act = function(state){
  return Infer({ model(){ return uniformDraw(actions); }});
};

var randomAgent = { act, params: {} };
var trajectory = simulateMDP(startState, world, randomAgent);
viz.gridworld(world, { trajectory });
~~~~

In gridworld the same actions are available in each state. When the actions available depend on the state, the agent's `act` function needs access to the environment's `stateToActions` method.

~~~~
///fold: Create a constructor for our gridworld
var makeSimpleGridWorld = function() {

  // '#' indicates a wall, and ' ' indicates a normal cell
  var ___ = ' ';

  // named features are terminal
  var G = { name: 'gold' };
  var S = { name: 'silver' };

  var grid = [
    [ G , ___, ___],
    [ S , ___, ___],
    ['#', '#', ___],
    ['#', '#', ___],
    [___, ___, ___]
  ];

  return makeGridWorldMDP({ grid, transitionNoiseProbability: 0 })
};

var simpleGridWorld = makeSimpleGridWorld();
var world = simpleGridWorld.world;

var startState = {
  loc: [0, 0],
  timeLeft: 10,
  terminateAfterAction: false
};

// The *makeUtilityFunction* method allows you to define
// a utility function in terms of named features
var makeUtilityFunction = simpleGridWorld.makeUtilityFunction;
var table = {
  gold: 2,
  silver: 1.8,
  timeCost: -0.5
};
var utility = makeUtilityFunction(table);
///

var makeRandomAgent = function(world) {
  var stateToActions = world.stateToActions;
  var act = function(state) {
    return Infer({ model() {
      return uniformDraw(stateToActions(state));
    }});
  };
  return { act, params: {} };
};

var randomAgent = makeRandomAgent(world);
var trajectory = simulateMDP(startState, world, randomAgent);

viz.gridworld(world, { trajectory });
~~~~

In the example above, the agent constructor `makeRandomAgent` takes the environment (`world`) as an argument in order to access `stateToActions`. Agent constructors will typically also use the environment's `transition` method to internally simulate state transitions.

>**Exercise:** Implement an agent who takes the action with highest expected utility under the random policy. (You can do this by making use of the codebox above. Use the `makeRandomAgent` and `simulateMDP` function within a new agent constructor.)

In addition to writing agents from scratch, you can build on the agents available in the library.

>**Exercise:** Start with the optimal MDP agent found [here](https://github.com/agentmodels/webppl-agents/blob/master/src/agents/makeMDPAgent.wppl#L3). Create a variant of this optimal agent that takes "epsilon-greedy" random actions instead of softmax random actions.

--------

### Creating POMDPs

POMDPs are introduced in agentmodels.org in this [chapter](/chapters/3c-pomdp.html). This section explains how to create your own POMDPs for use in the library.

As we explained above, MDPs in webppl-agents are objects with a `transition` method and a `stateToActions` method. POMDPs also have a `transition` method. Instead of `stateToActions`, they have a `beliefToActions` method, which maps a belief distribution over states to a set of available actions. POMDPs also have an `observe` method, which maps states to observations (typically represented as strings).

Here is a simple POMDP based on the "Line MDP" example above. The agent moves along the integer line as before. This time the agent is uncertain whether or not there is high reward at location 3. The agent can only find out by moving to location 3 and receiving an observation.

~~~~
// States have the same structure as in MDPs:
// the transition method needs to decrement
// the state's *timeLeft* attribute until termination

var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

var makeLinePOMDP = function() {

  var beliefToActions = function(belief){
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var observe = function(state) {
    if (state.loc == 3) {
      return state.treasureAt3 ? 'treasure' : 'no treasure';
    }
    return 'noObservation';
  };

  return { beliefToActions, transition, observe };

};
~~~~

To simulate an agent on this POMDP, we need to create a "POMDP" agent. POMDP agents have an `act` method which maps *beliefs* (rather than *states*) to distributions on actions. They also have an `updateBelief` method, mapping beliefs and observations to an updated belief.

This example uses the optimal POMDP agent. To construct a POMDP agent, we need to specify the agent's starting belief distribution on states. Here we assume the agent has a uniform distribution on whether or not there is "treasure" at location 3.

~~~~
///fold:
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};

var makeLinePOMDP = function() {

  var beliefToActions = function(belief){
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.loc + action;
    var stateNewLoc = extend(state, {loc: newLoc});
    return advanceStateTime(stateNewLoc);
  };

  var observe = function(state) {
    if (state.loc == 3) {
      return state.treasureAt3 ? 'treasure' : 'no treasure';
    }
    return 'noObservation';
  };

  return { beliefToActions, transition, observe };

};
///

var utility = function(state, action) {
  if (state.loc==3 && state.treasureAt3){ return 5; }
  if (state.loc==0){ return 1; }
  return 0;
};

var trueStartState = {
  timeLeft: 7,
  terminateAfterAction: false,
  loc: 0,
  treasureAt3: false
};

var alternativeStartState = extend(trueStartState, {treasureAt3: true});
var possibleStates = [trueStartState, alternativeStartState];

var priorBelief = Categorical({
  vs: possibleStates,
  ps: [.5, .5]
});

var params = {
  alpha: 1000,
  utility,
  priorBelief,
  optimal: true
};

var world = makeLinePOMDP();
var agent = makePOMDPAgent(params, world);
var trajectory = simulatePOMDP(trueStartState, world, agent, 'states');
print(trajectory);
~~~~

In POMDPs the agent does not directly observe their current state. However, in the Line POMDP (above) the "location" part of the agent's state is always known by the agent. The part of the state that is unknown is whether `treasureAt3` is true. So we could factor the state into attributes that are always known ("manifest") and parts that are not ("latent"). This factoring of the state can speed up the POMDP agent's belief-updating and is used for the POMDP environments in the library. The following codebox shows a factored version of the Line POMDP:

~~~~
///fold:
var advanceStateTime = function(state) {
  var newTimeLeft = state.timeLeft - 1;
  return extend(state, {
    timeLeft: newTimeLeft,
    terminateAfterAction: newTimeLeft > 1 ? state.terminateAfterAction : true
  });
};
///

var makeLinePOMDP = function() {
  var manifestStateToActions = function(manifestState){
    return [-1, 0, 1];
  };

  var transition = function(state, action) {
    var newLoc = state.manifestState.loc + action;
    var manifestStateNewLoc = extend(state.manifestState,{loc: newLoc});
    var newManifestState = advanceStateTime(manifestStateNewLoc);
    return {
      manifestState: newManifestState,
      latentState: state.latentState
    };
  };

  var observe = function(state) {
    if (state.manifestState.loc == 3){
      return state.latentState.treasureAt3 ? 'treasure' : 'no treasure';
    }
    return 'noObservation';
  };

  return { manifestStateToActions, transition, observe};
};

var utility = function(state, action) {
  if (state.manifestState.loc==3 && state.latentState.treasureAt3){ return 5; }
  if (state.manifestState.loc==0){ return 1; }
  return 0;
};

var trueStartState = {
  manifestState: {
    timeLeft: 7,
    terminateAfterAction: false,
    loc: 0
  },
  latentState: {
    treasureAt3: false
  }
};

var alternativeStartState = extend(trueStartState, {
  latentState: { treasureAt3: true }
});
var possibleStates = [trueStartState, alternativeStartState];

var priorBelief = Categorical({
  vs: possibleStates,
  ps: [.5, .5]
});

var params = {
  alpha: 1000,
  utility,
  priorBelief,
  optimal: true
};

var world = makeLinePOMDP();
var agent = makePOMDPAgent(params, world);
var trajectory = simulatePOMDP(trueStartState, world, agent, 'states');
print(trajectory);
~~~~

---------

### Footnotes


Modeling Agents with Probabilistic Programs

Owain Evans
Andreas Stuhlmüller
John Salvatier
Daniel Filan

2-webppl.md

---
layout: chapter
title: "Probabilistic programming in WebPPL"
description: "WebPPL is a functional subset of Javascript with automatic Bayesian inference via MCMC and gradient-based variational inference."
is_section: true
---

## Introduction

This chapter introduces the probabilistic programming language WebPPL (pronounced "web people"). The models for agents in this tutorial are all implemented in WebPPL and so it's important to understand how the language works.

We begin with a quick overview of probabilistic programming. If you are new to probabilistic programming, you might want to read an informal introduction (e.g. [here](http://www.pl-enthusiast.net/2014/09/08/probabilistic-programming/) or [here](https://moalquraishi.wordpress.com/2015/03/29/the-state-of-probabilistic-programming/)) or a more technical [survey](https://scholar.google.com/scholar?cluster=16211748064980449900&hl=en&as_sdt=0,5). For a practical introduction to both probabilistic programming and Bayesian modeling, we highly recommend [ProbMods](http://probmods.org), which also uses the WebPPL language.

The only requirement to run the code for this tutorial is a modern browser (e.g. Chrome, Firefox, Safari). If you want to explore the models in detail and to create your own, we recommend running WebPPL from the command line. Installation is simple and is explained [here](http://webppl.org).

## WebPPL: a purely functional subset of Javascript

WebPPL includes a subset of Javascript, and follows the syntax of Javascript for this subset.

This example program uses most of the Javascript syntax that is available in WebPPL:

~~~~
// Define a function using two external primitives:
// 1. Javascript's `JSON.stringify` for converting to strings
// 2. Underscore's _.isFinite for checking if a value is a finite number
var coerceToPositiveNumber = function(x) {
  if (_.isFinite(x) && x > 0) {
    return x;
  } else {
    print('- Input ' + JSON.stringify(x) +
          ' was not a positive number, returning 1 instead');
    return 1;
  }
};

// Create an array with numbers, an object, an a Boolean
var inputs = [2, 3.5, -1, { key: 1 }, true];

// Map the function over the array
print('Processing elements in array ' + JSON.stringify(inputs) + '...');
var result = map(coerceToPositiveNumber, inputs);
print('Result: ' + JSON.stringify(result));
~~~~

Language features with side effects are not allowed in WebPPL. The code that has been commented out uses assignment to update a table. This produces an error in WebPPL.

~~~~
// Don't do this:

// var table = {};
// table.key = 1;
// table.key = table.key + 1;
// => Syntax error: You tried to assign to a field of table, but you can
//                  only assign to fields of globalStore

// Instead do this:

var table = { key: 1 };
var tableTwo = { key: table.key + 1 };
print(tableTwo);

// Or use the library function `extend`:

var tableThree = extend(tableTwo, { key: 3 })
print(tableThree);
~~~~

There are no `for` or `while` loops. Instead, use higher-order functions like WebPPL's built-in `map`, `filter` and `zip`:

~~~~
var xs = [1, 2, 3];

// Don't do this:

// for (var i = 0; i < xs.length; i++){
//   print(xs[i]);
// }

// Instead of for-loop, use `map`:
map(print, xs);

"Done!"
~~~~

It is possible to use normal Javascript functions (which make *internal* use of side effects) in WebPPL. See the [online book](http://dippl.org/chapters/02-webppl.html) on the implementation of WebPPL for details (section "Using Javascript Libraries").

## WebPPL stochastic primitives

### Sampling from random variables

WebPPL has a large [library](http://docs.webppl.org/en/master/distributions.html) of primitive probability distributions. Try clicking "Run" repeatedly to get different i.i.d. random samples:

~~~~
print('Fair coins (Bernoulli distribution):');
print([flip(0.5), flip(0.5), flip(0.5)]);

print('Biased coins (Bernoulli distribution):');
print([flip(0.9), flip(0.9), flip(0.9)]);

var coinWithSide = function(){
  return categorical([.45, .45, .1], ['heads', 'tails', 'side']);
};

print('Coins that can land on their edge:')
print(repeat(5, coinWithSide)); // draw 5 i.i.d samples
~~~~

There are also continuous random variables:

~~~~
print('Two samples from standard Gaussian in 1D: ');
print([gaussian(0, 1), gaussian(0, 1)]);

print('A single sample from a 2D Gaussian: ');
print(multivariateGaussian(Vector([0, 0]), Matrix([[1, 0], [0, 10]])));
~~~~

You can write your own functions to sample from more complex distributions. This example uses recursion to define a sampler for the Geometric distribution:

~~~~
var geometric = function(p) {
  return flip(p) ? 1 + geometric(p) : 1
};

geometric(0.8);
~~~~

What makes WebPPL different from conventional programming languages is its ability to perform *inference* operations using these primitive probability distributions. Distribution objects in WebPPL have two key features:

1. You can draw *random i.i.d. samples* from a distribution using the special function `sample`. That is, you sample $$x \sim P$$ where $$P(x)$$ is the distribution.

2. You can compute the probability (or density) the distribution assigns to a value. That is, to compute $$\log(P(x))$$, you use `dist.score(x)`, where `dist` is the distribution in WebPPL.

The functions above that generate random samples are defined in the WebPPL library in terms of primitive distributions (e.g. `Bernoulli` for `flip` and `Gaussian` for `gaussian`) and the built-in function `sample`:

~~~~
var flip = function(p) {
  var p = (p !== undefined) ? p : 0.5;
  return sample(Bernoulli({ p }));
};

var gaussian = function(mu, sigma) {
  return sample(Gaussian({ mu, sigma }));
};

[flip(), gaussian(1, 1)];
~~~~

To create a new distribution, we pass a (potentially stochastic) function with no arguments---a *thunk*---to the function `Infer` that performs *marginalization*. For example, we can use `flip` as an ingredient to construct a Binomial distribution using enumeration:

~~~~
var binomial = function() {
  var a = flip(0.5);
  var b = flip(0.5);
  var c = flip(0.5);
  return a + b + c;
};

var MyBinomial = Infer({ model: binomial });

[sample(MyBinomial), sample(MyBinomial), sample(MyBinomial)];
~~~~

`Infer` is the *inference operator* that computes (or estimates) the marginal probability of each possible output of the function `binomial`. If no explicit inference method is specified, `Infer` defaults to enumerating each possible value of each random variable in the function body.

### Bayesian inference by conditioning

The most important use of inference methods is for Bayesian inference. Here, our task is to *infer* the value of some unknown parameter by observing data that depends on the parameter. For example, if flipping three separate coins produce exactly two Heads, what is the probability that the first coin landed Heads? To solve this in WebPPL, we can use `Infer` to enumerate all values for the random variables `a`, `b` and `c`. We use `condition` to constrain the sum of the variables. The result is a distribution representing the posterior distribution on the first variable `a` having value `true` (i.e. "Heads").

~~~~
var twoHeads = Infer({
  model() {
    var a = flip(0.5);
    var b = flip(0.5);
    var c = flip(0.5);
    condition(a + b + c === 2);
    return a;
  }
});

print('Probability of first coin being Heads (given exactly two Heads) : ');
print(Math.exp(twoHeads.score(true)));

var moreThanTwoHeads = Infer({
  model() {
    var a = flip(0.5);
    var b = flip(0.5);
    var c = flip(0.5);
    condition(a + b + c >= 2);
    return a;
  }
});

print('\Probability of first coin being Heads (given at least two Heads): ');
print(Math.exp(moreThanTwoHeads.score(true)));
~~~~

### Codeboxes and Plotting

The codeboxes allow you to modify our examples and to write your own WebPPL code. Code is not shared between boxes. You can use the special function `viz` to plot distributions:

~~~~
var appleOrangeDist = Infer({
  model() {
    return flip(0.9) ? 'apple' : 'orange';
  }
});

viz(appleOrangeDist);
~~~~

~~~~
var fruitTasteDist = Infer({
  model() {
    return {
      fruit: categorical([0.3, 0.3, 0.4], ['apple', 'banana', 'orange']),
      tasty: flip(0.7)
    };
  }
});

viz(fruitTasteDist);
~~~~

~~~~
var positionDist = Infer({
  model() {
    return {
      X: gaussian(0, 1),
      Y: gaussian(0, 1)};
  },
  method: 'forward',
  samples: 1000
});

viz(positionDist);
~~~~

### Next

In the [next chapter](/chapters/3-agents-as-programs.html), we will implement rational decision-making using inference functions.


4-reasoning-about-agents.md

---
layout: chapter
title: Reasoning about agents
description: Overview of Inverse Reinforcement Learning. Inferring utilities and beliefs from choices in Gridworld and Bandits.
is_section: true
---

## Introduction
The previous chapters have shown how to compute optimal actions for agents in MDPs and POMDPs. In many practical applications, this is the goal. For example, when controlling a robot, the goal is for the robot to act optimally given its utility function. When playing the stock market or poker, the goal is make money and one might use an approach based on the POMDP agent model from the [previous chapter](/chapters/3c-pomdp).

In other settings, however, the goal is to *learn* or *reason about* an agent based on their behavior. For example, in social science or psychology researchers often seek to learn about people's preferences (denoted $$U$$) and beliefs (denoted $$b$$). The relevant *data* (denoted $$\{a_i\}$$) are usually observations of human actions. In this situation, models of optimal action can be used as *generative models* of human actions. The generative model predicts the behavior *given* preferences and beliefs. That is:

$$
P( \{a_i\} \vert U, b)  =: \text{Generative model of optimal action}
$$

Statistical inference infers the preferences $$U$$ and beliefs $$b$$ *given* the observed actions $$\{a_i\}$$. That is:

$$
P( U, b \vert \{a_i\})  =: \text{Invert generative model via statistical inference}
$$

This approach, using generative models of sequential decision making, has been used to learn preferences and beliefs about education, work, health, and many other topics[^generative].

[^generative]: The approach in economics closest to the one we outline here (with models of action based on sequential decision making) is called "Structural Estimation". Some particular examples are reft:aguirregabiria2010dynamic and reft:darden2010smoking. A related piece of work in AI or computational social science is reft:ermon2014learning.

Agent models are also used as generative models in Machine Learning, under the label "Inverse Reinforcement Learning" (IRL). One motivation for learning human preferences and beliefs is to give humans helpful recommendations (e.g. for products they are likely to enjoy). A different goal is to build systems that mimic human expert performance. For some tasks, it is hard for humans to directly specify a utility/reward function that is both correct and that can be tractably optimized. An alternative is to *learn* the human's utility function by watching them perform the task. Once learned, the system can use standard RL techniques to optimize the function. This has been applied to building systems to park cars, to fly helicopters, to control human-like bots in videogames, and to play table tennis[^inverse].

[^inverse]: The relevant papers on applications of IRL: parking cars in reft:abbeel2008apprenticeship, flying helicopters in reft:abbeel2010autonomous, controlling videogame bots in reft:lee2010learning, and table tennis in reft:muelling2014learning.

This chapter provides an array of illustrative examples of learning about agents from their actions. We begin with a concrete example and then provide a general formalization of the inference problem. A virtue of using WebPPL is that doing inference over our existing agent models requires very little extra code.

## Learning about an agent from their actions: motivating example

Consider the MDP version of Bob's Restaurant Choice problem. Bob is choosing between restaurants, all restaurants are open (and Bob knows this), and Bob also knows the street layout. Previously, we discussed how to compute optimal behavior *given* Bob's utility function over restaurants. Now we infer Bob's utility function *given* observations of the behavior in the codebox:

~~~~
///fold: restaurant constants, donutSouthTrajectory
var ___ = ' ';
var DN = { name: 'Donut N' };
var DS = { name: 'Donut S' };
var V = { name: 'Veg' };
var N = { name: 'Noodle' };

var donutSouthTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"l"],
  [{"loc":[2,1],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"l"],
  [{"loc":[1,1],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[2,1]},"l"],
  [{"loc":[0,1],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[1,1]},"d"],
  [{"loc":[0,0],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[0,1],"timeAtRestaurant":0},"l"],
  [{"loc":[0,0],"terminateAfterAction":true,"timeLeft":7,"previousLoc":[0,0],"timeAtRestaurant":1},"l"]
];
///

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var world = makeGridWorldMDP({ grid, start: [3, 1] }).world;

viz.gridworld(world, { trajectory: donutSouthTrajectory });
~~~~

From Bob's actions, we infer that he probably prefers the Donut Store to the other restaurants. An alternative explanation is that Bob cares most about saving time. He might prefer Veg (the Vegetarian Cafe) but his preference is not strong enough to spend extra time getting there.

In this first example of inference, Bob's preference for saving time is held fixed and we infer (given the actions shown above) Bob's preference for the different restaurants. We model Bob using the MDP agent model from [Chapter 3.1](/chapters/3a-mdp.html). We place a uniform prior over three possible utility functions for Bob: one favoring Donut, one favoring Veg and one favoring Noodle. We compute a Bayesian posterior over these utility functions *given* Bob's observed behavior. Since the world is practically deterministic (with softmax parameter $$\alpha$$ set high), we just compare Bob's predicted states under each utility function to the states actually observed. To predict Bob's states for each utility function, we use the function `simulate` from [Chapter 3.1](/chapters/3a-mdp.html).

~~~~
///fold: restaurant constants, donutSouthTrajectory
var ___ = ' ';
var DN = { name: 'Donut N' };
var DS = { name: 'Donut S' };
var V = { name: 'Veg' };
var N = { name: 'Noodle' };

var donutSouthTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"l"],
  [{"loc":[2,1],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"l"],
  [{"loc":[1,1],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[2,1]},"l"],
  [{"loc":[0,1],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[1,1]},"d"],
  [{"loc":[0,0],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[0,1],"timeAtRestaurant":0},"l"],
  [{"loc":[0,0],"terminateAfterAction":true,"timeLeft":7,"previousLoc":[0,0],"timeAtRestaurant":1},"l"]
];
///

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});
var makeUtilityFunction = mdp.makeUtilityFunction;
var world = mdp.world;

var startState = donutSouthTrajectory[0][0];

var utilityTablePrior = function() {
  var baseUtilityTable = {
    'Donut S': 1,
    'Donut N': 1,
    'Veg': 1,
    'Noodle': 1,
    'timeCost': -0.04
  };
  return uniformDraw(
    [{ table: extend(baseUtilityTable, { 'Donut N': 2, 'Donut S': 2 }),
       favourite: 'donut' },
     { table: extend(baseUtilityTable, { Veg: 2 }),
       favourite: 'veg' },
     { table: extend(baseUtilityTable, { Noodle: 2 }),
       favourite: 'noodle' }]
  );
};

var posterior = Infer({ model() {
  var utilityTableAndFavourite = utilityTablePrior();
  var utilityTable = utilityTableAndFavourite.table;
  var favourite = utilityTableAndFavourite.favourite;

  var utility = makeUtilityFunction(utilityTable);
  var params = {
    utility,
    alpha: 2
  };
  var agent = makeMDPAgent(params, world);

  var predictedStateAction = simulateMDP(startState, world, agent, 'stateAction');
  condition(_.isEqual(donutSouthTrajectory, predictedStateAction));
  return { favourite };
}});

viz(posterior);
~~~~

## Learning about an agent from their actions: formalization

We will now formalize the kind of inference in the previous example. We begin by considering inference over the utilities and softmax noise parameter for an MDP agent. Later on we'll generalize to POMDP agents and to other agents.

Following [Chapter 3.1](/chapters/3a-mdp.html) the MDP agent is defined by a utility function $$U$$ and softmax parameter $$\alpha$$. In order to do inference, we need to know the agent's starting state $$s_0$$ (which might include both their *location* and their *time horizon* $$N$$). The data we condition on is a sequence of state-action pairs:

$$
(s_0, a_0), (s_1, a_1), \ldots, (s_n, a_n)
$$

The index for the final timestep is less than or equal to the time horzion:  $$n \leq N$$. We abbreviate this sequence as $$(s,a)_{0:n}$$. The joint posterior on the agent's utilities and noise given the observed state-action sequence is:

$$
P(U,\alpha | (s,a)_{0:n}) \propto P( {(s,a)}_{0:n} | U, \alpha) P(U, \alpha)
$$

where the likelihood function $$P( {(s,a)}_{0:n} \vert U, \alpha )$$ is the MDP agent model (for simplicity we omit information about the starting state). Due to the Markov Assumption for MDPs, the probability of an agent's action in a state is independent of the agent's previous or later actions (given $$U$$ and $$\alpha$$). This allows us to rewrite the posterior as **Equation (1)**:

$$
P(U,\alpha | (s,a)_{0:n}) \propto P(U, \alpha) \prod_{i=0}^n P( a_i | s_i, U, \alpha)
$$

The term $$P( a_i \vert s_i, U, \alpha)$$ can be rewritten as the softmax choice function (which corresponds to the function `act` in our MDP agent models). This equation holds for the case where we observe a sequence of actions from timestep $$0$$ to $$n \leq N$$ (with no gaps). This tutorial focuses mostly on this case. It is trivial to extend the equation to observing multiple independently drawn such sequences (as we show below). However, if there are gaps in the sequence or if we observe only the agent's states (not the actions), then we need to marginalize over actions that were unobserved.

## Examples of learning about agents in MDPs

### Example: Inference from part of a sequence of actions

The expression for the joint posterior (above) shows that it is straightforward to do inference on a part of an agent's action sequence. For example, if we know an agent had a time horizon $$N=11$$, we can do inference from only the agent's first few actions.

For this example we condition on the agent making a single step from $$[3,1]$$ to $$[2,1]$$ by moving left. For an agent with low noise, this already provides very strong evidence about the agent's preferences -- not much is added by seeing the agent go all the way to Donut South.

<!-- show_single_step_trajectory -->
~~~~
///fold: restaurant constants
var ___ = ' ';
var DN = { name: 'Donut N' };
var DS = { name: 'Donut S' };
var V = { name: 'Veg' };
var N = { name: 'Noodle' };
///

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var world = makeGridWorldMDP({ grid }).world;

var trajectory = [
  {
    loc: [3, 1],
    timeLeft: 11,
    terminateAfterAction: false
  },
  {
    loc: [2, 1],
    timeLeft: 10,
    terminateAfterAction: false
  }
];

viz.gridworld(world, { trajectory });
~~~~

Our approach to inference is slightly different than in the example at the start of this chapter. The approach is a direct translation of the expression for the posterior in Equation (1) above. For each observed state-action pair, we compute the likelihood of the agent (with given $$U$$) choosing that action in the state. In contrast, the simple approach above becomes intractable for long, noisy action sequences -- as it will need to loop over all possible sequences.

<!-- infer_from_single_step_trajectory -->
~~~~
///fold: create restaurant choice MDP
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});
///

var world = mdp.world;
var makeUtilityFunction = mdp.makeUtilityFunction;

var utilityTablePrior = function(){
  var baseUtilityTable = {
    'Donut S': 1,
    'Donut N': 1,
    'Veg': 1,
    'Noodle': 1,
    'timeCost': -0.04
  };
  return uniformDraw(
    [{ table: extend(baseUtilityTable, { 'Donut N': 2, 'Donut S': 2 }),
       favourite: 'donut' },
     { table: extend(baseUtilityTable, { 'Veg': 2 }),
       favourite: 'veg' },
     { table: extend(baseUtilityTable, { 'Noodle': 2 }),
       favourite: 'noodle' }]
  );
};

var observedTrajectory = [[{
  loc: [3, 1],
  timeLeft: 11,
  terminateAfterAction: false
}, 'l']];

var posterior = Infer({ model() {
  var utilityTableAndFavourite = utilityTablePrior();
  var utilityTable = utilityTableAndFavourite.table;
  var utility = makeUtilityFunction(utilityTable);
  var favourite = utilityTableAndFavourite.favourite;

  var agent  = makeMDPAgent({ utility, alpha: 2 }, world);
  var act = agent.act;

  // For each observed state-action pair, factor on likelihood of action
  map(
    function(stateAction){
      var state = stateAction[0];
      var action = stateAction[1];
      observe(act(state), action);
    },
    observedTrajectory);

  return { favourite };
}});

viz(posterior);
~~~~

Note that utility functions where Veg or Noodle are most preferred have almost the same posterior probability. Since they had the same prior, this means that we haven't received evidence about which the agent prefers. Moreover, assuming the agent's `timeCost` is negligible, then no matter where the agent above starts out on the grid, they choose Donut North or South. So we never get any information about whether they prefer the Vegetarian Cafe or Noodle Shop!

Actually, this is not quite right. If we wait long enough, the agent's softmax noise would eventually reveal information about which was preferred. However, we still won't be able to *efficiently* learn the agent's preferences by repeatedly watching them choose from a random start point. If there is no softmax noise, then we can make the stronger claim that even in the limit of arbitrarily many repeated i.i.d. observations, the agent's preferences are not *identified* by draws from this space of scenarios.

Unidentifiability is a frequent problem when inferring an agent's beliefs or utilities from realistic datasets. First, agents with low noise reliably avoid inferior states (as in the present example) and so their actions provide little information about the relative utilities among the inferior states. Second, using richer agent models means there are more possible explanations of the same behavior. For example, agents with high softmax noise or with false beliefs might go to a restaurant even if they don't prefer it. One general approach to the problem of unidentifiability in IRL is **active learning**. Instead of passively observing the agent's actions, you select a sequence of environments that will be maximally informative about the agent's preferences. For recent work covering both the nature of unidentifiability in IRL as well as the active learning approach, see reft:amin2016towards.

### Example: Inferring The Cost of Time and Softmax Noise

The previous examples assumed that the agent's `timeCost` (the negative utility of each timestep before the agent reaches a restaurant) and the softmax $$\alpha$$ were known. We can modify the above example to include them in inference.

~~~~
// infer_utilities_timeCost_softmax_noise
///fold: create restaurant choice MDP, donutSouthTrajectory
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var donutSouthTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"l"],
  [{"loc":[2,1],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"l"],
  [{"loc":[1,1],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[2,1]},"l"],
  [{"loc":[0,1],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[1,1]},"d"],
  [{"loc":[0,0],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[0,1],"timeAtRestaurant":0},"l"],
  [{"loc":[0,0],"terminateAfterAction":true,"timeLeft":7,"previousLoc":[0,0],"timeAtRestaurant":1},"l"]
];

var vegDirectTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"u"],
  [{"loc":[3,6],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5]},"r"],
  [{"loc":[4,6],"terminateAfterAction":false,"timeLeft":5,"previousLoc":[3,6]},"u"],
  [{"loc":[4,7],"terminateAfterAction":false,"timeLeft":4,"previousLoc":[4,6],"timeAtRestaurant":0},"l"],
  [{"loc":[4,7],"terminateAfterAction":true,"timeLeft":4,"previousLoc":[4,7],"timeAtRestaurant":1},"l"]
];
///

var world = mdp.world;
var makeUtilityFunction = mdp.makeUtilityFunction;

// Priors

var utilityTablePrior = function() {
  var foodValues = [0, 1, 2];
  var timeCostValues = [-0.1, -0.3, -0.6];
  var donut = uniformDraw(foodValues);
  return {
    'Donut N': donut,
    'Donut S': donut,
    'Veg': uniformDraw(foodValues),
    'Noodle': uniformDraw(foodValues),
    'timeCost': uniformDraw(timeCostValues)
  };
};

var alphaPrior = function(){
  return uniformDraw([.1, 1, 10, 100]);
};

// Condition on observed trajectory

var posterior = function(observedTrajectory){
  return Infer({ model() {
    var utilityTable = utilityTablePrior();
    var alpha = alphaPrior();
    var params = {
      utility: makeUtilityFunction(utilityTable),
      alpha
    };
    var agent = makeMDPAgent(params, world);
    var act = agent.act;

    // For each observed state-action pair, factor on likelihood of action
    map(
      function(stateAction){
        var state = stateAction[0];
        var action = stateAction[1]
        observe(act(state), action);
      },
      observedTrajectory);

    // Compute whether Donut is preferred to Veg and Noodle
    var donut = utilityTable['Donut N'];
    var donutFavorite = (
      donut > utilityTable.Veg &&
      donut > utilityTable.Noodle);

    return {
      donutFavorite,
      alpha: alpha.toString(),
      timeCost: utilityTable.timeCost.toString()
    };
  }});
};

print('Prior:');
var prior = posterior([]);
viz.marginals(prior);

print('Conditioning on one action:');
var posterior = posterior(donutSouthTrajectory.slice(0, 1));
viz.marginals(posterior);
~~~~

<!-- TODO: plot prior and posterior on same axes -->

The posterior shows that taking a step towards Donut South can now be explained in terms of a high `timeCost`. If the agent has a low value for $$\alpha$$, this step to the left is fairly likely even if the agent prefers Noodle or Veg. So including softmax noise in the inference makes inferences about other parameters closer to the prior.

>**Exercise:** Suppose the agent is observed going all the way to Veg. What would the posteriors on $$\alpha$$ and `timeCost` look like? Check your answer by conditioning on the state-action sequence `vegDirectTrajectory`. You will need to modify other parts of the codebox above to make this work.

As we noted previously, it is simple to extend our approach to inference to conditioning on multiple sequences of actions. Consider the two sequences below:

<!-- display_multiple_trajectories -->
~~~~
///fold: make restaurant choice MDP, naiveTrajectory, donutSouthTrajectory
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var naiveTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"l"],
  [{"loc":[2,5],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5],"timeAtRestaurant":0},"l"],
  [{"loc":[2,5],"terminateAfterAction":true,"timeLeft":6,"previousLoc":[2,5],"timeAtRestaurant":1},"l"]
];

var donutSouthTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"l"],
  [{"loc":[2,1],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"l"],
  [{"loc":[1,1],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[2,1]},"l"],
  [{"loc":[0,1],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[1,1]},"d"],
  [{"loc":[0,0],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[0,1],"timeAtRestaurant":0},"l"],
  [{"loc":[0,0],"terminateAfterAction":true,"timeLeft":7,"previousLoc":[0,0],"timeAtRestaurant":1},"l"]
];
///

var world = mdp.world;;

map(function(trajectory) { viz.gridworld(world, { trajectory }); },
    [naiveTrajectory, donutSouthTrajectory]);
~~~~

To perform inference, we just condition on both sequences. (We use concatenation but we could have taken the union of all state-action pairs).

<!-- infer_from_multiple_trajectories -->
~~~~
///fold: World and agent are exactly as above

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var donutSouthTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"l"],
  [{"loc":[2,1],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"l"],
  [{"loc":[1,1],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[2,1]},"l"],
  [{"loc":[0,1],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[1,1]},"d"],
  [{"loc":[0,0],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[0,1],"timeAtRestaurant":0},"l"],
  [{"loc":[0,0],"terminateAfterAction":true,"timeLeft":7,"previousLoc":[0,0],"timeAtRestaurant":1},"l"]
];

var naiveTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"l"],
  [{"loc":[2,5],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5],"timeAtRestaurant":0},"l"],
  [{"loc":[2,5],"terminateAfterAction":true,"timeLeft":6,"previousLoc":[2,5],"timeAtRestaurant":1},"l"]
];

var world = mdp.world;
var makeUtilityFunction = mdp.makeUtilityFunction;

// Priors

var utilityTablePrior = function() {
  var foodValues = [0, 1, 2];
  var timeCostValues = [-0.1, -0.3, -0.6];
  var donut = uniformDraw(foodValues);
  return {
    'Donut N': donut,
    'Donut S': donut,
    'Veg': uniformDraw(foodValues),
    'Noodle': uniformDraw(foodValues),
    'timeCost': uniformDraw(timeCostValues)
  };
};

var alphaPrior = function(){
  return uniformDraw([.1, 1, 10, 100]);
};

// Condition on observed trajectory

var posterior = function(observedTrajectory){
  return Infer({ model() {
    var utilityTable = utilityTablePrior();
    var alpha = alphaPrior();
    var params = {
      utility: makeUtilityFunction(utilityTable),
      alpha
    };
    var agent = makeMDPAgent(params, world);
    var act = agent.act;

    // For each observed state-action pair, factor on likelihood of action
    map(
      function(stateAction){
        var state = stateAction[0];
        var action = stateAction[1]
        observe(act(state), action);
      },
      observedTrajectory);

    // Compute whether Donut is preferred to Veg and Noodle
    var donut = utilityTable['Donut N'];
    var donutFavorite = (
      donut > utilityTable.Veg &&
      donut > utilityTable.Noodle);

    return {
      donutFavorite,
      alpha: alpha.toString(),
      timeCost: utilityTable.timeCost.toString()
    };
  }});
};

///
print('Prior:');
var prior = posterior([]);
viz.marginals(prior);

print('Posterior');
var posterior = posterior(naiveTrajectory.concat(donutSouthTrajectory));
viz.marginals(posterior);
~~~~
<!-- TODO: plot prior and posterior on same axes -->

## Learning about agents in POMDPs

### Formalization

We can extend our approach to inference to deal with agents that solve POMDPs. One approach to inference is simply to generate full state-action sequences and compare them to the observed data. As we mentioned above, this approach becomes intractable in cases where noise (in transitions and actions) is high and sequences are long.

Instead, we extend the approach in Equation (1) above. The first thing to notice is that Equation (1) has to be amended for POMDPs. In an MDP, actions are conditionally independent given the agent's parameters $$U$$ and $$\alpha$$ and the state. For any pair of actions $$a_{i}$$ and $$a_j$$ and state $$s_i$$:

$$
P(a_i \vert a_j, s_i, U,\alpha) = P(a_i \vert s_i, U,\alpha)
$$

In a POMDP, actions are only rendered conditionally independent if we also condition on the agent's *belief*. So Equation (1) can only be extended to the case where we know the agent's belief at each timestep. This will be realistic in some applications and not others. It depends on whether the agent's *observations* are part of the data that is conditioned on. If so, the agent's belief can be computed at each timestep (assuming the agent's initial belief is known). If not, we have to marginalize over the possible observations, making for a more complex inference computation.

Here is the extension of Equation (1) to the POMDP case, where we assume access to the agent's observations. <a id="pomdpDefine"></a>Our goal is to compute a posterior on the parameters of the agent. These include $$U$$ and $$\alpha$$ as before but also the agent's initial belief $$b_0$$.

We observe a sequence of state-observation-action triples:

$$
(s_0,o_0,a_0), (s_1,o_1,a_1), \ldots, (s_n,o_n,a_n)
$$

The index for the final timestep is at most the time horzion:  $$n \leq N$$. The joint posterior on the agent's utilities and noise given the observed sequence is:

$$
P(U,\alpha, b_0 | (s,o,a)_{0:n}) \propto P( (s,o,a)_{0:n} | U, \alpha, b_0)P(U, \alpha, b_0)
$$

To produce a factorized form of this posterior analogous to Equation (1), we compute the sequence of agent beliefs. This is given by the recursive Bayesian belief update described in [Chapter 3.3](/chapters/3c-pomdp):

$$
b_i = b_{i-1} \vert s_i, o_i, a_{i-1}
$$

$$
b_i(s_i) \propto
O(s_i,a_{i-1},o_i)
\sum_{s_i \in S} { T(s_{i-1}, a_{i-1}, s_i) b_{i-1}(s_{i-1})}
$$

The posterior can thus be written as **Equation (2)**: <a id="pomdpInfer"></a>

$$
P(U, \alpha, b_0 | (s,o,a)_{0:n}) \propto P(U, \alpha, b_0) \prod_{i=0}^n P( a_i | s_i, b_i, U, \alpha)
$$

### Application: Bandits

To learn the preferences and beliefs of a POMDP agent we translate Equation (2) into WebPPL. In a later [chapter](/chapters/5e-joint-inference.html), we apply this to the Restaurant Choice problem. Here we focus on the Bandit problems introduced in the [previous chapter](/chapters/3c-pomdp).

In the Bandit problems there is an unknown mapping from arms to non-numeric prizes (or distributions on such prizes) and the agent has preferences over these prizes. The agent tries out arms to discover the mapping and exploits the most promising arms. In the *inverse* problem, we get to observe the agent's actions. Unlike the agent, we already know the mapping from arms to prizes. However, we don't know the agent's preferences or the agent's prior about the mapping[^bandit].

[^bandit]: If we did not know the mapping from arms to prizes, the inference problem would not change fundamentally. We get information about this mapping by observing the prizes the agent receives when pulling different arms.

Often the agent's choices admit of multiple explanations. Recall the deterministic example in the previous chapter when (according to the agent's belief) `arm0` had the prize "chocolate" and `arm1` had either "champagne" or "nothing" (see also Figure 2 below). Suppose we observe the agent chosing `arm0` on the first of five trials. If we don't know the agent's utilities or beliefs, then this choice could be explained by either:

(1). the agent's preference for chocolate over champagne, or

(2). the agent's belief that `arm1` is very likely (e.g. 95%) to yield the "nothing" prize deterministically

Given this choice by the agent, we won't be able to identify which of (1) and (2) is true because exploration becomes less valuable every trial (and there's only 5 trials total).

The codeboxes below implements this example. The translation of Equation (2) is in the function `factorSequence`. This function iterates through the observed state-observation-action triples, updating the agent's belief at each timestep. It interleaves conditioning on an action (via `factor`) with computing the sequence of belief functions $$b_i$$. The variable names correspond as follows:

- $$b_0$$ is `initialBelief` (an argument to `factorSequence`)

- $$s_i$$ is `state`

- $$b_i$$ is `nextBelief`

- $$a_i$$ is `observedAction`

~~~~
var inferBeliefsAndPreferences = function(baseAgentParams, priorPrizeToUtility,
                                          priorInitialBelief, bandit,
                                          observedSequence) {

  return Infer({ model() {

    // 1. Sample utilities
    var prizeToUtility = (priorPrizeToUtility ? sample(priorPrizeToUtility)
                          : undefined);

    // 2. Sample beliefs
    var initialBelief = sample(priorInitialBelief);

    // 3. Construct agent given utilities and beliefs
    var newAgentParams = extend(baseAgentParams, { priorBelief: initialBelief });
    var agent = makeBanditAgent(newAgentParams, bandit, 'belief', prizeToUtility);
    var agentAct = agent.act;
    var agentUpdateBelief = agent.updateBelief;

    // 4. Condition on observations
    var factorSequence = function(currentBelief, previousAction, timeIndex){
      if (timeIndex < observedSequence.length) {
        var state = observedSequence[timeIndex].state;
        var observation = observedSequence[timeIndex].observation;
        var nextBelief = agentUpdateBelief(currentBelief, observation, previousAction);
        var nextActionDist = agentAct(nextBelief);
        var observedAction = observedSequence[timeIndex].action;
        factor(nextActionDist.score(observedAction));
        factorSequence(nextBelief, observedAction, timeIndex + 1);
      }
    };
    factorSequence(initialBelief,'noAction', 0);

    return {
      prizeToUtility,
      priorBelief: initialBelief
    };
  }});
};
~~~~

We start with a very simple example. The agent is observed pulling `arm1` five times. The agent's prior is known and assigns equal weight to `arm1` yielding "champagne" and to it yielding "nothing". The true prize for `arm1` is "champagne" (see Figure 1).

<img src="/assets/img/4-irl-bandit-1.png" alt="diagram" style="width: 500px;"/>

> **Figure 1:** Bandit problem where agent's prior is known. (The true state has the bold outline).

From the observation, it's obvious that the agent prefers champagne. This is what we infer below:

~~~~
///fold: inferBeliefsAndPreferences, getMarginal
var inferBeliefsAndPreferences = function(baseAgentParams, priorPrizeToUtility,
                                          priorInitialBelief, bandit,
                                          observedSequence) {

  return Infer({ model() {

    // 1. Sample utilities
    var prizeToUtility = (priorPrizeToUtility ? sample(priorPrizeToUtility)
                          : undefined);

    // 2. Sample beliefs
    var initialBelief = sample(priorInitialBelief);

    // 3. Construct agent given utilities and beliefs
    var newAgentParams = extend(baseAgentParams, { priorBelief: initialBelief });
    var agent = makeBanditAgent(newAgentParams, bandit, 'belief', prizeToUtility);
    var agentAct = agent.act;
    var agentUpdateBelief = agent.updateBelief;

    // 4. Condition on observations
    var factorSequence = function(currentBelief, previousAction, timeIndex){
      if (timeIndex < observedSequence.length) {
        var state = observedSequence[timeIndex].state;
        var observation = observedSequence[timeIndex].observation;
        var nextBelief = agentUpdateBelief(currentBelief, observation, previousAction);
        var nextActionDist = agentAct(nextBelief);
        var observedAction = observedSequence[timeIndex].action;
        factor(nextActionDist.score(observedAction));
        factorSequence(nextBelief, observedAction, timeIndex + 1);
      }
    };
    factorSequence(initialBelief,'noAction', 0);

    return {
      prizeToUtility,
      priorBelief: initialBelief
    };
  }});
};

var getMarginal = function(dist, key){
  return Infer({ model() {
    return sample(dist)[key];
  }});
};
///
// true prizes for arms
var trueArmToPrizeDist = {
  0: Delta({ v: 'chocolate' }),
  1: Delta({ v: 'champagne' })
};
var bandit = makeBanditPOMDP({
  armToPrizeDist: trueArmToPrizeDist,
  numberOfArms: 2,
  numberOfTrials: 5
});

// simpleAgent always pulls arm 1
var simpleAgent = makePOMDPAgent({
  act: function(belief){
    return Infer({ model() { return 1; }});
  },
  updateBelief: function(belief){ return belief; },
  params: { priorBelief: Delta({ v: bandit.startState }) }
}, bandit.world);

var observedSequence = simulatePOMDP(bandit.startState, bandit.world, simpleAgent,
                                    'stateObservationAction');

// Priors for inference

// We know agent's prior, which is that either arm1 yields
// nothing or it yields champagne.
var priorInitialBelief = Delta({ v: Infer({ model() {
  var armToPrizeDist = uniformDraw([
    trueArmToPrizeDist,
    extend(trueArmToPrizeDist, { 1: Delta({ v: 'nothing' }) })]);
  return makeBanditStartState(5, armToPrizeDist);
}})});

// Agent either prefers chocolate or champagne.
var likesChampagne = {
  nothing: 0,
  champagne: 5,
  chocolate: 3
};
var likesChocolate = {
  nothing: 0,
  champagne: 3,
  chocolate: 5
};
var priorPrizeToUtility = Categorical({
  vs: [likesChampagne, likesChocolate],
  ps: [0.5, 0.5]
});
var baseParams = { alpha: 1000 };
var posterior = inferBeliefsAndPreferences(baseParams, priorPrizeToUtility,
                                           priorInitialBelief, bandit,
                                           observedSequence);

print("After observing agent choose arm1, what are agent's utilities?");
print('Posterior on agent utilities:');
viz.table(getMarginal(posterior, 'prizeToUtility'));
~~~~
<!-- TODO - make output above less ugly -->

In the codebox above, the agent's preferences are identified by the observations. This won't hold for the next example, which we introduced previously. The agent's utilities for prizes are still unknown and now the agent's prior is also unknown. Either the agent is "informed" and knows the truth that `arm1` yields "champagne". Or the agent is misinformed and believes `arm1` is likely to yield "nothing". These two possibilities are depicted in Figure 2.

<img src="/assets/img/4-irl-bandit-2.png" alt="diagram" style="width: 600px;"/>

> **Figure 2:** Bandit where agent's prior is unknown. The two large boxes depict the prior on the agent's initial belief. Each possibility for the agent's initial belief has probability 0.5.

We observe the agent's first action, which is pulling `arm0`. Our inference approach is the same as above:

~~~~
///fold:
var inferBeliefsAndPreferences = function(baseAgentParams, priorPrizeToUtility,
                                          priorInitialBelief, bandit,
                                          observedSequence) {

  return Infer({ model() {

    // 1. Sample utilities
    var prizeToUtility = (priorPrizeToUtility ? sample(priorPrizeToUtility)
                          : undefined);

    // 2. Sample beliefs
    var initialBelief = sample(priorInitialBelief);

    // 3. Construct agent given utilities and beliefs
    var newAgentParams = extend(baseAgentParams, { priorBelief: initialBelief });
    var agent = makeBanditAgent(newAgentParams, bandit, 'belief', prizeToUtility);
    var agentAct = agent.act;
    var agentUpdateBelief = agent.updateBelief;

    // 4. Condition on observations
    var factorSequence = function(currentBelief, previousAction, timeIndex){
      if (timeIndex < observedSequence.length) {
        var state = observedSequence[timeIndex].state;
        var observation = observedSequence[timeIndex].observation;
        var nextBelief = agentUpdateBelief(currentBelief, observation, previousAction);
        var nextActionDist = agentAct(nextBelief);
        var observedAction = observedSequence[timeIndex].action;
        factor(nextActionDist.score(observedAction));
        factorSequence(nextBelief, observedAction, timeIndex + 1);
      }
    };
    factorSequence(initialBelief,'noAction', 0);

    return {
      prizeToUtility,
      priorBelief: initialBelief
    };
  }});
};
///
var trueArmToPrizeDist = {
  0: Delta({ v: 'chocolate' }),
  1: Delta({ v: 'champagne' })
};
var bandit = makeBanditPOMDP({
  numberOfArms: 2,
  armToPrizeDist: trueArmToPrizeDist,
  numberOfTrials: 5
});

var simpleAgent = makePOMDPAgent({
  // simpleAgent always pulls arm 0
  act: function(belief){
    return Infer({ model() { return 0; }});
  },
  updateBelief: function(belief){ return belief; },
  params: { priorBelief: Delta({ v: bandit.startState }) }
}, bandit.world);

var observedSequence = simulatePOMDP(bandit.startState, bandit.world, simpleAgent,
                                     'stateObservationAction');

// Agent either knows that arm1 has prize "champagne"
// or agent thinks prize is probably "nothing"

var informedPrior = Delta({ v: bandit.startState });
var noChampagnePrior = Infer({ model() {
  var armToPrizeDist = categorical(
    [0.05, 0.95],
    [trueArmToPrizeDist,
     extend(trueArmToPrizeDist, { 1: Delta({ v: 'nothing' }) })]);
  return makeBanditStartState(5, armToPrizeDist);
}});

var priorInitialBelief = Categorical({
  vs: [informedPrior, noChampagnePrior],
  ps: [0.5, 0.5]
});

// We are still uncertain about whether agent prefers chocolate or champagne
var likesChampagne = {
  nothing: 0,
  champagne: 5,
  chocolate: 3
};
var likesChocolate = {
  nothing: 0,
  champagne: 3,
  chocolate: 5
};

var priorPrizeToUtility = Categorical({
  ps: [0.5, 0.5],
  vs: [likesChampagne, likesChocolate]
});

var baseParams = {alpha: 1000};
var posterior = inferBeliefsAndPreferences(baseParams, priorPrizeToUtility,
                                           priorInitialBelief, bandit,
                                           observedSequence);

var utilityBeliefPosterior = Infer({ model() {
  var utilityBelief = sample(posterior);
  var chocolateUtility = utilityBelief.prizeToUtility.chocolate;
  var likesChocolate = chocolateUtility > 3;
  var isInformed = utilityBelief.priorBelief.support().length === 1;
  return { likesChocolate, isInformed };
}});

viz.table(utilityBeliefPosterior);
~~~~

Exploration is more valuable if there are more Bandit trials in total. If we observe the agent choosing the arm they already know about (`arm0`) then we get stronger inferences about their preference for chocolate over champagne as the total trials increases.

~~~~
// TODO simplify the code here or merge with previous example.
///fold:
var inferBeliefsAndPreferences = function(baseAgentParams, priorPrizeToUtility,
                                           priorInitialBelief, bandit,
                                           observedSequence) {

  return Infer({ model() {

    // 1. Sample utilities
    var prizeToUtility = (priorPrizeToUtility ? sample(priorPrizeToUtility)
                          : undefined);

    // 2. Sample beliefs
    var initialBelief = sample(priorInitialBelief);

    // 3. Construct agent given utilities and beliefs
    var newAgentParams = extend(baseAgentParams, { priorBelief: initialBelief });
    var agent = makeBanditAgent(newAgentParams, bandit, 'belief', prizeToUtility);
    var agentAct = agent.act;
    var agentUpdateBelief = agent.updateBelief;

    // 4. Condition on observations
    var factorSequence = function(currentBelief, previousAction, timeIndex){
      if (timeIndex < observedSequence.length) {
        var state = observedSequence[timeIndex].state;
        var observation = observedSequence[timeIndex].observation;
        var nextBelief = agentUpdateBelief(currentBelief, observation, previousAction);
        var nextActionDist = agentAct(nextBelief);
        var observedAction = observedSequence[timeIndex].action;
        factor(nextActionDist.score(observedAction));
        factorSequence(nextBelief, observedAction, timeIndex + 1);
      }
    };
    factorSequence(initialBelief,'noAction', 0);

    return {
      prizeToUtility,
      priorBelief: initialBelief
    };
  }});
};
///

var probLikesChocolate = function(numberOfTrials){
  var trueArmToPrizeDist = {
    0: Delta({ v: 'chocolate' }),
    1: Delta({ v: 'champagne' })
  };
  var bandit = makeBanditPOMDP({
    numberOfArms: 2,
    armToPrizeDist: trueArmToPrizeDist,
    numberOfTrials
  });

  var simpleAgent = makePOMDPAgent({
    // simpleAgent always pulls arm 0
    act: function(belief){
      return Infer({ model() { return 0; }});
    },
    updateBelief: function(belief){ return belief; },
    params: { priorBelief: Delta({ v: bandit.startState }) }
  }, bandit.world);

  var observedSequence = simulatePOMDP(bandit.startState, bandit.world, simpleAgent,
                                       'stateObservationAction');

  var baseParams = { alpha: 100 };

  var noChampagnePrior = Infer({ model() {
    var armToPrizeDist = (
      flip(0.2) ?
      trueArmToPrizeDist :
      extend(trueArmToPrizeDist, { 1: Delta({ v: 'nothing' }) }));
    return makeBanditStartState(numberOfTrials, armToPrizeDist);
  }});
  var informedPrior = Delta({ v: bandit.startState });
  var priorInitialBelief = Categorical({
    vs: [noChampagnePrior, informedPrior],
    ps: [0.5, 0.5],
  });

  var likesChampagne = {
    nothing: 0,
    champagne: 5,
    chocolate: 3
  };
  var likesChocolate = {
    nothing: 0,
    champagne: 3,
    chocolate: 5
  };

  var priorPrizeToUtility = Categorical({
    vs: [likesChampagne, likesChocolate],
    ps: [0.5, 0.5],
  });

  var posterior = inferBeliefsAndPreferences(baseParams, priorPrizeToUtility,
                                             priorInitialBelief, bandit,
                                             observedSequence);

  var likesChocInformed = {
    prizeToUtility: likesChocolate,
    priorBelief: informedPrior
  };
  var probLikesChocInformed = Math.exp(posterior.score(likesChocInformed));
  var likesChocNoChampagne = {
    prizeToUtility: likesChocolate,
    priorBelief: noChampagnePrior
  };
  var probLikesChocNoChampagne = Math.exp(posterior.score(likesChocNoChampagne));
  return probLikesChocInformed + probLikesChocNoChampagne;
};

var lifetimes = [5, 6, 7, 8, 9];
var probsLikesChoc = map(probLikesChocolate, lifetimes);

print('Probability of liking chocolate for lifetimes ' + lifetimes + '\n'
      + probsLikesChoc);

viz.bar(lifetimes, probsLikesChoc);
~~~~

This example of inferring an agent's utilities from a Bandit problem may seem contrived. However, there are practical problems that have a similar structure. Consider a domain where $$k$$ **sources** (arms) produce a stream of content, with each piece of content having a **category** (prizes). At each timestep, a human is observed choosing a source. The human has uncertainty about the stochastic mapping from sources to categories. Our goal is to infer the human's beliefs about the sources and their preferences over categories. The sources could be blogs or feeds that tag posts using the same set of tags. Alternatively, the sources could be channels for TV shows or songs. In this kind of application, the same issue of identifiability arises. An agent may choose a source either because they know it produces content in the best categories or because they have a strong prior belief that it does.

In the next [chapter](/chapters/5-biases-intro.html), we start looking at agents with cognitive bounds and biases.

<br>

### Footnotes


6b-inference-sampling.md

---
layout: chapter
title: Sampling-based methods
description: A brief review of sampling-based methods. Discuss MCMC over enumeration with caching for inverse planning.
status: stub
is_section: false
hidden: true
---