I really like that this paper has experiments with real humans. It's definitely a problem that IRL assumes that the expert is (approximately) optimal -- this means that you can't learn where the expert is likely to be wrong, and so it is hard to exceed the expert's performance. It's very difficult to figure out how to deal with the possbility of a biased expert, and I'm happy to see work that takes a shot at it.

Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior

Siddharth Reddy, Anca D. Dragan, Sergey Levine

Inverse reinforcement learning algorithms typically assume that the demonstrations come from an expert who is approximately optimal. However, this is often not the case, at least when the experts are fallible humans. This paper considers the case where the expert has an incorrect model of the dynamics (transition function) of the environment, and proposes learning the expert's model of the dynamics to improve reward function inference. However, this leads to severe unidentifiability problems, where many models of the dynamics are compatible with the observed behavior. To overcome this, they assume that they have multiple tasks with known reward functions, which they use to infer the expert's dynamics. This is then used to infer the reward function in a new task using an adaptation of max causal entropy IRL. The dynamics can be an arbitrary neural net while the reward function is a weighted linear combination of features. They evaluate the inference of the dynamics model with real humans on Lunar Lander. Given transcripts of humans playing Lunar Lander, they infer the underlying (incorrect) dynamics model. Then, when the human takes an action, they predict which next state the human wanted to achieve, and replace the human's action with the action that would actually get close to the state the human wanted.


The standard inverse reinforcement learning assumption about humans acting optimally seems unrealistic; I think this paper provides an insightful initial step in not having that assumption and models the non-optimal version of the problem in a clean and compelling way. I think it's a noteworthy observation that this problem is very sensitive to the player's learning model, and I agree with the paper that this suggests that we should put effort into researching actual human learning strategies. I am unsure how to think about the insights here generalizing to other inverse reinforcement learning cases.

The Assistive Multi-Armed Bandit

Lawrence Chan, Dylan Hadfield-Menell, Siddhartha Srinivasa, Anca Dragan

Standard approaches for inverse reinforcement learning assume that humans are acting optimally according to their preferences, rather than learning about their preferences as time goes on. This paper tries to model the latter by introducing the _assistive multi-armed bandit_ problem.

In the standard _multi-armed bandit_ problem, a player repeatedly chooses one of several “arms” to pull, where each arm provides reward according to some unknown distribution. Imagine getting 1000 free plays on your choice of 10 different, unknown slot machines. This is a hard problem since the player must trade off between exploration (learning about some arm) and exploitation (pulling the best arm so far). In _assistive multi-armed bandit_, a robot is given the opportunity to intercept the player every round and pull an arm of its choice. If it does not intercept, it can see the arm pulled by the player but not the reward the player receives. This formalizes the notion of an AI with only partial information trying to help a learning agent optimize their reward.

The paper does some theoretical analysis of this problem as well as an experimental set-up involving a neural network and players acting according to a variety of different policies. It makes several observations about the problem:
- A player better at learning does not necessarily lead to the player-robot team performing better-- the robot can help a suboptimal player do better in accordance with how much information the player's arm pulls convey about the reward of the arm.
- A robot is best at assisting when it has the right model for how the player is learning.
- A robot that models the player as learning generally does better than a robot that does not, even if the robot has the wrong model for the player's learning.
- The problem is very sensitive to which learning model the player uses and which learning model the robot assumes. Some player learning models can only be effectively assisted when they are correctly modeled. Some robot-assumed learning models effectively assist for a variety of actual player learning models.


BAIR blog post: Cooperatively Learning Human Values

Cooperative Inverse Reinforcement Learning

Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart Russell

This paper provides a formalization of the three principles from the book, in the case where there is a single human H and a single robot R. H and R are trying to optimize the same reward function. Since both H and R are represented in the environment, it can be the _human's_ reward: that is, it is possible to reward the state where the human drinks coffee, without also rewarding the state where the robot drinks coffee. This corresponds to the first principle: that machines should optimize _our_ objectives. The second principle, that machines should initially be uncertain about our objectives, is incorporated by assuming that _only H knows the reward_, requiring R to maintain a belief over the reward. Finally, for the third principle, R needs to get information about the reward from H's behavior, and so R assumes that H will choose actions that best optimize the reward (taking into account the fact that R doesn't know the reward).

This defines a two-player game, originally called a CIRL game but now called an _assistance game_. We can compute optimal joint strategies for H and R. Since this is an _interactive_ process, H can do better than just acting optimally as if R did not exist (the assumption typically made in IRL): H can _teach_ R what the reward is. In addition, R does not simply passively listen and then act, but interleaves learning and acting, and so must manage the explore-exploit tradeoff.

See also <@Learning to Interactively Learn and Assist@>, which is inspired by this paper and does a similar thing with deep RL.


While I agree with the broad thrust of this paper, I do have one nitpick: the game ends immediately after H chooses whether or not to shut off R. In reality, if R isn't shut off, the assistance game will continue, which changes the incentives. If R can be relatively confident in the utility of _some_ action (e.g. doing nothing), then it may be a better plan for it to disable the shutdown button, and then take that action and observe H in the mean time to learn the reward. Then, after it has learned more about the reward and figured out why H wanted to shut it down, it can act well and get utility (rather than being stuck with the zero utility from being shut down). While this doesn't seem great, it's not _obviously_ bad: R ends up doing nothing until it can figure out how to actually be useful, hardly a catastrophic outcome. Really bad outcomes only come if R ends up becoming confident in the wrong reward due to some kind of misspecification, as suggested in [Incorrigibility in the CIRL Framework](http://arxiv.org/abs/1709.06275), summarized next.

The Off-Switch Game

This paper studies theoretically the impact of uncertainty over the reward on R's incentives around potential off switches. It proposes the simplest model that the authors expect to lead to generalizable results. R and H are in an assistance game, in which R goes first. R may either take an action a, getting utility u, or shut itself down, getting utility 0. In either case, the game ends immediately. Alternatively, R can choose to wait, in which case H can either shut down R, getting utility 0, or allow R to go ahead with action a, getting utility u.

If H is perfectly rational, then waiting is always an optimal action for R, since H will ensure that the team gets max(u, 0) utility. There can be other optimal actions: if R is sure that u >= 0, then taking action a is also optimal, and similarly if R is sure that u <= 0, then shutting down is also optimal. However, if H is not rational, and sometimes fails to take the utility-maximizing action (in a way R can't predict), then things get murkier. If R is sure about the value of u, then it is never optimal to wait, better to just take the action a (if u >= 0) or shut down (if u < 0) rather than let H screw it up. If R is pretty confident that u is positive, it may still decide to take action a, rather than risk that H makes the wrong decision. However, if R is very uncertain about the sign of u, then waiting becomes optimal again. In general, more uncertainty over the reward leads to more deferential behavior (allowing H to shut it off), but at a cost: R is much less able to help H when it is very uncertain about the reward.


I really like this paper as an example of how to apply the third principle. This was the paper that caused me to start thinking about how we should be thinking about the assumed vs. actual information content in things (here, the key insight is that RL typically assumes that the reward function conveys much more information than it actually does). That probably influenced the development of <@Preferences Implicit in the State of the World@>(@Learning Preferences by Looking at the World@), which is also an example of the third principle and this information-based viewpoint, as it argues that the state of the world is caused by human behavior and so contains information about human preferences.

It's worth noting that in this paper the lava avoidance is both due to the belief over the true reward, _and the risk aversion_. The agent would also avoid pots of gold in the test environment if it never saw it in the training environment. IRD only gives you the correct uncertainty over the true reward; it doesn't tell you how to use that uncertainty. You would still need safe exploration, or some other source of information, if you want to reduce the uncertainty.

Inverse Reward Design

Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, Anca Dragan

Usually, in RL, the reward function is treated as the _definition_ of optimal behavior, but this conflicts with the third principle, which says that human behavior is the ultimate source of information about human preferences. Nonetheless, reward functions clearly have some information about our preferences: how do we make it compatible with the third principle? We need to connect the reward function to human behavior somehow.

This paper proposes a simple answer: since reward designers usually make reward functions through a process of trial-and-error where they test their reward functions and see what they incentivize, the reward function _tells us about optimal behavior in the **training** environment(s)_. The authors formalize this using a Boltzmann rationality model, where the reward designer is more likely to pick a _proxy reward_ when it gives higher _true reward_ in the _training environment_ (but it doesn't matter if the proxy reward becomes decoupled from the true reward in some test environment). With this assumption connecting the human behavior (i.e. the proxy reward function) to the human preferences (i.e. the true reward function), they can then perform Bayesian inference to get a posterior distribution over the _true_ reward function.

They demonstrate that by using risk-averse planning with respect to this posterior distribution, the agent can avoid negative side effects that it has never seen before and has no information about. For example, if the agent was trained to collect gold in an environment with dirt and grass, and then it is tested in an environment with lava, the agent will know that even though the specified reward was indifferent about lava, this doesn't mean much, since _any_ weight on lava would have led to the same behavior in the training environment. Due to risk aversion, it conservatively assumes that the lava is bad, and so successfully avoids it.

See also <@Active Inverse Reward Design@>, which builds on this work.


Overall, I found the idea expressed in this paper to be well-articulated. While I think that the grid-world environment is a bit simplistic, their results are interesting. Being able to learn intent in an online manner is an important problem to solve if we’re interested in robust collaboration between humans and autonomous agents. However, the authors point out that training on pixel input fails in the majority of cases, 64% of the time, which raises concerns about how well the method would generalize to non-trivial environments.

**Rohin's opinion:** I'm excited that the ideas from [CIRL](https://arxiv.org/abs/1606.03137) are making their way to deep RL. Ultimately I expect we'll want an agent that takes all of its sensory data as evidence about "what the human wants", rather than relying on a special reward channel, or a special type of data called "comparisons" or "demonstrations", and this work takes that sort of approach.

For these simple environments, an agent trained to perform well with another artificial agent will generalize reasonably well to real humans, because there's only a few reasonable strategies for the principal to take. However, with more complex environments, when there are many ways to interact, we can't expect such generalization. (I'll have a paper and blog post coming out soon about this phenomenon.)

Learning to Interactively Learn and Assist

Mark Woodward, Chelsea Finn, Karol Hausman

[Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137) proposed a model in which an AI assistant would help a human principal, where only the principal knows the task reward. This paper explores this idea in the context of deep reinforcement learning. In their grid-world environment, two agents move around and pick up lemons or plums. The principal is penalized for moving, but is the only one who knows whether plums or lemons should be picked up. The authors hypothesize that simply by jointly training the two agents to maximize rewards, they will automatically learn to interact in order for the assistant to learn the task, rather than requiring an explicit mechanism like comparisons or demonstrations.

Recurrent Q-networks are used for the agents, which are then trained via deep Q-learning. The authors run several experiments that show emergent interaction. In the first experiment, when the principal is penalized for moving it learns to demonstrate the task to the assistant, and then let the assistant finish the job. In the second experiment, when the assistant has a restricted field of view, it learns to follow the principal until it can infer whether the principal wants plums or lemons. In the third, they tell the assistant the task 50% of the time, and so the principal is initially unsure whether the agent needs any direction (and due to the motion cost, the principal would rather not do anything). When the agent knows the task, it performs it. When the agent doesn't know the task, it moves closer to the principal, in effect "asking" what the reward is, and the principal moves until it can see the object, and then "answers" by either moving towards the object (if it should be collected) or doing nothing (if not). Finally, the authors run an experiment using pixels as input. While they had to switch to dueling DQNs instead of vanilla DQNs, they show that the joint reward is competitive with the grid approach. They also run an experiment with human principals and show that the human/assistant pair outperforms the solo-human setup.


I'm really excited to see a project that is very directly aimed at inferring end user intent in a complex environment. This seems like a great direction for the field to move towards. I think Minecraft would also be great as a test bed for the setting in which _researchers or engineers_ (as opposed to end users) are trying to get an agent to do something: we can assume more expertise and knowledge here. Ideally, this would allow us to solve more complex tasks than can be accomplished with natural language from end users. I personally plan to do work with Minecraft along these lines.

While this project does need to infer intent, it probably won't require the sort of pragmatic understanding shown by e.g. [Cooperative IRL](https://arxiv.org/abs/1606.03137). Even understanding what the human is literally asking for in Minecraft is currently beyond our capabilities.

CraftAssist: A Framework for Dialogue-enabled Interactive Agents

Why Build an Assistant in Minecraft?

Arthur Szlam, Jonathan Gray, Kavya Srinet, Yacine Jernite, Armand Joulin, Gabriel Synnaeve, Douwe Kiela, Haonan Yu, Zhuoyuan Chen, Siddharth Goyal, Demi Guo, Danielle Rothermel, C. Lawrence Zitnick, Jason Weston

This position paper proposes a new challenge for AI research: building a bot that can provide assistance in [Minecraft](https://www.minecraft.net/en-us/) (creative mode). A [companion paper](https://research.fb.com/wp-content/uploads/2019/07/CraftAssist-A-Framework-for-Dialogue-enabled-Interactive-Agents-v3.pdf) presents an initial setup for such an agent.

The main goal here is to advance natural language understanding, intent inference and instruction following. As a result, there is no formal specification like a reward function -- in their own words, "the ultimate goal of the bot is to be a useful and fun assistant in a wide variety of tasks specified and evaluated by human players". They chose Minecraft in particular partly because it has a very rich space of _tasks_, even though the _execution_ of any given task is relatively straightforward. They script many low level policies to automate this execution in order to make learning easier (for example, they have policies to navigate to a location or to build specified structures) and focus the learning challenge on figuring out what the user wants.

The current version of the bot takes dialogue from the user and uses a neural model to parse it into an _action dictionary_ that unambiguously specifies what the agent should do -- I think this neural model is the main thing to be learned. There are a bunch of details on how the rest of the modules work as well. They have also released three datasets: a semantic parsing dataset that associates instructions with action dictionaries, a house dataset that has trajectories where a human builds a house, and a semantic segmentation dataset that labels various parts of houses.


While I don't disagree with the reasoning, I disagree with the main thrust of this post. I wrote a long [comment](https://www.alignmentforum.org/posts/BKjJJH2cRpJcAnP7T/thoughts-on-human-models#8BAMtuZfBNqyF7Jng) about it; the TL;DR is that since humans want very specific behavior out of AI systems, the AI system needs to get a lot of information from humans about what it should do, and if it understands all that information then it necessarily has a (maybe implicit) human model. In other words, if you require your AI system not to have human models, it will not be very useful, and people will use other techniques.

Thoughts on Human Models

Many approaches to AI safety involve modeling humans in some way, for example in order to correctly interpret their feedback. However, there are significant disadvantages to human modeling. First and most importantly, if we have AI systems do useful things _without_ modeling humans, then we can use human approval as a "test set": we can check whether the AI's behavior is something we approve of, and this is an _independent_ evaluation of the AI system. However, if the AI system had a human model, then it may have optimized its behavior for human approval, and so we cannot use approval as a "test set". Second, if our AI system has a catastrophic bug, it seems better if it doesn't have any human models. An AI system without human models will at worst optimize for some unrelated goal like paperclips, which at worst leads to it treating humans as obstacles and causing extinction. However, an AI system with human models with a catastrophic bug might optimize for human suffering, or having humans respond to email all day, etc. Thirdly, an AI system with human models might be simulating conscious beings that can suffer. Fourthly, since humans are agent-like, an AI system that models humans is likely to produce a subsystem that is agent-like and so dangerous.

The authors then discuss why it might be hard to avoid human models. Most notably, it is hard to see how to use a powerful AI system that avoids human models to produce a better future. In particular, human models could be particularly useful for interpreting specifications (in order to do what humans mean, as opposed to what we literally say) and for achieving performance given a specification (e.g. if we want to replicate aspects of human cognition). Another issue is that it is hard to avoid human modeling, since even "independent" tasks have some amount of information about human motivations in selecting that task.

Nevertheless, the authors would like to see more work on engineering-focused approaches to AI safety without human models, especially since this area is neglected, with very little such work currently. While MIRI does work on AI safety without human models, this is from a very theoretical perspective. In addition to technical work, we could also promote certain types of AI research that is less likely to develop human models "by default" (e.g. training AI systems in procedurally generated simulations, rather than on human-generated text and images).


It's exciting to see reward learning applied to a concrete problem that researchers are working on, and having it lead to an actually better system. This work uses reward learning in a context where we are trying to mimic human actions (sentence generation in this case) -- eventually we will want to be able to deal with different action spaces than humans (as in robotics) and aiming to reach superhuman performance.

No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling

Xin Wang, Wenhu Chen, Yuan-Fang Wang, William Yang Wang

This paper tackles visual story-telling, the task of generating a story that matches a sequence of photos. It proposes learning a reward function from the labeled dataset that can then be optimized with reinforcement learning, with the hope that the reward function is a good compression of what we want and so leads to more generalizable behavior. They show that the standard automated techniques for evaluating visual stories are not very good, and so they perform a Mechanical Turk study that shows very good results compared to prior work. MTurk workers are often unable to tell whether the stories were generated by their algorithm or a human!

How does it work? Their architecture has a policy network that creates the stories and a reward network that provides the supervision, which are trained adversarially. We can think of the reward function as inducing a probability distribution over stories, where stories with higher reward are more probable. Then, the reward network acts as a discriminator, trying to make its implied probability distribution similar to the empirical data distribution and dissimilar to the policy network distribution, while the policy network acts as a generator, creating a policy that tries to match the implied probability distribution of the reward network. (This is equivalent to maximizing the expected reward from the reward network.)


In addition to this blog post and the [paper](https://openreview.net/forum?id=rkevMnRqYQ), I also wrote a [post](https://www.alignmentforum.org/posts/7f6DNZhracD7RvxMr/learning-preferences-by-looking-at-the-world) on the Alignment Forum expressing opinions about the work. There are too many disparate opinions to put in here, so I'd recommend reading the post itself. I guess one thing I'll mention is that to infer preferences with a single state, you definitely need a good dynamics model, and a good set of features. While this may seem difficult to get, it's worth noting that dynamics are empirical facts about the world, and features might be, and there is already lots of work on learning both dynamics and features.

Paper: Preferences Implicit in the State of the World

Learning Preferences by Looking at the World

The key idea with this project of mine is that the state of the world is already optimized for our preferences, and so simply by looking at the world we can infer these preferences. Consider the case where there is a vase standing upright on the table. This is an unstable equilibrium -- it's very easy to knock over the vase so it is lying sideways, or is completely broken. The fact that this hasn't happened yet suggests that we care about vases being upright and intact; otherwise at some point we probably would have let it fall.

Since we have optimized the world for our preferences, the natural approach is to model this process, and then invert it to get the preferences. You could imagine that we could consider all possible reward functions, and put probability mass on them in proportion to how likely they make the current world state if a human optimized them. Basically, we are simulating the past in order to figure out what must have happened and why. With the vase example, we would notice that in any reward function where humans wanted to break vases, or were indifferent to broken vases, we would expect the current state to contain broken vases. Since we don't observe that, it must be the case that we care about keeping vases intact.

Our algorithm, Reward Learning by Simulating the Past (RLSP), takes this intuition and applies it in the framework of [Maximum Causal Entropy IRL](http://www.cs.cmu.edu/~bziebart/publications/maximum-causal-entropy.pdf) ([AN #12](https://mailchi.mp/bcb2c6f1d507/alignment-newsletter-12)), where you assume that the human was acting over T timesteps to produce the state that you observe. We then show a few gridworld environments in which applying RLSP can fix a misspecified reward function.


I'm always happy to see an experimental paper that tests how algorithms perform, I think we need more of these.

You might be tempted to think of this as evidence that in deep RL, a model-based method should outperform a model-free one. This isn't exactly right. The first ToM and black box model-based algorithms use an exact model of the dynamics of the environment modulo the human, that is, they can exactly predict the next state given the current state, the robot action, and the human action. The model-free algorithm must learn this from scratch, so it isn't an apples-to-apples comparison. (Typically in deep RL, both model-based and model-free algorithms have to learn the environment dynamics.) However, you _can_ think of the ToM as a model-based method and the Black-box model-based algorithm as a model-free algorithm, where both algorithms have to learn the _human model_ instead of the more traditional environment dynamics. With that analogy, you would conclude that model-based algorithms will be more sample efficient and more performant in low-data regimes, but will be outperformed by model-free algorithms with sufficient data, which agrees with my intuitions.

This kind of effect is a major reason for my position that the first powerful AI systems will be modular (analogous to model-based systems), but that they will eventually be replaced by more integrated, end-to-end systems (analogous to model-free systems). Initially, we will be in a (relatively speaking) low-data regime, where modular systems excel, but over time there will be more data and compute and we will transition to regimes where end-to-end systems will solve the same tasks better (though we may then have modular systems for more difficult tasks).

On the Utility of Model Learning in HRI

Rohan Choudhury, Gokul Swamy, Dylan Hadfield-Menell, Anca Dragan

In human-robot interaction (HRI), we often require a model of the human that we can plan against. Should we use a specific model of the human (a so-called "theory of mind", where the human is approximately optimizing some unknown reward), or should we simply learn a model of the human from data? This paper presents empirical evidence comparing three algorithms in an autonomous driving domain, where a robot must drive alongside a human.

The first algorithm, called Theory of Mind based learning, models the human using a theory of mind, infers a human reward function, and uses that to predict what the human will do, and plans around those actions. The second algorithm, called Black box model-based learning, trains a neural network to directly predict the actions the human will take, and plans around those actions. The third algorithm, model-free learning, simply applies Proximal Policy Optimization (PPO), a deep RL algorithm, to directly predict what action the robot should take, given the current state.

Quoting from the abstract, they "find that there is a significant sample complexity advantage to theory of mind methods and that they are more robust to covariate shift, but that when enough interaction data is available, black box approaches eventually dominate". They also find that when the ToM assumptions are significantly violated, then the black-box model-based algorithm will vastly surpass ToM. The model-free learning algorithm did not work at all, probably because it cannot take advantage of knowledge of the dynamics of the system and so the learning problem is much harder.


AI Alignment Podcast: Cooperative Inverse Reinforcement Learning

Dylan puts forward his conception of Cooperative Inverse Reinforcement Learning as a definition of what it means for a human-AI system to be rational, given the information bottleneck between a human's preferences and an AI's observations. He notes that there are some clear mismatches between this problem and reality, such as the CIRL assumption that humans have static preferences, and how fuzzy the abstraction of "rational agents with utility functions" becomes in the context of agents with bounded rationality. Nevertheless, he claims that this is a useful unifying framework for thinking about AI safety.

Dylan argues that the process by which a robot learns to accomplish tasks is best described not just as maximising an objective function but instead in a way which includes the system designer who selects and modifies the optimisation algorithms, hyperparameters, etc. In fact, he claims, it doesn't make sense to talk about how well a system is doing without talking about the way in which it was instructed and the type of information it got. In CIRL, this is modeled via the combination of a "teaching strategy" and a "learning strategy". The former can take many forms: providing rankings of options, or demonstrations, or binary comparisons, etc. Dylan also mentions an extension of this in which the teacher needs to learn their own values over time. This is useful for us because we don't yet understand the normative processes by which human societies come to moral judgements, or how to integrate machines into that process.


This seems like an interesting idea for how to build an AI system in practice, along the same lines as corrigibility. We notice that value learning is not very robust: if you aren't very good at value learning, then you can get very bad behavior, and human values are sufficiently complex that you do need to be very capable in order to be sufficiently good at value learning. With (a particular kind of) [corrigibility](https://ai-alignment.com/corrigibility-3039e668638), we instead set the goal to be to make an AI system that is trying to help us, which seems more achievable even when the AI system is not very capable. Similarly, if we formalize or learn informed consent reasonably well (which seems easier to do since it is not as complex as "human values"), then our AI systems will likely have good behavior (though they will probably not have the best possible behavior, since they are limited by having to respect informed consent).

However, this also feels different from corrigibility, in that it feels more like a limitation put on the AI system, while corrigibility seems more like a property of the AI's "motivational system". This might be fine, since the AI might just not be goal-directed. One other benefit of corrigibility is that if you are "somewhat" corrigible, then you would like to become more corrigible, since that is what the human would prefer; informed-consent-AI doesn't seem to have an analogous benefit.

Non-Consequentialist Cooperation?

One possible way to build useful AI systems is to have them [try to help us](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment). Taking more of a libertarian stance, a robot could help us in an autonomy-centric way, which would only take actions if we give it our informed consent. We can't ask for explicit consent for every action, since there's no clear way to break down actions, and it would certainly be too onerous to give informed consent to every possible motor command. As a result, our robots will need to infer when we have given consent. This increases the chances of misunderstanding, but we could try to have a high threshold, so that the robot asks us if it is even a little unsure about whether we have given consent.

If we want to precisely define consent, we'll need to solve some of the same problems that impact measures have to contend with. In particular, we would need to get consent for outcomes that the robot knows will happen as a result of its actions, but not ones that happen as a side effect. It's fine to give informed consent to the robot to buy bananas from a grocery store, even if that could cause a hurricane, as long as the robot doesn't _know_ that it would cause a hurricane. Another issue is that inferring consent requires you to confront the issue that humans can be irrational. A third issue is that we might prevent the robot from taking actions that would help us that we can't understand would help us -- consider trying to ask a dog for its informed consent to take it to the vet.


I strongly agree that due to the problems mentioned in this post, we shouldn't be trying to mix and match value learning methods to infer a static notion of human preferences that is then optimized over the long term. I don't personally work on building a theory of human values because it seems like I could apply the same critiques to the result: the theory is specialized to our current situation and won't capture changes in the future. But of course this is hard to predict without knowing the theory. My preferred solution is not to build a system that is optimizing a goal-directed utility function over the long term, and to find other ways of making an AI system that are still just as useful.

Why we need a *theory* of human values

There are many different sources of information for human values, such as behavior, speech, facial expressions/emotions, and extrapolations of what a human would do. These have all been honed to produce similar preferences in our current environment. However, the environment will change a lot due to the AI's actions, inducing a distributional shift, after which we should no longer expect the values inferred from these different methods to agree with each other. In addition, we also have the problem that each method only applies in some circumstances -- for example, people are likely to misrepresent their values if asked in a courtroom. We could try to patch these problems by having a meta-method that chooses from the various methods of value learning, as well as predicting human judgments about when each method applies. However, then we'd likely have similar issues with the meta-method and predictions, which are also likely to be specific to the current environment. Instead, we should have a _theory_ of human values, from which we can get principled approaches to resolve these problems.


The proposal sounds to me like a specific flavor of narrow value learning, where you learn reward functions to accomplish particular tasks, rather than trying to figure out the "true human utility function". The recursive aspect is similar to [iterated amplification](https://blog.openai.com/amplifying-ai-training/) and [debate](https://blog.openai.com/debate/). Iterated amplification and debate can be thought of as operating on a tree of arguments, where each node is the result of considering many child nodes (the considerations that go into the argument). Importantly, the child nodes are themselves arguments that can be decomposed into smaller considerations. Iterated amplification works by learning how to compose and decompose nodes from children, while debate works by having humans evaluate a particular path in the argument tree. Recursive reward modeling instead uses reward modeling to train agents that can help _evaluate outcomes_ on the task of interest. This seems less recursive to me, since the subagents are used to evaluate outcomes, which would typically be a different-in-kind task than the task of interest. This also still requires the tasks to be fast -- it is not clear how to use recursive reward modeling to eg. train an agent that can teach math to children, since it takes days or months of real time to even produce outcomes to evaluate. These considerations make me a bit less optimistic about recursive reward modeling, but I look forward to seeing future work that proves me wrong.

The post also talks about how reward modeling allows us to separate what to do (reward) from how to do it (policy). I think it is an open question whether this is desirable. [Past work](https://arxiv.org/abs/1806.01946) found that the reward generalized somewhat (whereas policies typically don't generalize at all), but this seems relatively minor. For example, rewards inferred using deep variants of inverse reinforcement learning often don't generalize. Another possibility is that the particular structure of "policy that optimizes a reward" provides a useful inductive bias that makes things easier to learn. It would probably also be easier to inspect a specification of "what to do" than to inspect learned behavior. However, these advantages are fairly speculative and it remains to be seen whether they pan out. There are also practical advantages: any advances in deep RL can immediately be leveraged, and reward functions can often be learned much more sample efficiently than behavior, reducing requirements on human labor. On the other hand, this design "locks in" that the specification of behavior must be a reward function. I'm not a fan of reward functions because they're so unintuitive for humans to work with -- if we could have agents that work with natural language, I suspect I do not want the natural language to be translated into a reward function that is then optimized.

Scalable agent alignment via reward modeling

This blog post and the [associated paper](https://arxiv.org/abs/1811.07871) outline a research direction that DeepMind's AGI safety team is pursuing. The key idea is to learn behavior by learning a reward and a policy simultaneously, from human evaluations of outcomes, which can scale to superhuman performance in tasks where evaluation is easier than demonstration. However, in many cases it is hard for humans to evaluate outcomes: in this case, we can train simpler agents using reward modeling that can assist the human in evaluating outcomes for the harder task, a technique the authors call recursive reward modeling. For example, if you want to train an agent to write a fantasy novel, it would be quite expensive to have a human evaluate outcomes, i.e. rate how good the produced fantasy novels are. We could instead use reward modeling to train agents that can produce plot summaries, assess prose quality and character development, etc. which allows a human to assess the fantasy novels. There are several research challenges, such as what kind of feedback to get, making it sufficiently sample efficient, preventing reward hacking and unacceptable outcomes, and closing the reward-result gap. They outline several promising approaches to solving these problems.


I'm excited to see work that combines information from multiple sources! In general with multiple sources you have the problem of figuring out what to do when the sources of information conflict, and this is no exception. Their approach tends to prioritize demonstrations over preferences when the two conflict, and so in cases where the preferences are better (as in Enduro) their approach performs poorly. I'm somewhat surprised that they prioritize demos over preferences, since it seems humans would be more reliable at providing preferences than demos, but perhaps they needed to give demos more influence over the policy in order to have the policy learn reasonably quickly. I'd be interested in seeing work that tries to use the demos as much as possible, but detect when conflicts happen and prioritize the preferences in that situation -- my guess is that this would let you get good performance across most Atari games.

Reward learning from human preferences and demonstrations in Atari

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, Dario Amodei

We have had lots of work on learning from preferences, demonstrations, proxy rewards, natural language, rankings etc. However, most such work focuses on one of these modes of learning, sometimes combined with an explicit reward function. This work learns to play Atari games using both preference and demonstration information. They start out with a set of expert demonstrations which are used to initialize a policy using behavioral cloning. They also use the demonstrations to train a reward model using the DQfD algorithm. They then continue training the reward and policy simultaneously, where the policy is trained on rewards from the reward model, while the reward model is trained using preference information (collected and used in the same way as Deep RL from Human Preferences) and the expert demonstrations. They then present a _lot_ of experimental results. The main thing I got out of the experiments is that when demonstrations are good (near optimal), they convey a lot of information about how to perform the task, leading to high reward, but when they are not good, they will actively hurt performance, since the algorithm assumes that the demonstrations are high quality and the demonstrations "override" the more accurate information collected via preferences. They also show results on efficiency, the quality of the reward model, and the reward hacking that can occur if you don't continue training the reward model alongside the policy.


I really like this paper, it showcases the benefits of both model-based planning and imitation learning. Since the problem has been decomposed into a predictive model, a goal G, and a planner, we can edit G directly to get new behavior at test time without any retraining (as they demonstrate with the pothole experiment). At the same time, they can get away with not specifying a full reward function, as many features of good driving, like passenger comfort and staying in the correct lane, are learned simply by imitating an expert.

That said, they initially state that one of their goals is to learn from offline data, even though offline data typically has no examples of crashes, and "A model ignorant to the possibility of a crash cannot know how to prevent it". I think the idea is that you never get into a situation where you could get in a crash, because you never deviate from expert behavior since that would have low P(s_{1:T} | φ). This is better than model-based planning on offline data, which would consider actions that lead to a crash and have no idea what would happen, outputting garbage. However, it still seems that situations could arise where a crash is imminent, which don't arise much (if at all) in the training data, and the car fails to swerve or brake hard, because it hasn't seen enough data.

Deep Imitative Models for Flexible Inference, Planning, and Control

Nicholas Rhinehart, Rowan McAllister, Sergey Levine

It's hard to apply deep RL techniques to autonomous driving, because we can't simply collect a large amount of experience with collisions in order to learn. However, imitation learning is also hard, because as soon as your car deviates from the expert trajectories that you are imitating, you are out of distribution, and you could make more mistakes, leading to accumulating errors until you crash. Instead, we can model the expert's behavior, so that we can tell when we are moving out of distribution, and take corrective action.

They split up the problem into three different stages. First, they generate a set of _waypoints_ along the path to be followed, which are about 20m away from each other, by using A* search on a map. Next, they use model-based planning using an imitative model to generate a plan (sequence of states) that would take the car to the next waypoint. Finally, they use a simple PID controller to choose low-level actions that keep the car on target towards the next state in the plan.

The key technical contribution is with the imitative model, which is a probabilistic model P(s_{1:T}, G, φ), where φ is the current observation (eg. LIDAR), s_{1:T} is the planned trajectory, and G is a goal. We can learn P(s_{1:T} | φ) from expert demonstrations. The goal G can be anything for which you can write down a specification P(G | s_{1:T}, φ). For example, if you simply want to reach a waypoint, you can use the normal distribution on the distance between the final state s_T and the waypoint. You can also incorporate a hand-designed cost on each state.

They evaluate in simulation on a static world (so no pedestrians, for example). They show decent transfer from one map to a second map, and also that they can avoid artificially introduced potholes at test time (despite not seeing them at training time), simply by adding a cost on states over a pothole (which they can take into account because they are performing model-based planning).


Performance increased with the inclusion of human data, even that from unrelated tasks, so one intuition I updated on was "More data is better, even if it's not perfect". This also feels related to "Data as regularization": to some extent, noisy data combats model overconfidence, and perhaps this would play an important role in aligning future systems.

Another thing I like about such pipeline papers is the opportunity to look for where systems might break. For example, in this work, the robot does actually need (prior) experience in the test environments with which to train the video predictor to be able to generate candidate solutions at test time. So in spite of the given result -- that DVD itself needs limited robot trajectories and no data from the test environments -- there's a potential point-of-failure far sooner in the pipeline, where if the robot did not have sufficient _background_ experience with diverse situations, it might not provide _any_ feasible candidate actions for DVD's evaluation.

Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos

This work demonstrates a method that learns a _generalizable multi-task reward function_ in the context of robotic manipulation; at deployment, this function can be conditioned on a human demonstration of an unseen task to generate reward signals for the robot, even in new environments.

A key insight here was to train a discriminative model that learned whether two given video clips were performing the same actions. These clips came from both a (large) dataset of human demonstrations and a relatively smaller set of robot expert trajectories, and each clip was labelled with a task-id. This training pipeline thus leveraged huge quantities of extant human behaviour from a diversity of viewpoints to learn a metric of 'functional similarity' between pairs of videos, independent of whether they were executed by human or machine.

Once trained, this model (called the 'Domain-agnostic Video Discriminator' or DVD) can determine if a candidate robotic behaviour is similar to a desired human-demonstrated action. Such candidates are drawn from an action-conditioned video predictor, and the best-scoring action sequence is selected for execution on the (simulated or real) robot.


You won’t be surprised to hear that I’m excited about this benchmark, given that I worked on it. While we listed a bunch of concrete advantages in the post above, I think many (though not all) of the advantages come from the fact that we are trying to mimic the situation we face in the real world as closely as possible, so there’s less opportunity for Goodhart’s Law to strike. For example, later in this newsletter we’ll see that synthetically generated demos are not a good proxy for human demos. Even though this is the norm for existing benchmarks, and we didn’t intentionally try to avoid this problem, BASALT (mostly) avoids it. With BASALT you would have to go pretty far out of your way to get synthetically generated demos, because by design the tasks are hard to complete synthetically, and so you _have_ to use human demos.

I’d encourage readers to [participate in the competition](https://www.aicrowd.com/challenges/neurips-2021-minerl-basalt-competition), because I think it’s especially good as a way to get started with ML research. It’s a new benchmark, so there’s a lot of low-hanging fruit in applying existing ideas to the benchmark, and in identifying new problems not present in previous benchmarks and designing solutions to them. It’s also pretty easy to get started: the BC baseline is fairly straightforward and takes a couple of hours to be trained on a single GPU. (That’s partly because BC doesn’t require environment samples; something like <@GAIL@>(@Generative Adversarial Imitation Learning@) would probably take a day or two to train instead.)

Paper: The MineRL BASALT Competition on Learning from Human Feedback

BASALT: A Benchmark for Learning from Human Feedback

Rohin Shah, Cody Wild, Steven H. Wang, Neel Alex, Brandon Houghton, William Guss, Sharada Mohanty, Anssi Kanervisto, Stephanie Milani, Nicholay Topin, Pieter Abbeel, Stuart Russell, Anca Dragan

A typical argument for AI risk, given in <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@), is that current AI systems treat their specifications as definite and certain, even though they are typically misspecified. This state of affairs can lead to the agent pursuing <@instrumental subgoals@>(@The Basic AI Drives@). To solve this, we might instead build AI systems that continually learn the objective from human feedback. This post and paper (on which I am an author) present the MineRL BASALT competition, which aims to promote research on algorithms that learn from human feedback.

BASALT aims to provide a benchmark with tasks that are realistic in the sense that (a) it is challenging to write a reward function for them and (b) there are many other potential goals that the AI system “could have” pursued in the environment. Criterion (a) implies that we can’t have automated evaluation of agents (otherwise that could be turned into a reward function) and so suggests that we use human evaluation of agents as our ground truth. Criterion (b) suggests choosing a very “open world” environment; the authors chose Minecraft for this purpose. They provide task descriptions such as “create a waterfall and take a scenic picture of it”; it is then up to researchers to create agents that solve this task using any method they want. Human evaluators then compare two agents against each other and determine which is better. Agents are then given a score using the [TrueSkill system](https://en.wikipedia.org/wiki/TrueSkill).

The authors provide a number of reasons to prefer the BASALT benchmark over more traditional benchmarks like Atari or MuJoCo:

1. In Atari or MuJoCo, there are often only a few reasonable goals: for example, in Pong, you either hit the ball back, or you die. If you’re testing algorithms that are meant to learn what the goal is, you want an environment where there could be many possible goals, as is the case in Minecraft.
2. There’s lots of Minecraft videos on YouTube, so you could test a “GPT-3 for Minecraft” approach.
3. The “true reward function” in Atari or MuJoCo is often not a great evaluation: for example, a Hopper policy trained to stand still using a constant reward gets 1000 reward! Human evaluations should not be subject to the same problem.
4. Since the tasks were chosen to be inherently fuzzy and challenging to formalize, researchers are allowed to take whatever approach they want to solving the task, including “try to write down a reward function”. In contrast, for something like Atari or MuJoCo, you need to ban such strategies. The only restriction is that researchers cannot extract additional state information from the Minecraft simulator.
5. Just as we’ve <@overestimated few-shot learning capabilities@>(@True Few-Shot Learning with Language Models@) by tuning prompts on large datasets of examples, we might also be overestimating the performance of algorithms that learn from human feedback because we tune hyperparameters on the true reward function. Since BASALT doesn’t have a true reward function, this is much harder to do.
6. Since Minecraft is so popular, it is easy to hire Minecraft experts, allowing us to design algorithms that rely on expert time instead of just end user time.
7. Unlike Atari or MuJoCo, BASALT has a clear path to scaling up: the tasks can be made more and more challenging. In the long run, we could aim to deploy agents on public multiplayer Minecraft servers that follow instructions or assist with whatever large-scale project players are working on, all while adhering to the norms and customs of that server.


This is cool, it actually works on a real robot, and it deals with the issue that a human and a robot have different action spaces.

Some form of meta-learning (ideally [MAML](http://bair.berkeley.edu/blog/2017/07/18/learning-to-learn/)).

One-Shot Imitation from Watching Videos

Can we get a robot to learn a task by watching a human do it? This is very different from standard imitation learning. First, we want to do it with a single demonstration, and second, we want to do it by _watching a human_ -- that is, we're learning from a video of a human, not a trajectory where the robot actions are given to us. Well, first consider how we could do this if we have demonstrations from a teleoperated robot. In this case, we do actually have demonstrations in the form of trajectories, so normal imitation learning techniques (behavioral cloning in this case) work fine. We can then take this loss function and use it with [MAML](http://bair.berkeley.edu/blog/2017/07/18/learning-to-learn/) to learn from a large dataset of tasks and demonstrations how to perform a new task given a single demonstration. But this still requires the demonstration to be collected by teleoperating the robot. What if we want to learn from a video of a human demonstrating? They propose learning a _loss function_ that given the human video provides a loss from which gradients can be calculated to update the policy. Note that at training time there are still teleoperation demonstrations, so the hard task of learning how to perform tasks is done then. At test time, the loss function inferred from the human video is primarily used to identify which objects to manipulate.


I really liked this paper -- it seems like a far more efficient use of human feedback to figure out what _features_ of the environment are important. This doesn’t need to be limited to reward learning: I expect that learning the right features to focus on would help with exploration in reinforcement learning, out-of-distribution generalization, etc. It also seems plausible that in more complex environments you could learn a set of features that was useful for all of these tasks, thus being somewhat general (though still specific to the environment).

It’s worth noting that in this setting you wouldn’t really want to use a vanilla deep IRL algorithm -- you’d instead want to do something like <@meta-IRL@>(@Multi-task Maximum Entropy Inverse Reinforcement Learning@).

Feature Expansive Reward Learning: Rethinking Human Input

Andreea Bobu*, Marius Wiggert*, Claire Tomlin, Anca D. Dragan

One goal we might have with our algorithms is that _after_ training, when the AI system is deployed with end users, the system would be personalized to those end users. You might hope that we could use deep inverse RL algorithms like <@AIRL@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@), but unfortunately they require a lot of data, which isn’t feasible for end users. You could use earlier IRL algorithms like <@MCEIRL@>(@Modeling Interaction via the Principle of Maximum Causal Entropy@) that require you to specify what features of the environment you care about, but in practice you’ll never successfully write down all of these features. Can we somehow get the best of both worlds?

<@Past work@>(@Learning under Misspecified Objective Spaces@) made progress on this front, by allowing the agent to at least _detect_ when it is missing some feature, by checking whether the human feedback is surprisingly inefficient given the existing features. But what do you do once you detect it? The key insight of this paper is that applying a deep IRL algorithm here would be inefficient because it has to implicitly learn the unknown feature, and we can do much better by explicitly querying the human for the unknown feature.

In particular, their method Feature Expansive Reward Learning (FERL) asks the human for a few _feature traces_: demonstrations in which the new feature’s value monotonically decreases. For example, suppose a robot arm carrying a cup of water gets too close to a laptop, but the arm doesn’t know the feature “close to a laptop”. Then a feature trace would start with the arm close to the laptop, and move it successively further away. Given a set of feature traces, we can convert this into a dataset of noisy comparisons, where earlier states are more likely to have higher feature values than later states, and use this to train a neural net to predict the feature value (similarly to the reward model in [Deep RL from Human Preferences](https://deepmind.com/blog/learning-through-human-feedback/)). We can then add this to our set of features, and learn rewards over the new set of features.

They evaluate their method with a few human-robot interaction scenarios (though without a user study due to COVID), comparing it against deep MaxEnt IRL, and find that their method does better on a variety of metrics.


I like the idea of separating "what to do" from "how to do it", since the "what to do" is more likely to generalize to new circumstances. Of course, this can also be achieved by learning a reward function, which is one way to encode "what to do". I'm also happy to see progress on the front of learning what humans want where we can take advantage of adversarial training that leads to a natural curriculum -- this has been key in many systems, most notably AlphaZero.

I'm somewhat surprised that dropping the top 25% of states ranked highly by the discriminator works. I would have guessed that states that are "near" the goal states might be misclassified by the discriminator, and the mistake will never be fixed because those states will always be in the top 25% and so will never show up as negative examples. I don't know whether I should expect this problem to show up in other environments, or whether there's a good reason to expect it won't happen.

I'm also surprised at the results from one of their experiments. In this experiment, they trained the agent in the normal environment, but then made red squares immovable in the test environment. This only changes the dynamics, and so the discriminator should work just as well (about 99.5% accuracy). The policy performance tanks (from 98% to 52%), as you'd expect when changing dynamics, but if you then finetune the policy, it only gets back to 69% success. Given that the discriminator should be just as accurate, you'd expect the policy to get back to 98% accuracy. Partly the discrepancy is that some tasks become unsolvable when red squares are immovable, but they say that this is a small effect. My hypothesis is before finetuning, the policy is very certain of what to do, and so doesn't explore enough during finetuning, and can't learn new behaviors effectively. This would mean that if they instead retrained the policy starting from a random initialization, they'd achieve better performance (though likely requiring many more samples).

Learning to Follow Language Instructions with Adversarial Reward Induction

Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, Edward Grefenstette

Adversarial Goal-Induced Learning from Examples (AGILE) is a way of training an agent to follow instructions. The authors consider a 5x5 gridworld environment with colored shapes that the agent can manipulate. The agent is given an instruction in a structured domain-specific language. Each instruction can correspond to many goal states -- for example, the instruction corresponding to "red square south of the blue circle" has many different goal states, since only the relative orientation of the shapes matters, not their absolute positions.

The key idea is to learn two things simultaneously -- an encoding of _what_ the agent needs to do, and a policy that encodes _how_ to do it, and to use these two modules to train each other. The "what" is encoded by a discriminator that can classify (state, instruction) pairs as either being a correct goal state or not, and the "how" is encoded by a policy. They assume they have some human-annotated goal states for instructions. The discriminator is then trained with supervised learning, where the positive examples are the human-annotated goal states, and the negative examples are states that the policy achieves during training (which are usually failures). The policy is trained using A3C with a reward function that is 1 if the discriminator says the state is more likely than not to be a goal state, and 0 otherwise. Of course, if the policy actually achieves the goal state, there is no way of knowing this apart from the discriminator -- so by default _all_ of the states that the policy achieves (including goal states) are treated as negative examples for the dsicriminator. This leads to the discriminator getting slightly worse over time as the policy becomes better, since it is incorrectly told that certain states are not goal states. To fix this issue, the authors drop the top 25% of states achieved by the policy that have the highest probability of being a goal state (according to the discriminator).

The authors compare AGILE against A3C with the true reward function (i.e. the reward function implied by a perfect discriminator) and found that AGILE actually performed _better_, implying that the inaccuracy of the discriminator actually _helped_ with learning. The authors hypothesize that this is because when the discriminator incorrectly rewards non-goal states, it is actually providing useful reward shaping that rewards progress towards the goal, leading to faster learning. Note though that A3C with an auxiliary reward prediction objective performed best. They have several other experiments that look at individual parts of the system.


It’s certainly true that we don’t have good methods for understanding how well our learned reward models generalize, and I’m glad that this work is pushing in that direction. I hope that future papers on reward models report EPIC distances to the ground truth reward as one of their metrics (code is available [here](https://github.com/HumanCompatibleAI/evaluating-rewards)).

One nice thing is that, roughly speaking, rewards are judged to be equivalent if they would generalize to any possible transition function that is consistent with DT. This means that by designing DT appropriately, we can capture how much generalization we want to evaluate. This is a useful knob to have: if we used the maximally large DT, the task would be far too difficult, as it would be expected to generalize far more than even humans can.

Blog post: An EPIC way to evaluate reward functions

Quantifying Differences in Reward Functions

Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, Jan Leike

Current work on reward learning typically evaluates the learned reward models by training a policy to optimize the learned reward, and seeing how well that policy performs according to the true reward. However, this only tests how well the reward works in the particular environment you test in, and doesn’t tell you how well the reward will generalize. For example, suppose the user loves apricots, likes plums, but hates durians. A reward that has apricots > durians > plums works perfectly -- until the store runs out of apricots, in which case it buys the hated durian.

So, it seems like we should evaluate reward functions directly, rather than looking at their optimal policies. This paper proposes Equivalent-Policy Invariant Comparison (EPIC), which can compare two reward functions while ignoring any potential shaping that doesn’t affect the optimal policy.

EPIC is parameterized by a distribution of states and actions DS and DA, as well as a distribution DT over transitions (s, a, s’). The first step is to find canonical versions of the two rewards to be compared, such that they have expected zero reward over DS and DA, and any potential shaping is removed. Then, we look at the reward each of these would assign to transitions in DT, and compute the Pearson correlation. This is transformed to be in the range [0, 1], giving the EPIC distance.

The authors prove that EPIC is a pseudometric, that is, it behaves like a distance function, except that it is possible for EPIC(R1, R2) to be zero even if R1 and R2 are different. This is desirable, since if R1 and R2 differ by a potential shaping function, then their optimal policies are guaranteed to be the same _regardless_ of transition dynamics, and so we should report the “distance” between them to be zero.

The authors show how to approximately compute the EPIC distance in high dimensional environments, and run experiments to showcase EPIC’s properties. Their first experiment demonstrates that EPIC is able to correctly detect that a densely shaped reward for various MuJoCo environments is equivalent to a sparse reward, whereas other baseline methods are not able to do so. The second experiment compares reward models learned from preferences, demonstrations, and direct regression, and finds that the EPIC distance for the rewards learned from demonstrations are much higher than those for preferences and regression. Indeed, when the rewards are reoptimized in a new test environment, the new policies work when using the preference or regression reward models, but not when using the demonstration reward model. The final experiment shows that EPIC is robust to variations in the visitation distribution DT, while baseline methods are not.


Natural language is a very flexible and intuitive way to convey instructions to AI. In some ways, this shifts the alignment problem from the RL agent to the supervised language model, which just needs to learn how to correctly interpret the meaning behind human speech. One advantage of this approach is that the language model is separately trained so it can be tested and verified for safety criteria before being used to train an RL agent. It also may be more competitive than alternatives such as reward modeling that require training a new reward model for each task.

I do see a couple downsides to this approach, however. The first is that humans are not perfect at conveying their values in natural language (e.g. King Midas wishing for everything he touches to turn to gold), and natural language may not have enough information to convey complex preferences. Even if humans give precise and correct commands, the language model needs to verifiably interpret those commands correctly. This could be difficult as current language models are difficult to interpret and contain many harmful biases.

Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text

Felix Hill, Sona Mokra, Nathaniel Wong, Tim Harley

This paper proposes the Simulation-to-Human Instruction Following via Transfer from Text (SHIFTT) method for training an RL agent to receive commands from humans in natural language. One approach to this problem is to train an RL agent to respond to commands based on a template; however, this is not robust to small changes in how humans phrase the commands. In SHIFTT, you instead begin with a pretrained language model such as BERT and first feed the templated commands through the language model. This is then combined with vision inputs to produce a policy. The human commands are later fed through the same language model, and they find that the model has zero-shot transfer to the human commands even if they differ in structure.


I think this paper shows two important things:

1. Embedding the natural language instructions in the same space as the image conditioning works well, and is a good way of extending the usefulness of human annotations.

2. Large pretrained language models can be used to improve the performance of language-conditioned reinforcement learning (in this case imitation learning) algorithms and policies.

Methods which enable us to scale human feedback to complex settings are useful, and this method seems like it could scale well, especially with the use of pretrained large language models which might reduce the amount of language annotations needed further.

Grounding Language in Play

This paper presents a new approach to learning to follow natural language human instruction in a robotics setting. It builds on similar ideas to <@Learning Latent Plans from Play@>, in that it uses unsupervised "play" data (trajectories of humans playing on the robot with no goal in mind).

The paper combines several ideas to enable training a policy which can follow natural language instructions with only limited human annotations.
* In *Hindsight Instruction Pairing*, human annotators watch small trajectories from the play data, and label them with the instruction which is being completed in the clip. This instruction can take any form, and means we don't need to choose the instructions and ask humans to perform specific tasks.
* *Multicontext Imitation Learning* is a method designed to allow goal-conditioned policies to be learned with multiple different types of goals. For example, we can have lots of example trajectories where the goal is an end state image (as these can be generated automatically without humans), and just a small amount of example trajectories where the goal is a natural language instruction (gathered using *Hindsight Instruction Pairing*). The approach is to learn a goal embedding network for each type of goal specification, and a single shared policy which takes the goal embedding as input.

Combining these two methods enables them to train a policy and embedding networks end to end using imitation learning from a large dataset of (trajectory, image goal) pairs and a small dataset of (trajectory, natural language goal) pairs. The policy can follow very long sequences of natural language instructions in a fairly complex grasping environment with a variety of buttons and objects. Their method performs better than the Learning from Play (LfP) method, even though LfP uses a goal image as the goal conditioning, instead of a natural language instruction.

Further, they propose that instead of learning the goal embedding for the natural language instructions, they use a pretrained large language model to produce the embeddings. This improves the performance of their method over learning the embedding from scratch, which the authors claim is the first example of the knowledge in large language models being transferred and improving performance in a robotics domain. This model also performs well when they create purposefully out of distribution natural language instructions (i.e. with weird synonyms, or google-translated from a different language).


I would explain this paper's point somewhat differently than the paper does. Consider an AI system in which we build in a prior over rewards and an update rule, and then have it act in the world. At the end of the trajectory, it is rewarded according to the expected reward of the trajectory under the inferred posterior over rewards. Then, the AI system is incentivized to choose actions under which the resulting posterior is easy to maximize.

This doesn't require the reward function to be ambiguous; it just requires that the update rule isn't perfect. For example, imagine that Alice has a real preference for apples over bananas, and you use the update rule "if Alice eats an apple, infer that she likes apples; if Alice eats a banana, infer that she likes bananas". The robot finds it easier to grasp the rigid apple, and so can get higher expected reward in the worlds where Alice likes apples. If you train a robot in the manner above, then the robot will learn to throw away the bananas, so that Alice's only choice is an apple (that we assume she then eats), allowing the robot to "infer" that Alice likes apples, which it can then easily maximize. This sort of problem could happen in most current reward learning setups, if we had powerful enough optimizers.

It seems to me that the problem is that you are training the actor, but not training the update rule, and so the actor learns to "trick" the update rule. Instead, it seems like we should train both. This is kind of what happens with <@assistance games / CIRL@>(@Cooperative Inverse Reinforcement Learning@), in which you train a policy to maximize expected reward under the _prior_, and so the policy is incentivized to take the best information gathering actions (which, if you squint, is like "training to update well"), and to maximize what it thinks is the true reward. Of course, if your prior / update rule within the game are misspecified, then bad things can happen. See also Stuart's reactions [here](https://www.alignmentforum.org/posts/gbuwgyYG9WvtsErki/how-should-ais-update-a-prior-over-human-preferences) and [here](https://www.alignmentforum.org/posts/EYEkYX6vijL7zsKEt/reward-functions-and-updating-assumptions-can-hide-a), as well as my comments on those posts.

Blog post: Learning and manipulating learning

Pitfalls of learning a reward function online

Stuart Armstrong, Jan Leike, Laurent Orseau, Shane Legg

It can be dangerous to learn the metric that you are trying to optimize: if you don't set it up correctly, you may end up incentivizing the agent to "update in a particular direction" in the metric learning for the sake of future optimization (a point previously made in [Towards Interactive Inverse Reinforcement Learning](https://jan.leike.name/publications/Towards%20Interactive%20Inverse%20Reinforcement%20Learning%20-%20Armstrong,%20Leike%202016.pdf)). This paper analyzes the problems that can arise when an agent simultaneously learns a reward function, and optimizes that reward function.

The agent may have an incentive to "rig" the reward learning process, such that it finds a reward that is easy to optimize. For example, consider a student Sandra who must figure out the deadline and evaluation criteria for a project from a teacher Trisha. Sandra expects that if she asks Trisha when the deadline is, she will say that the deadline is later this week. So, Sandra might cleverly ask, "Is the project due next week, or the week after", to which Trisha might respond "next week". In this way, Sandra can rig the deadline-learning process in order to obtain a more favorable deadline.

Worse, in such scenarios the need to rig the learning process can destroy value for _every_ reward function you are considering. For example, let's suppose that if Trisha couldn't be manipulated, Sandra's optimal policy would be to start the project today, _regardless_ of when the actual deadline is. However, given that Trisha _can_ be manipulated, Sandra will spend today manipulating Trisha into setting a later deadline -- an action that seems clearly suboptimal from the perspective of any fixed deadline. The paper describes this as _sacrificing reward with certainty_.

To avoid such situations, we need _unriggable_ learning processes, that is, ones where at all times, the expected final learned reward (deadline) is independent of the agent's (Sandra's) policy. This unriggability property is nearly equivalent to the property of _uninfluencability_, in which we must be able to posit some background variables in the environment such that the learning process can be said to be "learning" these variables. Technically, an unriggable process need not be uninfluenceable, though it usually is (see the paper for details).

However, these properties only constrain the _expectation over environments_ of the final reward distribution: it doesn't prevent the agent from somehow shuffling around reward functions to be matched with suitable environments. For example, without knowing which projects are easy or hard, Sandra could manipulate Trisha into giving early deadlines for easy projects, and late deadlines for hard projects, in a manner that preserved the _distribution_ over early and late deadlines; this would satisfy the unriggable property (and probably also the uninfluenceable property, depending on the exact formalization).

The authors demonstrate these problems in a simple gridworld example. They also point out that there's a simple way to make any learning process uninfluenceable: choose a specific policy π that gathers information about the reward, and then define the new learning process to be "whatever the original learning process would have said if you executed π".


I like this framework for a couple of reasons. First, it allows for multiple kinds of queries, which correspond to different ways of specifying tasks, increasing the number of types of inputs we can give in order to communincate our intent to an AI. Concretely, the framework can handle both demonstrations (as in IRL) and examples of goal states. Second, it reduces learning a reward function to learning the probabilities of events, which has been studied in much more depth in the machine learning community and so will hopefully work better.

Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition

Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, Sergey Levine

For reinforcement learning, we can create a probabilistic model in which there are events for the state the agent is in and the action the agent takes. We can also add events e_t corresponding roughly to "the agent achieved something good in timestep t". We set P(e_t = 1 | s_t, a_t) to be exp(R(s_t, a_t)). Then, we can simply set all of the e_t to 1, and infer the likely state-action pairs that would have led to that. This leads to maximum entropy reinforcement learning, which in the setting of deterministic dynamics is equivalent to soft Q-learning. The authors then note that in this setup, the reward corresponds to the log probability of event e_t happening. So, instead of specifying a reward function, we can instead define binary events that we care about, model their probability of occurring, and then find the actions that maximize the likelihood of the event occurring. The authors derive backup equations for three kinds of queries -- ALL (the event must happen every timestep), AT (the event happens at a particular timestep), and ANY (the event happens on some timestep).

In this setup, specifying a reward function corresponds to explicitly writing down probabilities P(e | s, a). Of course, we can learn these probabilities from data using standard ML techniques, and this now corresponds to learning a reward function! If we use the ALL query, this corresponds to inverse reinforcement learning. However, by using the AT or ANY query instead, we only require examples of the event e_t for a single s_t and a_t -- for example, images that represent a goal state. They derive an algorithm for this query and show experimentally that this framework can learn event probabilities that lead to good behavior on Mujoco environments.


The formulation of a POMDP with uncertainty over the goal is remarkably similar to the formulation of <@Cooperative Inverse Reinforcement Learning@> (and predates it), with the main difference being that there is only one actor (the robot hardware).

Shared Autonomy via Hindsight Optimization

Shervin Javdani, Siddhartha S. Srinivasa, J. Andrew Bagnell

This paper considers a shared autonomy task in which a user controls a robot to achieve some goal, and the robot learns to assist the user, without knowing the goal in advance. They formalize this as a POMDP in which the state includes the user's goal, which the robot does not get to observe. However, the POMDP observation model assigns higher probability to user actions that better achieve the goal (a standard Boltzmann rationality model), and this allows the agent to reason about what the goal must be. In practice, for computational tractability, rather than choosing optimal actions in the overall POMDP, the robot chooses optimal actions using a technique called hindsight optimization, which _assumes that the robot will never learn more information about the user's goal_.


This seems like a good paper, and seems very much like the natural extension of Boltzmann models to include accounting for similar trajectories. As the paper notes, I largely worry about the reliance on a pre-specified set of robot features-- in more complicated cases of inference, it could be impractical to hand-specify relevant features and too difficult to have the robot infer them. In the worst case, it seems like misspecified features could make performance worse than Boltzmann via suggesting similarities that are irrelevant.

**Rohin's opinion:** (Note that this paper comes from the InterACT lab, which I am a part of.)

The Boltzmann model of human behavior has several theoretical justifications: it's the maximum entropy (i.e. [minimum encoded information](https://en.wikipedia.org/wiki/Principle_of_maximum_entropy)) distribution over trajectories subject to the constraint that the feature expectations match those of the observed human behavior; it's the maximum entropy distribution under the assumption that humans satisfice for expected reward above some threshold, etc. I have never found these very compelling, and instead see it as something far simpler: you want your model to encode the fact that humans are more likely to take good actions than bad actions, and you want your model to assign non-zero probability to all trajectories; the Boltzmann model is the simplest model that meets these criteria. (You could imagine removing the exponential in the model as "even simpler", but this is equivalent to a monotonic transformation of the reward function.)

I view this paper as proposing a model that meets my two criteria before, and adds in a third one: when we can cluster trajectories based on similarity, then we should view the human as choosing between _clusters_, rather than choosing between trajectories. Given a good similarity metric, this seems like a much better model of human behavior -- if I'm walking and there's a tree in my path, I will choose which side of the tree to go around, but I'm not going to put much thought into exactly where my footsteps will fall.

I found the claim that Boltzmann overlearns in sparse areas to be unintuitive, and so I delved into it deeper in this [comment](https://www.alignmentforum.org/posts/TmHRACaxXrLbXb5tS/rohinmshah-s-shortform?commentId=Hb6m2C9wtKWTzYL8d). My overall takeaway was that the claim will often hold in practice, but it isn't guaranteed.

LESS is More: Rethinking Probabilistic Models of Human Behavior

Andreea Bobu*, Dexter R.R. Scobee*, Jaime F. Fisac, S. Shankar Sastry, Anca D. Dragan

This paper introduces a new model for robots inferring human preferences called LESS. The traditional Boltzmann noisily-rational decision model assumes people approximately optimize a reward function and choose trajectories in proportion to their exponentiated reward. The Boltzmann model works well when modeling decisions among different discrete options, but runs into problems when modeling human trajectories in a continuous space, e.g. path finding, because it is very sensitive to the number of trajectories, even if they are similar-- if a robot using a Boltzmann model must predict whether a human navigates around an obstacle by taking one path on the left or one of three very-similar paths on the right, it will assign the same probability to each path by default.

To fix this, LESS predicts human behavior by treating each trajectory as part of a continuous space and mapping each one to a feature vector. The likelihood of selecting a trajectory is inversely proportional to its feature-space similarity with other trajectories, meaning similar trajectories are appropriately deweighted.

The paper tests the predictive performance of LESS vs. Boltzmann in several experimental environments, including an artifically constructed task where humans are asked to choose between similar paths for navigating around an obstacle, and a real-world task where humans demonstrate appropriate behaviors to a 7-degree-of-freedom robotic arm. In general, LESS performs better than Boltzmann when given a small number of samples of human behavior, but does equally well as the sample size is increased. In the robotic arm task, Boltzmann performed better when demonstrations were aggregated into a single batch and inference was run on the whole batch at once, representing trying to approximate the 'average' user rather than customizing behavior to each user. The paper claims that this happens because Boltzmann overlearns from demonstrations in sparse regions, and underlearns from dense demonstrations. As you increase the number of samples, you approximate the “true” trajectory space better and better, so the 10 trajectory sets vary less and less, which means Boltzmann won’t underperform so much. Since the single batch demonstration aggregated demonstrations, it had a similar effect in approximating the "true" trajectory space.

The paper notes that one limitation of this method is a reliance on a pre-specified set of robot features, though a small set of experimental results suggested that LESS still performed better than Boltzmann when adding a small number of irrelevant features.


Since this work "integrates ideas from context-based meta-learning, deep latent variable generative models, and maximum entropy inverse RL" and covers the relevant mathematics, it is an involved, if rewarding, study into multi-task IRL. I am convinced that this is a big step forward for IRL, but I'd be interested in seeing comparisons on setups that are more complicated.

'Data efficiency' is implied as a desirable quality, and the paper makes a case that they learn from a limited number demonstrations at meta-test time. However, it does not specify how many demonstrations were required for each task during _meta-training_. Additionally, for two environments, _tens of millions_ of environment interactions were required, which is entirely infeasible for real systems.

Meta-Inverse Reinforcement Learning with Probabilistic Context Variables

Lantao Yu*, Tianhe Yu*, Chelsea Finn, Stefano Ermon

This work explores improving performance on multi-task inverse reinforcement learning in a single-shot setting by extending <@Adversarial Inverse Reinforcement Learning@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@) with "latent context variables" that condition the learned reward function. The paper makes two notable contributions: 1) It details an algorithm to simultaneously learn a flexible reward function and a conditional policy with competitive few-shot generalization abilities from expert demonstrations of multiple related tasks _without_ task specifications or identifiers; 2) The authors empirically demonstrate strong performance of a policy trained on the inferred reward of a structurally similar task with modified environmental dynamics, claiming that in order to succeed "the agent must correctly infer the underlying goal of the task instead of simply mimicking the demonstration".


Workshop on Safety and Robustness in Decision Making at NeurIPS 2019

The idea of using preference orderings (Bradley-Terry) to speed up the posterior probability calculation was ingenious. While B-REX isn't strictly better than T-REX in terms of rewards achieved, the ability to construct confidence intervals for performance is a major benefit. My takeaway is that Bayesian IRL is getting more efficient and may have good potential as a practical approach to safe value learning.

Deep Bayesian Reward Learning from Preferences

Bayesian inverse reinforcement learning (IRL) is ideal for safe imitation learning since it allows uncertainty in the reward function estimator to be quantified. This approach requires thousands of likelihood estimates for proposed reward functions. However, each likelihood estimate requires training an agent according to the hypothesized reward function. Predictably, such a method is computationally intractable for high dimensional problems.

**In this paper, the authors propose Bayesian Reward Extrapolation (B-REX), a scalable preference-based Bayesian reward learning algorithm.** They note that in this setting, a likelihood estimate that requires a loop over all demonstrations is much more feasible than an estimate that requires training a new agent. So, they assume that they have a set of _ranked_ trajectories, and evaluate the likelihood of a reward function by its ability to reproduce the preference ordering in the demonstrations. To get further speedups, they fix all but the last layer of the reward model using a pretraining step: the reward of a trajectory is then simply the dot product of the last layer with the features of the trajectory as computed by all but the last layer of the net (which can be precomputed and cached once).

The authors test B-REX on pixel-level Atari games and show competitive performance to <@T-REX@>(@Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations@), a related method that only computes the MAP estimate. Furthermore, the authors can create confidence intervals for performance since they can sample from the reward distribution.


I like the focus on integrating active selection of trajectory queries into reward model training, and especially the four different kinds of active criteria that they consider, and the detailed experiments (including an ablation study) on the benefits of these criteria. These seem important for improving the efficiency of reward modeling.

However, I don't buy the argument that this allows us to train an agent without visiting unsafe states. In their actual experiments, they use a dataset gathered from a random policy, which certainly will visit unsafe states. If you instead use a dataset of safe human demonstrations, your generative model will only place probability mass on safe demonstrations, and so you'll never generate trajectories that visit unsafe states, and your reward model won't know that they are unsafe. (_Maybe_ your generative model will generalize properly to the unsafe states, but that seems unlikely to me.) Such a reward model will either be limited to imitation learning (sticking to the same trajectories as in the demonstrations, and never finding something like AlphaGo's move 37), or it will eventually visit unsafe states.

Paper: Learning Human Objectives by Evaluating Hypothetical Behavior

Learning human objectives by evaluating hypothetical behaviours

Siddharth Reddy, Anca D. Dragan, Sergey Levine, Shane Legg, Jan Leike

[Deep RL from Human Preferences](https://deepmind.com/blog/learning-through-human-feedback/) updated its reward model by collecting human comparisons on on-policy trajectories where the reward model ensemble was most uncertain about what the reward should be. However, we want our reward model to be accurate off policy as well, even in unsafe states. To this end, we would like to train our reward model on _hypothetical_ trajectories. This paper proposes learning a generative model of trajectories from some dataset of environment dynamics, such as safe expert demonstrations or rollouts from a random policy, and then finding trajectories that are "useful" for training the reward model. They consider four different criteria for usefulness of a trajectory: _uncertain rewards_ (which intuitively are areas where the reward model needs training), _high rewards_ (which could indicate reward hacking), _low rewards_ (which increases the number of unsafe states that the reward model is trained on), and _novelty_ (which covers more of the state space). Once a trajectory is generated, they have a human label it as good, neutral, or unsafe, and then train the reward model on these labels.

The authors are targeting an agent that can _explore safely_: since they already have a world model and a reward model, they use a model-based RL algorithm to act in the environment. Specifically, to act, they use gradient descent to optimize a trajectory in the latent space that maximizes expected rewards under the reward model and world model, and then take the first action of that trajectory. They argue that the world model can be trained on a dataset of safe human demonstrations (though in their experiments they use rollouts from a random policy), and then since the reward model is trained on hypothetical behavior and the model-based RL algorithm doesn't need any training, we get an agent that acts without us ever getting to an unsafe state.


This paper goes into detail arguing why causal misidentification is a huge problem in imitation learning and I find its argument compelling. I am excited about attempts to address the problem, and I am tentatively excited about the method the paper proposes for finding representative causal graphs, with the caveat that I don't feel equipped to evaluate whether it could efficiently generalize past the constrained experiments presented in the paper.

**Rohin's opinion:** While the conclusion that more information hurts sounds counterintuitive, it is actually straightforward: you _don't_ get more data (in the sense of the size of your training dataset); you instead have _more features_ in the input state data. This increases the number of possible policies (e.g. once you add the car dashboard, you can now express the policy "if brake light is on, apply brakes", which you couldn't do before), which can make you generalize worse. Effectively, there are more opportunities for the model to pick up on spurious correlations instead of the true relationships. This would happen in other areas of ML as well; surely someone has analyzed this effect for fairness, for example.

The success of their method over DAgger comes from improved _policy exploration_ (for their environments): if your learned policy is primarily paying attention to the brake light, it's a very large change to instead focus on whether there is an obstacle visible, and so gradient descent is not likely to ever try that policy once it has gotten to the local optimum of paying attention to the brake light. In contrast, their algorithm effectively trains separate policies for scenarios in which different parts of the input are masked, which means that it is forced to explore policies that depend only on the brake light, and policies that depend only on the view outside the windshield, and so on. So, the desired policy has been explored already, and it only requires a little bit of active learning to identify the correct policy.

Like Asya, I like the approach, but I don't know how well it will generalize to other environments. It seems like an example of [quality diversity](https://www.frontiersin.org/articles/10.3389/frobt.2016.00040/full), which I am generally optimistic about.

Causal Confusion in Imitation Learning

Pim de Haan, Dinesh Jayaraman, Sergey Levine

This paper argues that _causal misidentification_ is a big problem in imitation learning. When the agent doesn't have a good model of what actions cause what state changes, it may mismodel the effects of a state change as a cause-- e.g., an agent learning to drive a car may incorrectly learn that it should turn on the brakes whenever the brake light on the dashboard is on. This leads to undesirable behavior where more information actually causes the agent to perform worse.

The paper presents an approach for resolving causal misidentification by (1) Training a specialized network to generate a "disentangled" representation of the state as variables, (2) Representing causal relationships between those variables in a graph structure, (3) Learning policies corresponding to each possible causal graph, and (4) Performing targeted interventions, either by querying an expert, or by executing a policy and observing the reward, to find the correct causal graph model.

The paper experiments with this method by testing it in environments artificially constructed to have confounding variables that correlate with actions but do not cause them. It finds that this method is successfully able to improve performance with confounding variables, and that it performs significantly better per number of queries (to an expert or of executing a policy) than any existing methods. It also finds that directly executing a policy and observing the reward is a more efficient strategy for narrowing down the correct causal graph than querying an expert.