I'm excited to see approaches to reward learning that use information from natural language -- that definitely seems like a rich source of information we have not made use of yet. That said, there are a lot of moving parts in this system and the parts that analyze natural language impose a lot of structure, to the extent that it feels like a domain-specific language instead of natural language. (I'm not confident about this, it's hard to tell from the paper.) I also don't understand the experiments. They say that the sparse reward couldn't train policies with episode length 3 -- sparse reward is a hard problem, but it isn't _that_ hard, so I must be misunderstanding something here.

Reward Learning from Narrated Demonstrations

Hsiao-Yu Fish Tung, Adam W. Harley, Liang-Kang Huang, Katerina Fragkiadaki

This paper learns and optimizes rewards given demonstrations of behavior along with a description of the behavior in natural language. Their dataset is a set of videos of humans demonstrating a task and describing it with natural language (such as "the orange is in the bowl"). They combine several techniques to use this dataset to teach a robot. First, using speech recognition, they get a transcript of the natural language aligned with the video. They use object detectors to figure out what things are present in the image, and a syntactic parser to figure out the subject and object of the sentence, and match up these two results to figure out which objects in the image the natural language refers to, and extract their spatial features. They then train a classifier to take the spatial features and detecting whether it has achieved the goal, conditioned on the natural language description of the task. Now that they have a reward function (1 at a goal state, 0 otherwise) they can train a robot using DQN, though to get this to work they infer 3D object configurations from 2D images and use distance to the goal as a shaped reward.


AI Alignment Podcast: Human Cognition and the Nature of Intelligence

Joshua Greene's lab has two research directions. The first is how we combine concepts to form thoughts: a process which allows us to understand arbitrary novel scenarios (even ones we don't think ever occurred). He discusses some of his recent reseach, which uses brain imaging to infer what's happening when humans think about compound concepts. While Joshua considers the combinatorial nature of thought to be important, he argues that to build AGI, it's necessary to start with "grounded cognition" in which representations are derived from perception and physical action, rather than just learning to manipulate symbols (like language).

Joshua also works on the psychology and neuroscience of morality. He discusses his recent work in which participants are prompted to consider Rawls' Veil of Ignorance argument (that when making decisions affecting many people, we should do so as if we don't know which one we are) and then asked to evaluate moral dilemmas such as trolley problems. Joshua argues that the concept of impartiality is at the core of morality, and that it pushes people towards more utilitarian ideas (although he wants to rebrand utilitarianism as "deep pragmatism" to address its PR problems).


I found the categorization useful for understanding the differences between views on human values, which can be quite varied and hard to compare.

What AI Safety Researchers Have Written About the Nature of Human Values

This post categorizes theories of human values along three axes. First, how complex is the description of the values? Second, to what extent are "values" defined as a function of behavior (as opposed to being a function of eg. the brain's algorithm)? Finally, how broadly applicable is the theory: could it apply to arbitrary minds, or only to humans? The post then summarizes different positions on human values that different researchers have taken.


This is a cool example of how huge amounts of existing unlabeled video data might be utilised. The task they attempt is significantly more complex than those in other similar work (such as [this paper](https://arxiv.org/abs/1805.11592) which learns to play Atari games from Youtube videos); however, this also makes it difficult to judge how well the learned policy performed, and how much potential it has to transfer into the real world.

Learning from Demonstration in the Wild

Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu, João Gomes, Supratik Paul, Frans A. Oliehoek, João Messias, Shimon Whiteson

This paper learns traffic trajectories from unsupervised data by converting traffic camera footage into a Unity scene simulation, using that simulation to generate pseudo-LIDAR readings for each "expert trajectory", and then training an agent to imitate them using a variant of generative adversarial imitation learning (GAIL).


Shared autonomy is an interesting setting because the human is still necessary in order to actually perform the task, whereas in typical reward learning settings, once you have learned the reward function and the AI is performing well, the human does not need to be present in order to execute a good policy.

Shared Autonomy via Deep Reinforcement Learning

In shared autonomy, an AI system assists a human to complete a task. The authors implement shared autonomy in a deep RL framework by simply extending the state with the control input from the human, and then learning a policy that chooses actions given the extended state. They show that the human-AI team performs better than either one alone in the Lunar Lander environment.


The key insight that rewards are _not_ invariant to additions of a constant when you have variable-length episodes is useful and I'm glad that it's been pointed out, and a solution proposed. However, the experiments are really strange -- in one case (Figure 4, HalfCheetah) their algorithm outperforms the expert (which has access to the true reward), and in another (Figure 5, right) the blue line implies that using a uniformly zero reward lets you achieve around a third of expert performance (!!).

Addressing Sample Inefficiency and Reward Bias in Inverse Reinforcement Learning

Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, Jonathan Tompson

Deep IRL algorithms typically work by training a discriminator that distinguishes between states and actions from the expert from states and actions from the learned policy, and extracting a reward function from the discriminator. In any environment where the episode can end after a variable number of timesteps, this assumes that the reward is zero after the episode ends. The reward function from the discriminator often takes a form where it must always be positive, inducing a survival incentive, or a form where it must always be negative, inducing a living cost. For example, [GAIL](https://arxiv.org/abs/1606.03476)'s reward is always positive, giving a survival incentive. As a result, _without any reward learning at all_ GAIL does better on Hopper than behavioral cloning, and fails to learn on a reaching or pushing task (where you want to do the task as quickly as possible, so you want the living cost). To solve this, they learn an "absorbing state reward", which is a reward given after the episode ends -- this allows the algorithm to learn for itself whether it should have a survival incentive or living cost.

They also introduce a version that keeps a replay buffer of experience and uses an off-policy algorithm to learn from the replay buffer in order to improve sample efficiency.


Active Inverse Reward Design

Sören Mindermann, Rohin Shah, Adam Gleave, Dylan Hadfield-Menell

Inverse Reward Design (IRD) introduced the idea that a hand-designed reward function should be treated as an _observation_ about the intended reward function. Any particular hand-designed reward function (called a _proxy reward_) is likely if it incentivizes good behavior in the training environment, as measured by the true reward function. In this paper, instead of asking the reward designer to choose a proxy reward from the space of all possible reward functions, the designer is presented with a small subset of possible reward functions and asked to choose the best option from those. The subset is chosen such that the answer will be maximally informative about the true reward (in expectation). This is an easier query for the reward designer to answer, and can convey more information about the true reward function. To see this, we can imagine pairing each possible reward function with the trajectory that it incentivizes in the training environment. Then original IRD gets information about the best such trajectory, which could correspond to multiple true reward functions, whereas active IRD can get information about the best trajectory in any subset of trajectories, and so can get more information in total. In some cases, that extra information can narrow down the space of possible reward functions to a single true reward function. The paper discusses two kinds of queries that can be asked (discrete and continuous), and several optimizations for actually computing the best query to ask (which can be computationally intensive). The technique is demonstrated on a contextual bandits problem and gridworlds.


I only skimmed through the math, so I don't understand the paper well enough to have a good opinion on it. The overall objective of having more risk-sensitivity seems useful for safety. That said, I do find the VNM utility theorem compelling, and it suggests that risk aversion is a bad strategy. I currently resolve this by saying that while the VNM theorem is true, if you want to optimize expected reward over a long time horizon in an environment with high-downside actions but not high-upside actions, even if you are maximizing expected utility you would not take low-probability-of-high-downside actions. (Here a high-downside action is one that causes something like death/episode termination.) Since humans are (probably) scope-insensitive with respect to time, it becomes important for humans to have a heuristic of risk aversion in order to actually maximize expected utility in practice. I'd be interested in seeing experiments with current (risk neutral) RL algorithms in long-horizon environments with actions with high downside, and see if they automatically learn behavior that we would call "risk-averse".

Take this with a grain of salt -- it's a lot more speculative than most of my opinions, which can already be quite speculative. Most of the steps in that argument are handwavy intuitions I have that aren't based on any research that's been done (though I haven't looked for any such research). Though you can think of the argument for focusing on long-term AI safety at all as an instance of this idea, where the argument is that our risk-aversion heuristic is only sufficient for timescales on the orders of human lifetimes, not for cosmic timescales, and so we should explicitly be more risk-averse and focus on reducing existential risk.

Risk-Sensitive Generative Adversarial Imitation Learning

Jonathan Lacotte, Mohammad Ghavamzadeh, Yinlam Chow, Marco Pavone

This paper extends GAIL to perform imitation learning where we try to optimize a policy for the mean reward collected under the constraint that the policy is no more risky than the expert policy. Since we don't know the true cost function, we have to approximate this problem with another problem where we infer the cost function as well, and evaluate the risk profile relative to the inferred cost function. The algorithm ends up looking very similar to the original GAIL algorithm, where the gradient updates change in order to include terms dependent on the conditional value-at-risk (CVaR). They evaluate against GAIL and RAIL (another risk-sensitive imitation learning algorithm) and find that their method performs the best on the Hopper and Walker Mujoco environments.


One thing to note about this paper is that it is an incredible work of scholarship; it fluently cites research across a variety of disciplines, including AI safety, and provides a useful organizing framework for many such papers. If you need to do a literature review on inverse reinforcement learning, this paper is a good place to start.

Inverse Decision Modeling: Learning Interpretable Representations of Behavior

Daniel Jarrett*, Alihan Hüyük*, Mihaela van der Schaar

There's lots of work on learning preferences from demonstrations, which varies in how much structure they assume on the demonstrator: for example, we might consider them to be <@Boltzmann rational@>(@Modeling Interaction via the Principle of Maximum Causal Entropy@) or [risk sensitive](https://rss2017.lids.mit.edu/static/papers/62.pdf), or we could try to <@learn their biases@>(@Learning biases and rewards simultaneously@). This paper proposes a framework to encompass all of these choices: the core idea is to model the demonstrator as choosing actions according to a _planner_; some parameters of this planner are fixed in advance to provide an assumption on the structure of the planner, while others are learned from data. This also allows them to separate beliefs, decision-making, and rewards, so that different structures can be imposed on each of them individually.

The paper provides a mathematical treatment of both the forward problem (how to compute actions in the planner given the reward, think of algorithms like value iteration) and the backward problem (how to compute the reward given demonstrations, the typical inverse reinforcement learning setting). They demonstrate the framework on a medical dataset, where they introduce a planner with parameters for flexibility of decision-making, optimism of beliefs, and adaptivity of beliefs. In this case they specify the desired reward function and then run backward inference to conclude that, with respect to this reward function, clinicians appear to be significantly less optimistic when diagnosing dementia in female and elderly patients.


One way this paper differs from my intuitions is that it finds that assuming Boltzmann rationality performs very poorly if the demonstrator is in fact systematically suboptimal. I would have instead guessed that Boltzmann rationality would do okay -- not as well as in the case where there is no misspecification, but only a little worse than that. (That’s what I found in <@my paper@>(@Learning biases and rewards simultaneously@), and it makes intuitive sense to me.) Some hypotheses for what’s going on, which the lead author agrees are at least part of the story:

1. When assuming Boltzmann rationality, you infer a distribution over reward functions that is “close” to the correct one in terms of incentivizing the right behavior, but differs in rewards assigned to suboptimal behavior. In this case, you might get a very bad log loss (the metric used in this paper), but still have a reasonable policy that is decent at acquiring true reward (the metric used in my paper).

2. The environments we’re using may differ in some important way (for example, in the environment in my paper, it is primarily important to identify the goal, which might be much easier to do than inferring the right behavior or reward in the autonomous driving environment used in this paper).

Human irrationality: both bad and good for reward inference

Lawrence Chan, Andrew Critch, Anca Dragan

Last summary, we saw a framework for inverse reinforcement learning with suboptimal demonstrators. This paper instead investigates the qualitative effects of performing inverse reinforcement learning with a suboptimal demonstrator. The authors modify different parts of the Bellman equation in order to create a suite of possible suboptimal demonstrators to study. They run experiments with exact inference on random MDPs and FrozenLake, and with approximate inference on a simple autonomous driving environment, and conclude:

1. **Irrationalities can be helpful for reward inference**, that is, if you infer a reward from demonstrations by an irrational demonstrator (where you know the irrationality), you often learn _more_ about the reward than if you inferred a reward from optimal demonstrations (where you know they are optimal). Conceptually, this happens because optimal demonstrations only tell you about what the best behavior is, whereas most kinds of irrationality can also tell you about preferences between suboptimal behaviors.

2. **If you fail to model irrationality, your performance can be very bad**, that is, if you infer a reward from demonstrations by an irrational demonstrator, but you assume that the demonstrator was Boltzmann rational, you can perform quite badly.


NeurIPS 2021 Track Datasets and Benchmarks

While the authors do a good job advocating for the problem of preference-based RL, I'm less convinced their particular benchmark is a large step forward. In particular, it seems the main contribution is not a suite of tasks, but rather a collection of different ways to add irrationality to the teacher oracle. The main takeaway of this paper is that current algorithms don't seem to perform well when the teacher can make mistakes, but this is quite similar to having a misspecified reward function. Beyond that criticism, the experiments support the areas suggested for future work.

B-Pref: Benchmarking Preference-Based Reinforcement Learning

Kimin Lee, Laura Smith, Anca Dragan, Pieter Abbeel

Deep RL has become a powerful method to solve a variety of sequential decision tasks using a known reward function for training. However, in practice, rewards are hard to specify making it hard to scale Deep RL for many applications. Preference-based RL provides an alternative by allowing a teacher to indicate preferences between a pair of behaviors. Because the teacher can interactively give feedback to an agent, preference-based RL has the potential to help address this limitation of Deep RL. Despite the advantages of preference-based RL it has proven difficult to design useful benchmarks for the problem. This paper introduces a benchmark (B-Pref) that is useful for preference-based RL in various locomotion and robotic manipulation tasks.

One difficulty with designing a useful benchmark is that teachers may have a variety of irrationalities. For example, teachers might be myopic or make mistakes. The B-Pref benchmark addresses this by emphasizing measuring performance under a variety of teacher irrationalities. They do this by providing various performance metrics to introduce irrationality into otherwise deterministic reward criteria. While previous approaches to preference-based RL work well when the teacher responses are consistent, experiments show they are not robust to feedback noise or teacher mistakes. Experiments also show that how queries are selected has a major impact on performance. With these results, the authors identify these two problems as areas for future work.


This seems like a reasonable approach. It has a similar ethos as Boltzmann rationality. In Boltzmann rationality, it seems like all you need to do is model the demonstrator as having some noise but still being more likely to choose higher-reward actions, and that’s enough to get decent performance; similarly here you just need to model different demonstrators as applying different amounts of Gaussian noise to the optimal policy and that’s enough to distinguish good from bad.

Note that, while the experimental results are good, the paper doesn’t have experiments with real human demonstrations; as we saw in the previous summary these can often be quite different (in ways that matter) from machine-generated demonstrations.

VILD: Variational Imitation Learning with Diverse-quality Demonstrations

Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, Masashi Sugiyama

We saw in the previous summary that existing methods struggle to cope with datasets of demonstrations of mixed quality. This paper aims to tackle exactly this problem. They consider a model in which there are k demonstrators with varying levels of quality. Each demonstrator is modeled as computing an action Boltzmann-rationally and then applying some Gaussian noise; the standard deviation of the Gaussian noise differs across the demonstrators (with higher standard deviation corresponding to lower quality).

They use variational inference to derive an algorithm for this problem that infers the reward function as well as an optimal policy to go along with it. In addition, they oversample data from the demonstrations that the model thinks are high quality in order to get more informative gradients. (They use an importance sampling correction in order to keep the gradient estimate unbiased.)

Their experiments on machine-generated data show significant improvement over existing imitation learning algorithms, both in the case where we synthetically add Gaussian noise (matching the model) and when we add time-signal-dependent (TSD) noise (in which case the model is misspecified).


I found the experimental comparisons to be a bit misleading. If you compare with the results in the original ValueDICE and SQIL paper the algorithms are closer in performance than this paper implies. It's also not clear that you need to use the soft-Bellman operator especially in the continuous-control setting which was what ValueDICE originally focused on. However, overall, non-adversarial methods are generally more stable so I found this paper to be a good contribution.

IQ-Learn: Inverse soft-Q Learning for Imitation

Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, Stefano Ermon

A popular way to view imitation learning is as a distribution matching problem. In this approach, the goal is to have the imitator induce a state-action distribution that closely matches that of the expert. Methods such as <@GAIL@>(@Generative Adversarial Imitation Learning@) and <@Value-DICE@>(@Imitation Learning via Off-Policy Distribution Matching@) propose adversarial methods, similar to GANs, to carry out the distribution matching. However, such methods can be difficult to train due to the difficulty of solving saddle-point problems. In this paper, the authors present a non-adversarial method that allows distribution matching to be carried out in a fully offline and non-adversarial fashion. They do this by building on Value-DICE and introducing a soft-Bellman operator which allows the saddle-point problem to be reduced to estimating a Q-function. In fact, the authors show this reduction is related to off-policy RL algorithms with the reward set to zero. In experiments, the method is shown to be competitive with other state-of-the-art methods in both the offline and image-based setting.


The paper is organized well and I found the examples to be interesting in their own right. On the other hand, binary classification is a fairly restrictive setting and IDT in this paper seems to require access to class posterior probabilities. These probabilities generally are not easy to estimate. Moreover, if you have access to that function it seems you could elicit the loss function with exponentially fewer human observations by sorting/sub-sampling the class posterior values. Despite these shortcomings, I'm interested to see how this work can be extended further.

Uncertain Decisions Facilitate Better Preference Learning

Human preference learning has been studied from various perspectives such as inverse reinforcement learning (IRL) and active learning. However, the IRL problem is underspecified, that is, even with access to the full behavioral policy, you cannot uniquely determine the preferences that led to that policy. Meanwhile, active learning often has a **description-experience gap**: the stated preferences in response to a question in active learning may not be the same as the preferences that would be revealed from demonstrations.

In this work, the authors study an alternative paradigm known as inverse decision theory (IDT) that aims to learn a loss function for binary classification using strictly observational data while returning unique solutions. (Such a loss function effectively specifies how good correct predictions are and how bad incorrect predictions are.) The authors show that preferences can be uniquely determined whenever there is uncertainty in the classification problem. This happens because we need observations predicting classes at different levels of certainty to identify a transition point where we switch from predicting one class over another. In contrast, without uncertainty, we won't be able to precisely identify that threshold. The authors then strengthen this result by showing it holds even in cases where the underlying decision rule is sub-optimal.

The authors argue that since learning could be done efficiently in this setting, IDT could have broader applicability. For example, one application to fairness could be to collect a set of decisions from a trained classifier, split them across groups (e.g. race or gender), and compare the inferred loss functions to detect bias in the trained classifier.


[Identifiability in inverse reinforcement learning](https://arxiv.org/abs/2106.03498) has the same motivation and studies a very similar setting, but has a few different results. It's also easier to read if you're not as familiar with MaxEnt methods.

Reward Identification in Inverse Reinforcement Learning

Kuno Kim, Kirankumar Shiragur, Shivam Garg, Stefano Ermon

As mentioned in the previous summary, a major challenge with inverse reinforcement learning is that rewards are unidentifiable: even given perfect knowledge of the policy, we cannot recover the reward function that produces it. This is partly for boring reasons like "you can add a constant to a reward function without changing anything", but even if you exclude those kinds of reasons, others remain. For example, since every policy is optimal for the constant reward function, the zero reward function can rationalize any policy.

For this reason, the authors instead focus on the case where we assume the policy is a solution to the maximum entropy RL objective (you can think of this as Boltzmann rationality, if you're more familiar with that). The solution to MaxEnt RL for a zero reward is a uniformly random policy, so the zero reward no longer rationalizes every policy. Perhaps rewards are identifiable in this case?

(You might have noticed that I neglected the question of whether the MaxEnt RL model was better than the regular RL model in cases that we care about. As far as I can tell the paper doesn't address this. But if they did so, perhaps they might say that in realistic situations we are dealing with boundedly-rational agents, and Boltzmann rationality / MaxEnt RL is a common model in such situations.)

Well, we still need to deal with the "additive constant" argument. To address this, the authors define two reward functions to be equivalent if they agree up to an additive constant. There are actually two versions of this: "trajectory equivalence" means that they agree on the rewards for all feasible trajectories, while "state-action equivalence" means that they agree on the rewards for all state-action pairs. Correspondingly, "weak identifiability" means that you can identify rewards up to trajectory equivalence, while "strong identifiability" means you can identify them up to state-action equivalence. Strong identifiability implies weak identifiability, since if you know the rewards on state-action pairs, that determines the reward for any given trajectory.

All deterministic MDPs are weakly identifiable under the MaxEnt RL model, since in this case a trajectory τ is selected with probability p(τ) proportional to exp(r(τ)), so the probability p(τ) can then be inverted to get r(τ). However, stochastic MDPs need not be weakly identifiable. Imagine an MDP in which no matter what you do, you are teleported to a random state. In such an MDP, the agent has no control over the trajectory, and so the MaxEnt RL objective will choose a uniformly random policy, no matter what the reward is, and so the reward must be unidentifiable.

Now the question is, assuming you have weak identifiability (i.e. you can infer r(τ)), when do you also have strong identifiability (i.e. you can infer r(s, a))? Intuitively, there needs to be a sufficient "diversity" of feasible trajectories τ, that cover a wide variety of possible (s, a) pairs, so that you can use the r(τ) values to infer the r(s, a) values. The authors prove a sufficient condition called "coverage": there exists some timestep T, such that for every state there is some feasible trajectory that reaches that state at timestep T. (They also require the horizon to be at least 2T.) Coverage can be a fairly easy property to have; for example, if you can get to any state from any other state in some number of steps, then all you need is a single self-loop somewhere in the MDP that allows you to "waste time" so that you reach the desired state at exactly timestep T (instead of reaching too early).


Hierarchy seems to be a very important tool that humans use, so I'm glad to see work on it. Currently, the algorithm is very computationally expensive, and can only be applied in small domains right now, and requires the options to be specified ahead of time, but it does lead to a benefit on the environments they consider, despite the inevitable misspecification from having to hardcode the options. I would be very interested to see an extension to high-dimensional data where the options are learned (analogous to [Meta-Learning Shared Hierarchies](https://blog.openai.com/learning-a-hierarchy/) for hierarchical RL). Not only would this be more realistic, it could perform better because the options would be learned, not hardcoded.

Exploring Hierarchy-Aware Inverse Reinforcement Learning

One heuristic that humans use to deal with bounded computation is to make plans hierarchically, building long-term plans out of slightly smaller building blocks. How can we incorporate this knowledge into an IRL algorithm? This paper extends [Bayesian IRL](https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-416.pdf) to the setting where the demonstrator has access to a set of _options_, which are (to a first approximation) policies that can be used to achieve some subgoal. Now, when you are given a trajectory of states and actions, it is no longer clear which options the demonstrator was using to generate that trajectory. The authors provide an algorithm that can enumerate all the options that are consistent with the trajectory, and assign probabilities to them according to the Boltzmann-rational model. They evaluate on a taxi driver gridworld often used in hierarchical planning, as well as on real human data from a game called Wikispeedia.


We can look at this from an alignment perspective -- the child is giving the AI system a misspecified reward, relative to the parent's goal of "provide good suggestions that do not have inappropriate content". While the researchers solve it using contextual bandits, it could be interesting to consider how AI alignment approaches could deal with this situation.

Using Contextual Bandits with Behavioral Constraints for Constrained Online Movie Recommendation

IBM researchers train AI to follow code of ethics

Parents want movie recommendation systems not to recommend particular kinds of movies to children, but we would also like the recommendation system to suggest movies that the children will actually like. Researchers solved this problem by first learning a model for what kinds of movies should not be recommended, and then combined that with a contextual bandit model that learns online from the child's data to provide good suggestions that follow the parent's constraints.


Ultimately my guess is that if you want to predict human judgments well, you need to train against human judgments, rather than the proxy task of distinguishing between human and agent behavior. That being said, I do think these proxy tasks can serve as valuable pretraining objectives, or as auxiliary objectives that help to improve sample efficiency.

Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation

Sam Devlin*, Raluca Georgescu*, Ida Momennejad*, Jaroslaw Rzepecki*, Evelyn Zuniga*, Gavin Costello, Guy Leroy, Ali Shaw, Katja Hofmann

Since rewards are hard to specify, we are likely going to have to train AI agents using human feedback. However, human feedback is particularly expensive to collect, so we would like to at least partially automate this using reward models. This paper looks at one way of building such a reward model: training a classifier to distinguish between human behavior and agent behavior (i.e. to be the judge of a Turing Test). This is similar to the implicit or explicit reward model used in adversarial imitation learning algorithms such as <@GAIL@>(@Generative Adversarial Imitation Learning@) or <@AIRL@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@).

Should we expect these classifiers to generalize, predicting human judgments of how human-like a trajectory is on all possible trajectories? This paper conducts a user study in order to answer the question: specifically, they have humans judge several of these Turing Tests, and see whether the classifiers agree with the human judgments. They find that while the classifiers do agree with human judgments when comparing a human to an agent (i.e. the setting on which the classifiers were trained), they do not agree with human judgments when comparing two different kinds of artificial agents. In fact, it seems like they are _anti-correlated_ with human judgments, rather than simply having no correlation at all -- only one of the six classifiers tested does better than chance (at 52.5%), the median is 45%, and the worst classifier gets 22.5%. (Note however that the sample size is small, I believe n = 40 though I'm not sure.)


[Paper: Learning What To Do by Simulating the Past](https://arxiv.org/abs/2104.03946)

[Thesis: Extracting and Using Preference Information from the State of the World](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-210.html)

David Lindner, Rohin Shah, Pieter Abbeel, Anca Dragan

Since the state of the world has already been optimized for human preferences, it can be used to infer those preferences. For example, it isn't a coincidence that vases tend to be intact and on tables. An agent with an understanding of physics can observe that humans haven't yet broken a particular vase, and infer that they care about vases not being broken.

<@Previous work@>(@Learning Preferences by Looking at the World@) provides an algorithm, RLSP, that can perform this type of reasoning, but it is limited to small environments with known dynamics and features. In this paper (on which I am an author), we introduce a deep variant of the algorithm, called Deep RLSP, to move past these limitations. While RLSP assumes known features, Deep RLSP learns a feature function using self-supervised learning. While RLSP computes statistics for all possible past trajectories using dynamic programming, deep RLSP learns an inverse dynamics model and inverse policy to _simulate_ the most likely past trajectories, which serve as a good approximation for the necessary statistics.

We evaluate the resulting algorithm on a variety of Mujoco tasks, with promising results. For example, given a single state of a HalfCheetah balancing on one leg, Deep RLSP is able to learn a (noisy) policy that somewhat mimics this balancing behavior. (These results can be seen [here](https://sites.google.com/view/deep-rlsp).)


Paper: Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification

Recursive Classification: Replacing Rewards with Examples in RL

Benjamin Eysenbach, Sergey Levine, Ruslan Salakhutdinov

Previous work has suggested learning a reward model from examples of successfully solving the task. This paper suggests that rather than a two stage process of learning a reward model and then optimizing it using RL, we can instead directly learn a policy from the examples by building an equivalent of Bellman backups that apply directly to examples (rather than having to go through intermediate rewards). Their experiments show that this works well.


Four Motivations for Learning Normativity

We've <@previously seen@>(@Learning Normativity: A Research Agenda@) desiderata for agents that learn normativity from humans: specifically, we would like such agents to:

1. **Learn at all levels:** We don't just learn about uncertain values, we also learn how to learn values, and how to learn to learn values, etc. There is **no perfect loss function** that works at any level; we assume conservatively that Goodhart's Law will always apply. In order to not have to give infinite feedback for the infinite levels, we need to **share feedback between levels**.
2. **Learn to interpret feedback:** Similarly, we conservatively assume that there is **no perfect feedback**; so rather than fixing a model for how to interpret feedback, we want feedback to be **uncertain** and **reinterpretable**.
3. **Process-level feedback:** Rather than having to justify all feedback in terms of the consequences of the agent's actions, we should also be able to provide feedback on the way the agent is reasoning. Sometimes we'll have to judge the entire chain of reasoning with **whole-process feedback**.

This post notes that we can motivate these desiderata from multiple different frames:

1. _Outer alignment:_ The core problem of outer alignment is that any specified objective tends to be wrong. This applies at all levels, suggesting that we need to **learn at all levels**, and also **learn to interpret feedback** for the same reason. **Process-level feedback** is then needed because not all decisions can be justified based on consequences of actions.
2. _Recovering from human error:_ Another view that we can take is that humans don't always give the right feedback, and so we need to be robust to this. This motivates all the desiderata in the same way as for outer alignment.
3. _Process-level feedback:_ We can instead view process-level feedback as central, since having agents doing the right type of _reasoning_ (not just getting good outcomes) is crucial for inner alignment. In order to have something general (rather than identifying cases of bad reasoning one at a time), we could imagine learning a classifier that detects whether reasoning is good or not. However, then we don't know whether the reasoning of the classifier is good or not. Once again, it seems we would like to **learn at all levels**.
4. _Generalizing learning theory:_ In learning theory, we have a distribution over a set of hypotheses, which we update based on how well the hypotheses predict observations. **Process-level feedback** would allow us to provide feedback on an individual hypothesis, and this feedback could be **uncertain**. **Reinterpretable feedback** on the other hand can be thought of as part of a (future) theory of meta-learning.


Here, the task definition continues to be the reward function, and the human demonstration is used to help the agent effectively optimize the reward function. Such agents are still vulnerable to misspecified reward functions -- in fact, the agent discovers a bug in the emulator that wouldn't have happened if it was trying to imitate the human. I would still expect the agent to be more human-like than one trained with standard RL, since it only learns the environment near the human policy.

Learning Montezuma’s Revenge from a Single Demonstration

Montezuma's Revenge is widely considered to be one of the hardest Atari games to learn, because the reward is so sparse -- it takes many actions to reach the first positive reward, and if you're using random exploration, it will take exponentially many actions (in N, the number of actions till the first reward) to find any reward. A human demonstration should make the exploration problem much easier. In particular, we can start just before the end of the demonstration, and train the RL agent to get as much score as the demonstration. Once it learns that, we can start it at slightly earlier in the demonstration, and do it again. Repeating this, we eventually get an agent that can perform the whole demonstration from start to finish, and it takes time linear in the length of the demonstration. Note that the agent must be able to generalize a little bit to states "around" the human demonstration -- when it takes random actions it will eventually reach a state that is similar to a state it saw earlier, but not exactly the same, and it needs to generalize properly. It turns out that this works for Montezuma's Revenge, but not for other Atari games like Gravitar and Pitfall.


Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning

Valerie Chen, Abhinav Gupta, Kenneth Marino

It is particularly challenging for RL agents to perform hierarchical tasks when there is only a sparse reward. One natural piece of feedback in this setting is instructions in natural language specifying the different subtasks needed to solve the task. In particular, this paper assumes we have access to a dataset of human demonstrations paired with natural language instructions for each subtask that they complete.

We then have an architecture that first generates the language instruction for the current subtask given the final task and the current state, and then takes a low-level action computed from the current state and the language instruction. This is trained via imitation learning on the human demonstrations.

Using a small Minecraft-inspired gridworld, the authors show that the language generation is crucial for good generalization: if the agent is trained on "cobblestone block" and "iron ingot", then it is able to generalize to cobblestone ingot, _as long as_ it was trained to generate the language instruction as well. Intuitively, the combinatorial structure of language leads to better generalization than direct imitation on low-level actions.


My understanding is the MEM only handles the six natural language instructions used in the StarCraft minigame, and so is roughly equivalent to training six classifiers using the hardcoded programs to generate datasets. Thus, these two papers ultimately boil down to “decompose the task into six steps, train classifiers for these six steps, and then do RL where the reward function gives positive reward every time a particular step is marked as complete”.

However, this is primarily because the authors had to ground the natural language instructions themselves. If we could instead leverage a pretrained model which already grounds natural language, such as [CLIP](https://openai.com/blog/clip/), then it seems like this approach could in fact save a lot of human effort in specifying what the AI system should do.

A Narration-based Reward Shaping Approach using Grounded Natural Language Commands

Nicholas Waytowich, Sean L. Barton, Vernon Lawhern, Garrett Warnell

One way to specify what an AI system should do is to simply specify it in natural language. If we have some way to map natural language instructions to states, then we could turn natural language into a reward function and use RL to optimize it.

This paper proposes specifying a task by breaking it down into a sequence of steps to be completed. Given a mapping from natural language to states, they define a reward function that gives a positive reward every time the mapping detects that the agent has completed the next stage in the sequence of steps. They show that this outperforms vanilla reinforcement learning on a win/loss reward function in a StarCraft minigame.

For the mapping of language to states, the authors use a mutual embedding model (MEM) they developed in [previous work](https://arxiv.org/abs/1906.02671). The core idea is to write down programs that identify states matching a particular natural language instruction, use this to generate a dataset of states and the corresponding natural language instruction, and then training a model to map the natural language instructions to be "close to" the mappings of the states (which are produced by a CNN).


Learning Rewards from Linguistic Feedback

Theodore R. Sumers, Mark K. Ho, Robert D. Hawkins, Karthik Narasimhan, Thomas L. Griffiths

This paper proposes another approach to reinforcement learning using natural language. After the agent plays an episode, we can ask a human for feedback in natural language. We then take their response, figure out what features of the environment the response mentions, and then use sentiment analysis to determine how to update the weights on the features. For sentiment analysis we can use an off-the-shelf classifier; the hard part is in determining the relevant environment feature vectors:

1. **Evaluative** feedback is feedback about the trajectory the agent produced, for example "good job", so we can just use the features of this trajectory.
2. **Imperative** feedback specifies what the agent should have done, e.g. "you should have gone to the top right corner". In this case, we must find the features consistent with the given instruction.
3. **Descriptive** feedback provides feedback directly about the reward, for example "yellow objects are bad". In this case, we use a feature vector that has a 1 for every feature mentioned (in this case, the feature for yellow objects) and 0 everywhere else.

Types 2 and 3 require some domain knowledge in order to write down programs that map language to the relevant features. The environment the authors used was simple enough that they were able to do this.

Once we have the feature vector f and the sentiment s, we perform a Bayesian update on our weight distribution. This is similar to the way we perform Bayesian updates on the reward distribution upon seeing a human action as evidence, as in <@Bayesian IRL@>(@Bayesian Inverse Reinforcement Learning@) or <@reward-rational implicit choice@>(@Reward-rational (implicit) choice: A unifying formalism for reward learning@).

This model so far performs reasonably well. By adding a couple of heuristics inspired by pragmatics (e.g. assuming that features that aren't mentioned aren't decision-relevant), they reach approximately human-level performance.


Probing Emergent Semantics in Predictive Agents via Question Answering

Imitating Interactive Intelligence

Interactive Agents Group, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Stephen Clark, Andrew Dudzik, Petko Georgiev, Aurelia Guy, Tim Harley, Felix Hill, Alden Hung, Zachary Kenton, Jessica Landon, Timothy Lillicrap, Kory Mathewson, Alistair Muldal, Adam Santoro, Nikolay Savinov, Vikrant Varma, Greg Wayne, Nathaniel Wong, Chen Yan, Rui Zhu

While <@existing@>(@Learning to Follow Language Instructions with Adversarial Reward Induction@) <@work@>(@Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text@) has trained agents to follow natural language instructions, it may be the case that achieving AGI requires more interactivity: perhaps we need to train agents to both give and follow instructions, or engage in a full dialogue, to accomplish tasks in a 3-D embodied environment. This paper makes progress on this goal.

The authors introduce a 3-D room environment in which agents can interact with objects and move them around, leading to a combinatorial space of possible high-level actions. So far the authors have only worked on question-answering (e.g. "what is the color of the chair?") and instruction-following (e.g. "please lift up the purple object"), but they hope to eventually also work on dialogue and play.

They collect demonstrations of games between humans in which one human is given a goal, and then is asked to give a natural language instruction. The other human sees this instruction and must then execute it in the environment. The authors then use various kinds of imitation learning algorithms to learn a policy that can both set instructions and execute them. They also train models that can evaluate whether a particular trajectory successfully completes the goal or not.

The authors show that the learned policies are capable of some generalization -- for example, if during training they remove all rooms containing orange ducks (but don't remove other orange objects, or other colors of duck), the resulting policies are still able to handle rooms containing orange ducks.


Learning Normativity: A Research Agenda

To build aligned AI systems, we need to have our AI systems learn what to do from human feedback. However, it is unclear how to interpret that feedback: any particular piece of feedback could be wrong; economics provides many examples of stated preferences diverging from revealed preferences. Not only would we like our AI system to be uncertain about the interpretation about any particular piece of feedback, we would also like it to _improve_ its process for interpreting human feedback. This would come from human feedback on the meta-level process by which the AI system learns. This gives us _process-level feedback_, where we make sure the AI system gets the right answers _for the right reasons_.

For example, perhaps initially we have an AI system that interprets human statements literally. Switching from this literal interpretation to a Gricean interpretation (where you also take into account the fact that the human chose to say this statement rather than other statements) is likely to yield improvements, and human feedback could help the AI system do this. (See also [Gricean communication and meta-preferences](https://www.alignmentforum.org/posts/8NpwfjFuEPMjTdriJ/gricean-communication-and-meta-preferences), [Communication Prior as Alignment Strategy](https://www.alignmentforum.org/posts/zAvhvGa6ToieNGuy2/communication-prior-as-alignment-strategy), and [multiple related CHAI papers](https://www.alignmentforum.org/posts/zAvhvGa6ToieNGuy2/communication-prior-as-alignment-strategy?commentId=uWBFsKK6XFbL4Hs4z).)

Of course, if we learn _how_ to interpret human feedback, that too is going to be uncertain. We can fix this by "going meta" once again: learning how to learn to interpret human feedback. Iterating this process we get an infinite tower of "levels" of learning, and at every level we assume that feedback is not perfect and the loss function we are using is also not perfect.

In order for this to actually be feasible, we clearly need to share information across these various "levels" (or else it would take infinite time to learn across all of the levels). The AI system should not just learn to decrease the probability assigned to a single hypothesis, it should learn what _kinds_ of hypotheses tend to be good or bad.


**On feedback types:** It seems like the scheme introduced here is relying quite strongly on the ability of humans to give good process-level feedback _at arbitrarily high levels_. It is not clear to me that this is something humans can do: it seems to me that when thinking at the meta level, humans often fail to think of important considerations that would be obvious in an object-level case. I think this could be a significant barrier to this scheme, though it’s hard to say without more concrete examples of what this looks like in practice.

**On interaction:** I’ve previously <@argued@>(@Human-AI Interaction@) that it is important to get feedback _online_ from the human; giving feedback “all at once” at the beginning is too hard to do well. However, the idealized algorithm here does have the feedback “all at once”. It’s possible that this is okay, if it is primarily process-level feedback, but it seems fairly worrying to me.

**On desiderata:** The desiderata introduced in the first post feel stronger than they need to be. It seems possible to specify a method of interpreting feedback that is _good enough_: it doesn’t exactly capture everything, but it gets it sufficiently correct that it results in good outcomes. This seems especially true when talking about process-level feedback, or feedback one meta level up -- as long as the AI system has learned an okay notion of “being helpful” or “being corrigible”, then it seems like we’re probably fine.

Often, just making feedback uncertain can help. For example, in the preference learning literature, Boltzmann rationality has emerged as the model of choice for how to interpret human feedback. While there are several theoretical justifications for this model, I suspect it is successful simply because it makes feedback uncertain: if you want to have a model that assigns higher likelihood to high-reward actions, but still assigns some probability to all actions, it seems like you end up choosing the Boltzmann model (or something functionally equivalent). Note that there is work trying to improve upon this model, such as by [modeling humans as pedagogic](https://papers.nips.cc/paper/2016/file/b5488aeff42889188d03c9895255cecc-Paper.pdf), or by <@incorporating a notion of similarity@>(@LESS is More: Rethinking Probabilistic Models of Human Behavior@).

So overall, I don’t feel convinced that we need to aim for learning at all levels. That being said, the second post introduced a different argument: that the method does as well as we “could” do given the limits of human reasoning. I like this a lot more as a desideratum; it feels more achievable and more tied to what we care about.

Recursive Quantilizers II

This post gives an example scheme inspired by the previous post. Like [iterated amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd), it defines an ideal (analogous to <@HCH@>(@Humans Consulting HCH@)), and then an approximation to it that could be computed in practice.

Like HCH, we imagine a tree of systems that improve as we increase the depth of the tree. However, the nodes in the tree are question-answering (QA) _systems_, rather than direct questions. Given a few QA systems from a lower level, we construct a QA system at a higher level by asking one low-level QA system "what's a good safe distribution over QA systems", and a different low-level QA system "what's a good metric that we can use to judge QA systems".  We then use [quantilization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) ([AN #48](https://mailchi.mp/3091c6e9405c/alignment-newsletter-48)) to select better-performing QA systems, without optimizing too hard and falling prey to Goodhart's Law. In the infinite limit, this should converge to a stable equilibrium.

By having the tree reason about what good safe distributions are, and what good metrics are, we are explicitly improving the way that the AI system learns to interpret feedback (this is what the "good metric" is meant to evaluate), thus meeting the desiderata from the previous post.

To implement this in practice, we do something similar to iterated amplification. Iterated amplification approximates depth-limited HCH by maintaining a model that can answer _arbitrary_ questions (even though each node is a single question); similarly here we maintain a model that has a _distribution_ over QA systems (even though each node is a single QA system). Then, to sample from the amplified distribution, we sample two QA systems from the current distribution, ask one for a good safe distribution and the other for a good metric, and use quantilization to sample a new QA system given these ingredients. We use distillation to turn this slow quantilization process into a fast neural net model.

Considering the problem of <@inaccessible information@>(@Inaccessible information@), the hope is that, as we amplify the QA system, we will eventually be able to approve of some safe reasoning process about inaccessible information. If this doesn't happen, then it seems that no human reasoning could approve of reasoning about that inaccessible information, so we have done as well as possible.


Understanding Learned Reward Functions

Eric J. Michaud, Adam Gleave, Stuart Russell

This paper investigates what exactly learned reward functions have learned, through the use of interpretability techniques. The authors hope that it will be easier to identify misalignment by directly interpreting learned reward functions as opposed to policies trained on them, as it seems plausible that reward functions will stay relatively similar in complexity, even when the policies become more complex as AI systems become more capable. Specifically, the authors look at:

1. Saliency maps, which plot the gradient of the reward with respect to each pixel, intuitively quantifying "how important is this pixel to the reward"
2. Occlusion maps, which show how much the reward changes if a certain area of the image is blurred
3. Counterfactual inputs, in which the authors manually craft input images to see what the learned reward function outputs.

In a simple gridworld where the agent must find the goal, the authors coded the reward function "1 if the agent moves to a previously visible goal location, else 0", but they show that the learned reward is instead "0 if there is a currently visible goal location, else 1". These are identical in the training environment, where there is always exactly one goal location (that the agent may be standing on, in which case that location is not visible). However, if there are changes at test time, e.g. multiple goal locations, then the learned reward will diverge from the true reward.

They then apply a similar methodology to Atari. They find that if the score is not hidden, then the learned reward model will simply check whether the score pixels are changing to detect reward -- _unless_ the score pixels change at a later time than reward is accrued, in which case this is not a viable strategy. They thus suggest that future reward learning work on Atari should ensure that the score is removed from the screen. Another important direction would be to develop new interpretability techniques that are specific to reward functions, e.g. to handle the fact that reward functions are invariant to potential shaping.


The derivation of the method is interesting in its own right since it also offers an alternative derivation of GAIL. However, the main assumption of this paper is that the agent policy can be decomposed into an expert and non-expert mixture. This seems like a strong assumption and makes me skeptical of the theoretical analysis. Nevertheless, the experimental results do indicate that RIL-Co is able to outperform other naive approaches. I'm also surprised this paper has no reference to DART, which directly analyzes imitation learning in the noise-injection setting. Specifically, given that these experts are deterministic and we have a significant number of demonstrations, I'd expect that a reasonable amount of noise should be directly removable with regular behavioral cloning. Yet, there is no comparison with behavioral cloning.

<@GAIL@>(@Generative Adversarial Imitation Learning@)

Robust Imitation Learning from Noisy Demonstrations

Voot Tangkaratt, Nontawat Charoenphakdee, Masashi Sugiyama

One weakness of vanilla imitation learning is that it struggles to handle demonstrations from sub-optimal experts. Let's consider a simplified setting where the sub-optimal experts are modeled as optimal policies with injected gaussian noise. Ideally, the agent would learn to separate the noise from the true optimal policy.

This paper proposes an algorithm that is able to do this separation. The authors assume that the sub-optimal demonstrations and the learned agent policy can both be decomposed into a mixture of expert-policy and noise distributions. The main insight is that we can then learn a single classifier to distinguish noisy data from expert data; this classifier can then be used to define a reward function for an RL agent.

One issue is that since there is no ground truth for what is expert vs. noise, the classifier has to be trained on its own predictions, which can lead to overconfidence via positive feedback loops. To stabilize training, the authors train two models concurrently (co-training); each model is used to create training data for the other model. The authors call this approach RIL-Co. The experimental results show their algorithm RIL-Co is able to perform better than GAIL and other algorithms in the noisy regime.


The EMPATHIC Framework for Task Learning from Implicit Human Feedback

Yuchen Cui*, Qiping Zhang*, Alessandro Allievi, Peter Stone, Scott Niekum, W. Bradley Knox

A problem with learning from human feedback is that human feedback is quite expensive to collect. Can we instead learn from the facial expressions that humans automatically make anyway? This paper shows that the answer is yes: they first record human reactions while watching an autonomous agent, and use that to train a model that predicts reward given human reactions. They then transfer this model to a new task.


While I found the motivation and description of this paper somewhat unclear or misleading (Maia seems to me to be identical to behavior cloning, in which case it would not just be a “connection”), the experiments they run are pretty cool and it was interesting to see the pretty stark differences between models trained on a performance metric and models trained to imitate humans.

Aligning Superhuman AI and Human Behavior: Chess as a Model System

Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg, Ashton Anderson

Current AI systems are usually focused on some well-defined performance metric. However, as AI systems become more intelligent, we would presumably want to have humans learn from and collaborate with such systems. This is currently challenging since our superintelligent AI systems are quite hard to understand and don't act in human-like ways.

The authors aim to study this general issue within chess, where we have access both to superintelligent AI systems and lots of human-generated data. (Note: I'll talk about "ratings" below; these are not necessarily ELO ratings and should just be thought of as some "score" that functions similarly to ELO.) The authors are interested in whether AI systems play in a human-like way and can be used as a way of understanding human gameplay. One particularly notable aspect of human gameplay is that there is a wide range in skill: as a result we would like an AI system that can make predictions conditioned on varying skill levels.

For existing algorithms, the authors analyze the traditional Stockfish engine and the newer Leela (an open-source version of <@AlphaZero@>(@AlphaZero: Shedding new light on the grand games of chess, shogi and Go@)). They can get varying skill levels by changing the depth of the tree search (in Stockfish) or changing the amount of training (in Leela).

For Stockfish, they find that _regardless of search depth_, Stockfish action distributions monotonically increase in accuracy as the skill of the human goes up -- even when the depth of the search leads to a Stockfish agent with a similar skill rating as an amateur human. (In other words, if you take a low-ELO Stockfish agent and treat it as a predictive model of human players, it isn't a great predictive model ever, but it is best at predicting human experts, not human amateurs.) This demonstrates that Stockfish plays very differently than humans.

Leela on the other hand is somewhat more human-like: when its rating is under 2700, its accuracy is highest on amateur humans; at a rating of 2700 its accuracy is about constant across humans, and above 2700 its accuracy is highest on expert humans. However, its accuracy is still low, and the most competent Leela model is always the best predictor of human play (rather than the Leela model with the most similar skill level to the human whose actions are being predicted).

The authors then develop their own method, Maia. They talk about it as a "modification of the AlphaZero architecture", but as far as I can tell it is simply behavior cloning using the neural net architecture used by Leela. As you might expect, this does significantly better, and finally satisfies the property we would intuitively want: the best predictive model for a human of some skill level is the one that was trained on the data from humans at that skill level.

They also investigate a bunch of other scenarios, such as decisions in which there is a clear best action and decisions where humans tend to make mistakes, and find that the models behave as you'd expect (for example, when there's a clear best action, model accuracy increases across the board).


The experiments in this paper are a bit underwhelming. While they run a large number of experiments, ONAIL only occasionally outperforms value-dice consistently in the HalfCheetah environment. The authors justify this by noting that ONAIL wasn't regularized. Additionally, the policies are initialized with behavior cloning, something that value-dice doesn't require. However, the theoretical insight on iterative tightening is interesting, and together with the recent work on value-dice indicates that the design space of imitation learning algorithms is far from being exhausted.

Non-Adversarial Imitation Learning and its Connections to Adversarial Methods

Viewing imitation learning as a distribution matching problem has become more popular in recent years (see <@Value-Dice@>(@Imitation Learning via Off-Policy Distribution Matching@) / <@I2L@>(@State-only Imitation with Transition Dynamics Mismatch@)). However, the authors in this paper argue that such methods are unstable due to their formulation as saddle-point problems which means they have weak convergence guarantees due to the assumption that the policy is slowly updated. In this paper, the authors reformulate <@Adversarial IRL@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@) as a non-adversarial problem allowing for much stronger convergence guarantees to be proved. In particular, the authors derive a lower-bound on the discrimination reward which allows for larger policy updates and then introduce a method to iteratively tighten this bound. They also build on prior work for value-dice and derive a soft actor-critic algorithm (ONAIL) that they evaluate on a variety of control tasks.


It has <@previously been argued@>(@The easy goal inference problem is still hard@) that in order to do better than the demonstrator, you need to have a model of how the demonstrator makes mistakes. In this work, that model is something like, “while running A* search, the demonstrator may fail to find all the states, or may find a suboptimal path before an optimal one”. This obviously isn’t exactly correct, but is hopefully moving in the right direction.

Note that in the domains that the paper evaluates on, the number of possible goals is fairly small (at most 20), presumably because of computational cost. However, even if we ignore computational cost, it’s not clear to me whether this would scale to a larger number of goals. Conceptually, this algorithm is looking for the most likely item out of the set of (optimal demonstrations and plausible suboptimal or failed demonstrations). When the number of goals is low, this set is relatively small, and the true answer will likely be the clear winner. However, once the number of goals is much larger, there may be multiple plausible answers. (This is similar to the fact that since neural nets encode many possible algorithms and there are multiple settings that optimize your objective, usually instead of getting the desired algorithm you get one that fails to transfer out of distribution.)

Online Bayesian Goal Inference for Boundedly-Rational Planning Agents

Tan Zhi-Xuan, Jordyn L. Mann, Tom Silver, Joshua B. Tenenbaum, Vikash K. Mansinghka

Typical approaches to learning from demonstrations rely on assuming that the demonstrator is either optimal or noisily optimal. However, this is a pretty bad description of actual human reasoning: it is more accurate to say we are _boundedly-rational planners_. In particular, it makes more sense to assume that our plans are computed from a noisy process. How might we capture this in an algorithm?

This paper models the demonstrator as using a bounded probabilistic [A* search](https://en.wikipedia.org/wiki/A*_search_algorithm) to find plans for achieving their goal. The planner is also randomized to account for the difficulty of planning: in particular, when choosing which state to "think about" next, it chooses randomly with higher probability for more promising states (as opposed to vanilla A* which always chooses the most promising state).

The search may fail to find a plan that achieves the goal, in which case the demonstrator follows the actions of the most promising plan found by A* search until no longer possible (either an action leads to a state A* search hadn't considered, or it reaches the end of its partial plan). Thus, this algorithm can assign significant probability to plans that fail to reach the goal.

The experiments show that this feature allows their SIPS algorithm to infer goals even when the demonstrator fails to reach their goal. For example, if an agent needs to get two keys to unlock two doors to get a blue gem, but only manages to unlock the first door, the algorithm can still infer that the agent's goal was to obtain the blue gem.

I really like that this paper is engaging with the difficulty of dealing with systematically imperfect demonstrators, and it shows that it can do much better than Bayesian IRL for the domains they consider.


This seems right to me -- we can also see this by looking at the various paradoxes found in the philosophy of ethics, which involve taking everyday moral intuitions and finding extreme situations in which they conflict, and it is unclear which moral intuition should “win”.

"Go west, young man!" - Preferences in (imperfect) maps

This post argues that by default, human preferences are strong views built upon poorly defined concepts, that may not have any coherent extrapolation in new situations. To put it another way, humans build mental maps of the world, and their preferences are defined on those maps, and so in new situations where the map no longer reflects the world accurately, it is unclear how preferences should be extended. As a result, anyone interested in preference learning should find some incoherent moral intuition that other people hold, and figure out how to make it coherent, as practice for the case we will face where our own values will be incoherent in the face of new situations.


I'm excited by this improvement, since now we can actually solve non-trivial CIRL games -- one of the games they solve has around 10 billion states. With this we can run experiments with real humans, which seems really important, and the paper does mention a very preliminary pilot study run with real humans.

An Efficient, Generalized Bellman Update For Cooperative Inverse Reinforcement Learning

Dhruv Malik, Malayandi Palaniappan, Jaime F. Fisac, Dylan Hadfield-Menell, Stuart Russell, Anca D. Dragan

Previously, Cooperative Inverse Reinforcement Learning (CIRL) games were solved by reducing them to a POMDP with an exponentially-sized action space, and then solving with POMDP algorithms that are exponential in the size of the action space, leading to a doubly-exponential algorithm. This paper leverages the fact that the human has perfect information to create a modified Bellman update that still computes the optimal policy, but no longer requires an exponential action space. The modified Bellman update works with the human's policy, and so we can now swap in more accurate models of the human, including eg. noisy rationality (whereas previously the human had to be exactly optimal). They show huge speedups in experiments, and discuss some interesting qualitative behavior that arises out of CIRL games -- for example, sometimes the human _waits_ instead of making progress on the task, because it is a good signal to the robot of what the human wants.


The intuition behind this method makes a lot of sense to me, but I wish the experiments were clearer in showing how the method compares to other methods. They show that, on Cartpole and Acrobat, they can match the results of behavioral cloning with 50,000 state-action pairs using 50,000 state observations and 200 environment interactions, but I don't know if behavioral cloning actually needed that many state-action pairs. Similarly, I'm not sure how much environment interaction would be needed if you inferred a reward function but not the dynamics, since they don't compare against such a method. I'm also unclear on how hard it is to assign transitions to latent actions -- they only test on MDPs with at most 3 actions, it's plausible to me that with more actions it becomes much harder to figure out which hidden action a state transition should correspond to.

Imitating Latent Policies from Observation

Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker, Charles L. Isbell

Typically in imitation learning, we assume that we have access to demonstrations that include the actions that the expert took. However, in many realistic settings we only have access to state observations (eg. driving videos). In this setting, we could still infer a reward function and then use reinforcement learning (RL) to imitate the behavior, but this would require a lot of interaction with the environment to learn the dynamics of the environment. Intuitively, even demonstrations with only states and no actions should give us a lot of information about the dynamics -- if we can extract this information, then we would need much less environment interaction during RL. (For example, if you watch a friend play a video game, you only see states, not actions; yet you can infer a lot about the game rules and gameplay.) The key idea is that each action probably causes similar effects on different states. So, they create a model with hidden action nodes z, and use the state observations to learn a policy P(z | s) and dynamics s' = g(s, z) (they assume deterministic dynamics). This is done end-to-end with neural nets, but essentially the net is looking at the sequence of states and figuring out how to assign actions z to each s (this is P(z | s)), such that we can learn a function g(s, z) that outputs the next observed state s'. Once this is trained, intuitively g(s, z) will already have captured most of the dynamics, and so now we only require a small number of environment actions to figure out how the true actions a correspond to the hidden actions z -- concretely, we train a model P(a | s, z). Then, in any state s, we first choose the most likely hidden action z* according to P(z | s), and then the most likely action a* according to P(a | s, z*).


In one of the experiments, the baseline of running IRL from scratch performed second best, beating out two other methods of meta-learning. I'd guess that this is because both MandRIL and standard IRL benefit from assuming the maxent IRL distribution over trajectories (which I believe is how the demonstrations were synthetically generated), whereas the other two meta learning baselines do not have any such assumption, and must learn this relationship.

Learning a Prior over Intent via Meta-Inverse Reinforcement Learning

Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, Chelsea Finn

For complex rewards, such as reward functions defined on pixels, standard IRL methods require a large number of demonstrations. However, many tasks are very related, and so we should be able to leverage demonstrations from one task to learn rewards for other tasks. This naturally suggests that we use meta learning. The authors adapt [MAML](https://arxiv.org/abs/1703.03400) to work with maximum entropy IRL (which requires differentiating through the MaxEnt IRL gradient). They evaluate their approach, called MandRIL, on a navigation task whose underlying structure is a gridworld, but the state is represented as an image so that the reward function is nonlinear and requires a convnet.


I think it's easy to forget that observations aren't necessarily equivalent to state representations. This paper did a good job of reminding me that using state features on the MujoCo tasks is different from using images to train imitation learning agents. In practice, trying to learn just from images can fail because of partial observability, but introducing proprioception is a natural solution here. I broadly agree with the authors' conclusion that resolving embodiment mismatch and viewpoint mismatch are natural next steps for this kind of research.

Imitation Learning from Video by Leveraging Proprioception

Faraz Torabi, Garrett Warnell, Peter Stone

Recent work into imitation learning from observation (IfO) allows agents to perform a task from visual demonstrations that do not include state and action information. In this paper the authors are interested in leveraging proprioception information, knowledge of internal states, to create an efficient IfO algorithm. As opposed to GAIfO, which typically uses only the observation vector, this algorithm only allows images to be used for discrimination but lets the agent make use of internal states to generate actions. They test their proposed technique on several MujoCo domains and show that it outperforms other imitation from observation algorithms. The authors note that in practice occlusion and fast movement in environments like Walker2d and HalfCheetah make it difficult to learn directly from images which partly explains the success of using proprioceptive features.


The main value of this paper is that the difference between LfO and LfD is clarified by introducing the notion of inverse disagreement. Related to this analysis, the authors note that GAIfO has the same objective as the inverse disagreement model if we replace KL with JS divergence. This makes me suspect that there's a general LfO [divergence minimization perspective](https://arxiv.org/abs/1911.02256) relating all of these methods together. In other words, the fact that the objectives for LfO and LfD can be related via KL/JS divergence indicates that there is an entire class of methods underlying this approach to LfO. Specifically, I'd hypothesize that regularized inverse reinforcement learning from observation followed by reinforcement learning would be equivalent to a divergence minimization problem.

[GAIfO](https://arxiv.org/abs/1807.06158) and [Recent Advances in LfO](https://arxiv.org/pdf/1905.13566.pdf)

Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement

Chao Yang*, Xiaojian Ma*, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, Chuang Gan

Learning from observation (LfO) focuses on imitation learning in situations where we want to learn from state-only demonstrations. This contrasts with learning from demonstration (LfD) which needs both state and action information. In practice, LfO is the more common situation due to the prevalence of unannotated data, such as video. In this paper, the authors show that the gap between LfO and LfD comes from the disagreement of inverse dynamics models between the imitator and the expert. If the inverse dynamics model is perfect, then state transitions can be labeled with actions and LfD can be performed on the result. However, it's often the case that many actions can generate the same state transition. They then show that optimizing an upper-bound on this gap leads to improved performance as compared to other LfO methods such as GAIfO (GAIL extended to LfO).


The hypothesis for the failure of meta-AIRL is that if the base thing that you are trying to use in your meta-learning fails with some probability, then meta-learning will fail with high probability, since even a small number of failures lead to very bad inputs into meta-learning, which cause it to perform badly. I'd be interested in seeing experiments evaluating this hypothesis in other domains. This would also explain why meta-learning has had its greatest successes in supervised learning settings, which are typically much more robust than RL settings.

Learning a Prior over Intent via Meta-Inverse Reinforcement Learning

Multi-task Maximum Entropy Inverse Reinforcement Learning

This paper tackles multi-task inverse reinforcement learning, where there are multiple related tasks which have similar rewards, and the goal is to infer the reward for a new task after having already seen demonstrations for previous tasks. In the tabular setting, where we can enumerate all of the states and actions, they adapt Maximum Causal Entropy IRL to the multi-task setting. To make any progress, it is necessary to have some model of how the tasks are related -- in this work, they assume that the reward weights are close to each other, and so they penalize the L2 distance from the reward weights to the mean weights across tasks. Experiments show that this approach can learn a new task reward in 1-2 trajectories, while learning from scratch takes 50+ trajectories. They then consider how to generalize to continuous control environments such as MountainCar. They propose an algorithm that applies <@Reptile@>(@Reptile: A Scalable Meta-Learning Algorithm@) to <@adversarial IRL@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@), with the goal of learning a good initialization of the reward neural net which can quickly move to the correct reward function given data from a new task. This works well when the policy is unimodal (eg. for MountainCar, a policy that always goes left or always goes right), but not when the policy is multimodal (eg. you have to go either left or right depending on the color of the flag). Experiments suggest that this is because adversarial IRL does not do well with a multimodal policy. Meta-AIRL would require AIRL to produce good results on multiple environments -- if even one of the environments has a bad policy, there's a garbage input to meta-AIRL, which then tanks its performance.


On the Foundations of Expected Expected Utility

Suppose you want to understand someone's preferences over a set of outcomes, and lotteries over those outcomes. The VNM theorem shows that under certain reasonable assumptions about those preferences, we can represent the preferences using a utility function over outcomes (_not_ lotteries), such that the person prefers lottery A to lottery B if and only if the expected utility of lottery A is greater than the expected utility of lottery B. Furthermore, the utility function is unique up to an affine transformation.

What if we want to have uncertainty over the utility function? The natural approach would be to maintain a probability distribution over utility functions, and then say that the person prefers lottery A to lottery B if and only if the expected expected utility of lottery A exceeds the expected expected utility of lottery B. Here, the first expectation is taken over the probability distribution over utility functions, and the second expectation is taken over the probabilities in the lottery. This decision rule is called Maximum Expected Expected Utility (MEEU).

However, the MEEU decision rule gives different answers if you transform one of the utility functions in an affine way. This isn't great, since affine transformations of utility functions encode the same preferences, and so you'd like your decision rule to be invariant to them. This is the standard problem that utility functions are not by default _commensurable_.

This paper shows that everything is fine if you assume that all the utility functions agree on the best and worst outcomes. In this case, any utility function can be uniquely characterized by stating for each state s, for what p are you indifferent between s and {p: s_best, (1-p): s_worst}. We can then define the base decision rule as follows: when presented with a lottery A = {p_i : s_i, ...}, you convert it into a lottery over utility functions: {q_U : A_U, ...}, where for each utility function, you define A_U as A, but replacing all the s_i with the corresponding {p: s_best, (1-p): s_worst} lotteries as defined by U. So now, we have a compound lottery of the form {q_U : {p_i : {p: s_best, (1-p): s_worst}, ...}, ...}, which can be reduced to a simple lottery of the form {x: s_best, (1-x): s_worst}. When comparing two such lotteries, you prefer lottery A to lottery B if and only if x_A > x_B.

So that's what you do without using utility functions, and only using preference orderings. They show that the MEEU decision rule is identical to this base decision rule, justifying the use of MEEU, as long as you normalize all of your utility functions so that they all assign the same value to the best and worst outcomes.


Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft

Christian Scheller, Yanick Schraner, Manfred Vogel

This paper explains the technique used by the 3rd place team in the MineRL competition (summarized above). They used behavior cloning to train their neural net on human demonstrations, and then used reinforcement learning (specifically, IMPALA) with experience replay and advantage clipping to improve. There are more details about their architecture and design choices in the paper.


I agree with the general spirit of "get the AI system to understand common sense; then give it instructions that it interprets correctly". I usually expect future ML research to figure out the common sense part, so I don't look for particular implementations (in this case, simultaneous training on vision and natural language), but just assume we'll have that capability somehow. The hard part is then how to leverage that capability to provide _correctly interpreted_ instructions. It may be as simple as providing instructions in natural language, as this post suggests. I'm much less worried about instrumental subgoals in such a scenario, since part of "understanding what we mean" includes "and don't pursue this instruction literally to extremes". But we still need to figure out how to translate natural language instructions into actions.

What's the dream for giving natural language commands to AI?

We could try creating AI systems that take the "artificial intentional stance" towards humans: that is, they model humans as agents that are trying to achieve some goals, and then we get the AI system to optimize for those inferred goals. We could do this by training an agent that jointly models the world and understands natural language, in order to ground the language into actual states of the world. The hope is that with this scheme, as the agent gets more capable, its understanding of what we want improves as well, so that it is robust to scaling up. However, the scheme has no protection against Goodharting, and doesn't obviously care about metaethics.


This is a nice illustration of the very abstract algorithm proposed before; I'd love it if more people illustrated their algorithms this way.

Full toy model for preference learning

This post applies Stuart's general preference learning algorithm to a toy environment in which a robot has a mishmash of preferences about how to classify and bin two types of objects.


I like this paper, largely because learning from advice hasn't been applied much to 3D worlds, and this is a compelling proof of concept. I think it's also a noteworthy though expected result that advice that sticks temporally helps a lot when the ground truth visual evidence is difficult to interpret.

Improving Deep Reinforcement Learning in Minecraft with Action Advice

This paper uses maze-traversal in Minecraft to look at the extent to which human advice can help with _aliasing_ in 3D environments, the problem where many states share nearly identical visual features. The paper compares two advice-giving algorithms that rely on neural nets which are trained to explore and predict the utilities of possible actions they can take, sometimes accepting human advice. The two algorithms differ primarily in whether they provide advice for the current action, or provide advice that persists for several actions.

Experimental results suggest that both algorithms, but especially the one that applies to multiple actions, help with the problem of 3D aliasing, potentially because the system can rely on the movement advice it got in previous timesteps rather than having to discern tricky visual features in the moment. The paper also varies the frequency and accuracy of the advice given, and finds that receiving more advice significantly improves performance, even if that advice is only 50% accurate.


So far, this is an idea that hasn't been turned into a proposal yet, so it's hard to evaluate. The most obvious implementation (to me) would involve an explicit estimate of reward uncertainty, and then an explicit model for how to update that uncertainty (that would not be Bayes Rule, since that would narrow the uncertainty over time). At this point it's not clear to me why we're even using the expected utility formalism; it feels like adding epicycles in order to get a single particular behavior that breaks other things. You could also make the argument that there will be misspecification of the model of how update the uncertainty. But again, this is just the most obvious completion of the idea; it's plausible that there's a different way of doing this that's better.

Defeating Goodhart and the closest unblocked strategy problem

One issue with the idea of <@reward uncertainty@> based on a model of uncertainty that we specify is that we tend to severely underestimate how uncertain we should be. This post makes the point that we could try to build an AI system that starts with this estimate of our uncertainty, but then corrects the estimate based on its understanding of humans. For example, if it notices that humans tend to become much more uncertain when presented with some crucial consideration, it could realize that its estimate probably needs to be widened significantly.


I think the success at generalization comes primarily because of the MaxEnt IRL during training: it provides a lot of structure and inductive bias that means that the rewards on which the reward predictor is trained are "close" to the intended reward function. For example, in the navigation tasks, the demonstrations for a command like "go to the vase" will involve trajectories through the state of many houses that end up in the vase. For each demonstration, MaxEnt IRL "assigns" positive reward to the states in the demonstration, and negative reward to everything else. However, once you average across demonstrations in different houses, the state with the vase gets a huge amount of positive reward (since it is in all trajectories) while all the other states are relatively neutral (since they will only be in a few trajectories, where the agent needed to pass that point in order to get to the vase). So when this is "transferred" to the neural net via gradients, the neural net is basically "told" that high reward only happens in states that contain vases, which is a strong constraint on the learned reward. On the other hand, for "move X to Y" tasks, while the same argument suggests that you will have high reward on reaching X, the rest of the trajectories from X to Y will all be the same and MaxEnt IRL will learn a shaped reward that doesn't transfer well. So, I predict that on "move X to Y" tasks at test time, the agent will successfully pick up the object X, but fail to then move it to Y.

To be clear, this is not meant as a critique of the paper: indeed, I think when you want out-of-distribution generalization, you _have_ to do it by imposing structure/inductive bias, and this is a new way to do it that I hadn't seen before.

From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following

Justin Fu, Anoop Korattikara, Sergey Levine, Sergio Guadarrama

Rewards and language commands are more generalizable than policies: "pick up the vase" would make sense in any house, but the actions that navigate to and pick up a vase in one house would not work in another house. Based on this observation, this paper proposes that we have a dataset where for several (language command, environment) pairs, we are given expert demonstrations of how to follow the command in that environment. For each data point, we can use IRL to infer a reward function, and use that to train a neural net that can map from the language command to the reward function. Then, at test time, given a language command, we can convert it to a reward function, after which we can use standard deep RL techniques to get a policy that executes the command.

The authors evaluate on a 3D house domain with pixel observations, and two types of language commands: navigation and pick-and-place. During training, when IRL needs to be done, since deep IRL algorithms are computationally expensive they convert the task into a small, tabular MDP with known dynamics for which they can solve the IRL problem exactly, deriving a gradient that can then be applied in the observation space to train a neural net that given image observations and a language command predicts the reward. Note that this only needs to be done at training time: at test time, the reward function can be used in a new environment with unknown dynamics and image observations. They show that the learned rewards generalize to novel combinations of objects within a house, as well as to entirely new houses (though to a lesser extent).


Using Natural Language for Reward Shaping in Reinforcement Learning

Prasoon Goyal, Scott Niekum, Raymond J. Mooney

This paper constructs a dataset for grounding natural language in Atari games, and uses it to improve performance on Atari. They have humans annotate short clips with natural language: for example, "jump over the skull while going to the left" in Montezuma's Revenge. They use this to build a model that predicts whether a given trajectory matches a natural language instruction. Then, while training an agent to play Atari, they have humans give the AI system an instruction in natural language. They use their natural language model to predict the probability that the trajectory matches the instruction, and add that as an extra shaping term in the reward. This leads to faster learning.


I don't fully understand this paper -- how exactly are they searching over states, when there are exponentially many of them? Are they sampling them somehow? It's definitely possible that this is in the paper and I missed it, I did skim it fairly quickly.

Risk-Aware Active Inverse Reinforcement Learning

Daniel S. Brown, Yuchen Cui, Scott Niekum

This paper presents an algorithm that actively solicits demonstrations on states where it could potentially behave badly due to its uncertainty about the reward function. They use Bayesian IRL as their IRL algorithm, so that they get a distribution over reward functions. They use the most likely reward to train a policy, and then find a state from which that policy has high risk (because of the uncertainty over reward functions). They show in experiments that this performs better than other active IRL algorithms.


I'm excited to see work on dealing with conflicting preference information, particularly from multiple data sources. To my knowledge, there isn't any work on this -- while there is work on multimodal input, usually those inputs don't conflict, whereas this post explicitly has several examples of conflicting preferences, which seems like an important problem to solve. However, I would aim for a solution that is less fixed (i.e. not one specific aggregation rule), for example by an active approach that presents the conflict to the human and asks how it should be resolved, and learning an aggregation rule based on that. I'd be surprised if we ended up using a particular mathematical equation presented here as an aggregation mechanism -- I'm much more interested in what problems arise when we try to aggregate things, what criteria we might want to satisfy, etc.

Acknowledging Human Preference Types to Support Value Learning

Humans often have multiple "types" of preferences, which any value learning algorithm will need to handle. This post concentrates on one particular framework -- liking, wanting and approving. Liking corresponds to the experience of pleasure, wanting corresponds to the motivation that causes you to take action, and approving corresponds to your conscious evaluation of how good the particular action is. These correspond to different data sources, such as facial expressions, demonstrations, and rankings respectively. Now suppose we extract three different reward functions and need to use them to choose actions -- how should we aggregate the reward functions? They choose some desiderata on the aggregation mechanism, inspired by social choice theory, and develop a few aggregation rules that meet some of the desiderata.


I'm confused by this paper, because they only compute the empowerment for a _single action_. I would expect that in most states, different actions lead to different next states, which suggests that the empowerment will be the same for all states. Why then does it have any effect? And even if the empowerment was computed over longer action sequences, what is the reason that this leads to learning generalizable rewards? My normal model is that IRL algorithms don't learn generalizable rewards because they mostly use the reward to "memorize" the correct actions to take in any given state, rather than learning the underlying true reward. I don't see why empowerment would prevent this from happening. Yet, their experiments show quite large improvements, and don't seem particularly suited to empowerment.

Adversarial Imitation via Variational Inverse Reinforcement Learning

Ahmed H. Qureshi, Byron Boots, Michael C. Yip

A short history of deep IRL algorithms: [GAIL](https://arxiv.org/abs/1606.03476) introduced the idea of training a policy that fools a discriminator that tries to distinguish a policy from expert demonstrations, [GAN-GCL](https://arxiv.org/abs/1611.03852) showed how to recover a reward function from the discriminator, and [AIRL](https://arxiv.org/abs/1710.11248) ([AN #17](https://mailchi.mp/ad852629e45a/alignment-newsletter-17)) trains on (s, a, s') tuples instead of trajectories to reduce variance, and learns a reward shaping term separately so that it transfers better to new environments. This paper proposed that the reward shaping term be the _empowerment_ of a state. The empowerment of a state is the maximum mutual information between a sequence of actions from a state, and the achieved next state. Intuitively, this would lead to choosing to go to states from which you can reach the most possible future states. Their evaluation shows that they do about as well as AIRL in learning to imitate an expert, but perform much better in transfer tasks (where the learned reward function must generalize to a new environment).