I certainly agree that many problems with figuring out what to optimize stem from embedded agency issues with humans, and any <@formal account@>(@Why we need a *theory* of human values@) of this will benefit from general progress in understanding embeddedness. Unlike many others, I do not think we need a formal account of human values, and that a "common-sense" understanding will suffice, including for the embeddedness problems detailed in this post. (See also this [comment thread](https://www.alignmentforum.org/posts/WJzsTmsDctYCCyMfy/humans-are-embedded-agents-too#opd3EeESfWiEyekDh) and the next summary.)

Humans Are Embedded Agents Too

<@Embedded agency@>(@Embedded Agents@) is not just a problem for AI systems: humans are embedded agents too; many problems in understanding human values stem from this fact. For example, humans don't have a well-defined output channel: we can't say "anything that comes from this keyboard is direct output from the human", because the AI could seize control of the keyboard and wirehead, or a cat could walk over the keyboard, etc. Similarly, humans can "self-modify", e.g. by drinking, which often modifies their "values": what does that imply for value learning? Based on these and other examples, the post concludes that "a better understanding of embedded agents in general will lead to substantial insights about the nature of human values".


Stuart Armstrong Research Agenda Online Talk

AI Alignment Podcast: Synthesizing a human’s preferences into a utility function

Stuart Armstrong's <@agenda@>(@Research Agenda v0.9: Synthesising a human's preferences into a utility function@) involves extracting partial preferences from a human and synthesizing them together into an _adequate_ utility function. Among other things, this podcast goes into the design decisions underlying the agenda:

First, why even have a utility function? In practice, there are [many pressures](https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities) suggesting that maximizing expected utility is the "right" thing to do -- if you aren't doing this, you're leaving value on the table. So any agent that isn't maximizing a utility function will want to self-modify into one that is using a utility function, so we should just use a utility function in the first place.

Second, why not defer to a long reflection process, as in [Indirect Normativity](https://ordinaryideas.wordpress.com/2012/04/21/indirect-normativity-write-up/), or some sort of reflectively stable values? Stuart worries that such a process would lead to us prioritizing simplicity and elegance, but losing out on something of real value. This is also why he focuses on _partial preferences_: that is, our preferences in "normal" situations, without requiring such preferences to be extrapolated to very novel situations. Of course, in any situation where our moral concepts break down, we will have to extrapolate somehow (otherwise it wouldn't be a utility function) -- this presents the biggest challenge to the research agenda.


This paper seems like a natural extension of prior work. The imitation learning problem from observation is well-known and difficult. Introducing human feedback with a structured state space definitely seems like a viable way to get around a lot of the known difficulties with other methods such as a GAIL.

Learning from Observations Using a Single Video Demonstration and Human Feedback

Sunil Gandhi, Tim Oates, Tinoosh Mohsenin, Nicholas Waytowich

Designing rewards can be a long and consuming process, even for experts. One common method to circumvent this problem is through demonstration. However, it might be difficult to record demonstrations in a standard representation, such as joint positions. **In this paper, the authors propose using human feedback to circumvent the discrepancy between how demonstrations are recorded (video) and the desired standard representation (joint positions).** First, humans provide similarity evaluations of short clips of an expert demonstration to the agent's attempt and a similarity function is learned by the agent. Second, this similarity function is used to help train a policy that can imitate the expert. Both functions are learned jointly. The algorithm can learn to make a Hopper agent back-flip both from a Hopper demonstration of a back-flip, and from a YouTube video of a human backflipping. Ultimately, the authors show that their method improves over another method that uses human feedback without direct comparison to desired behavior.


I certainly agree that if the agent is wrong in its beliefs about the reward, then it is quite likely that it would not obey shutdown commands. For example, in the off switch game, if the agent is incorrectly certain that u is positive, then it will take action a, even though the human would want to shut it down. See also <@these@>(@Latent Variables and Model Mis-Specification@) <@posts@>(@Model Mis-specification and Inverse Reinforcement Learning@) on model misspecification and IRL. For a discussion of how serious the overall critique is, both from HC's perspective and mine, see the opinion on the next post.

Incorrigibility in the CIRL Framework

This paper demonstrates that when the agent has an _incorrect_ belief about the human's reward function, then you no longer get the benefit that the agent will obey shutdown instructions. It argues that since the purpose of a shutdown button is to function as a safety measure of last resort (when all other measures have failed), it should not rely on an assumption that the agent's belief about the reward is correct.


As with the previous paper, this argument is only really a problem when the agent's belief about the reward function is _wrong_: if it is correct, then at the point where there is no more information to gain, the agent should already know that humans don't like to be killed, do like to be happy, etc. and optimizing the expectation of the reward distribution should lead to good outcomes. Both this and the previous critique are worrisome when you can't even put a reasonable _prior_ over the reward function, which is quite a strong claim.

HC's response is that the agent should never assign zero probability to any hypothesis. It suggests that you could have an expandable hierarchical prior, where initially there are relatively simple hypotheses, but as hypotheses become worse at explaining the data, you "expand" the set of hypotheses, ultimately bottoming out at (perhaps) the universal prior. I think that such an approach could work in principle, and there are two challenges in practice. First, it may not be computationally feasible to do this. Second, it's not clear how such an approach can deal with the fact that human preferences _change_ over time. (HC does want more research into both of these.)

Fully updated deference could also be a problem if the observation model used by the agent is incorrect, rather than the prior. I'm not sure if this is part of the argument.

Problem of fully updated deference

This article points out that even if you have an agent with uncertainty over the reward function, it will acquire information and reduce its uncertainty over the reward, until eventually it can't reduce uncertainty any more, and then it would simply optimize the expectation of the resulting distribution, which is equivalent to optimizing a known objective, and has the same issues (such as disabling shutdown buttons).


I still have a personal sense that Behavioral Cloning is too brittle of a conceptual frame to build really robust imitative agents with, but this seems like a clever and relatively clean way to build in a bias towards high-confidence states. I find myself wondering if the same general idea of penalizing being in high-model-error states could be more broadly applied as a sort of regularizer in other off-policy settings where exploration can be risky.

SQIL: Imitation Learning via Regularized Behavioral Cloning

Siddharth Reddy, Anca D. Dragan, Sergey Levine

Behavioral Cloning is one of the most direct forms of imitation learning: it learns to predict the action the expert would have taken in a given state of the world. A clear weakness of the approach is that, if cloning models are only trained on pairs of (state, expert action) drawn from the expert's policy distribution, that means the model is underconstrained and thus likely to have high error on states that would have been unseen or just highly unlikely to be visited by the expert. This weakness means that errors within behavioral cloning systems can compound: if the system takes an incorrect action that leads it to a state it never saw the expert in, it will have a difficult time knowing what to do there.

The main contribution of this paper is to suggest a fix for this weakness, by learning a Q function to represent expert behavior, and by penalizing the model for being in states where its temporal difference error on the Q function (otherwise known as the Bellman error) is high. Intuitively, the hope is that this term, which can also be seen as a reward for being in states the expert has seen more frequently (equivalently, states where the model had more training experience) will propagate outward, and give the model a loss surface that pulls it back into states where its predictions are more confident.


Research Agenda v0.9: Synthesising a human's preferences into a utility function

One approach to AI alignment involves learning a specification of human values that can then be optimized. This agenda proposes that we learn an adequate representation of values (i.e. _not_ <@ambitious value learning@>(@What is ambitious value learning?@)). We first obtain partial preferences and associated weights from human mental models whose symbols have been adequately grounded. Calling these "preferences" is a normative assumption to avoid an <@impossibility result in value learning@>(@Humans can be assigned any values whatsoever…@): the hope is that the AI could correct for incorrect human beliefs. The preferences are then extended to all possible states, and are normalized so that they are comparable to each other, and then synthesized into a utility function that the AI can optimize.

The partial preferences are divided into a few categories: individual preferences, preferences about the rest of the world, and meta-preferences, some of which can be about the synthesis procedure itself. The hope is that further categories of preferences would be handled by the synthesis procedure; these categories are the ones that seem most important to get right, or couldn't be obtained any other way.


I agree with both of these critiques, especially the one about the bitter lesson. It seems like Stuart's approach imposes a particular structure or algorithm for how to synthesize the utility function; I am generally skeptical of such approaches. Also, as you might already know, I think it is neither necessary nor sufficient for AI alignment to find a utility function or "goal" that the AI can safely optimize. Since this promises to be a very difficult enterprise (Section 0.2 notes that it aims to "solve at least 5 major open problems in philosophy, to a level rigorous enough that we can specify them in code"), I prefer to look into other approaches that seem more tractable.

I do think that the problems that motivate the various aspects of the agenda are important and useful to think about, and I am happy that they have all been put into this single post. I also like the fact that the research agenda is directly aiming for a full solution to AI alignment.

Some Comments on Stuart Armstrong's "Research Agenda v0.9"

This post makes two main critiques of the research agenda in the previous entry. First, the research agenda involves a lot of human-designed features and modules, but <@The Bitter Lesson@> is that machine learning tends to shine with highly abstract large models that can make use of a lot of compute. Second, the symbol grounding part of the agenda requires the AI system to develop representations of the world that match the representations that humans use, and we have no idea how to do that, or even what it would mean to "match human representations" when the AI is more intelligent than humans. The post also includes some more specific comments that I'm not summarizing.


It's interesting to see what kinds of assumptions are necessary in order to get AI systems that can avoid catastrophically bad outcomes, and the notion of "traps" seems like a good way to formalize this. I worry about there being a Cartesian boundary between the agent and the environment, though perhaps even here as long as the advisor is aware of problems caused by such a boundary, they can be modeled as traps and thus avoided.

Of course, if we want the advisor to be a human, both of the assumptions are unrealistic, but I believe Vanessa's plan is to make the assumptions more realistic in order to see what assumptions are actually necessary.

One thing I wonder about is whether the focus on traps is necessary. With the presence of traps in the theoretical model, one of the main challenges is in preventing the agent from falling into a trap due to ignorance. However, it seems extremely unlikely that an AI system manages to take some irreversible catastrophic action _by accident_ -- I'm much more worried about the case where the AI system is adversarially optimizing against us and intentionally takes an irreversible catastrophic action.

Delegative Reinforcement Learning

Consider environments that have “traps”: states that permanently curtail the long-term value that an agent can achieve. A world without humans could be one such trap. Traps could also happen after any irreversible action, if the new state is not as useful for achieving high rewards as the old state.

In such an environment, an RL algorithm could simply take no actions, in which case it incurs regret that is linear in the number of timesteps so far. (Regret is the difference between the expected reward under the optimal policy and the policy actually executed, so if the average reward per timestep of the optimal policy is 2 and doing nothing is always reward 0, then the regret will be ~2T where T is the number of timesteps, so regret is linear in the number of timesteps.) Can we find an RL algorithm that will _guarantee_ regret sublinear in the number of timesteps, regardless of the environment?

Unsurprisingly, this is impossible, since during exploration the RL agent could fall into a trap, which leads to linear regret. However, let's suppose that we could delegate to an advisor who knows the environment: what must be true about the advisor for us to do better? Clearly, the advisor must be able to always avoid traps (otherwise the same problem occurs). However, this is not enough: getting sublinear regret also requires us to explore enough to eventually find the optimal policy. So, the advisor must have at least some small probability of being optimal, which the agent can then learn from. This paper proves that with these assumptions there does exist an algorithm that is guaranteed to get sublinear regret.


This is an elegant paper that makes good use of the toolkit of active learning for human preference solicitation, but it's batch heuristics are all very reliant on having a set of high level trajectory features in which Euclidean distance between points is a meaningful similarity metric, which feels like a not impossible to generalize but still somewhat limiting constraint.

<@Active Preference-Based Learning of Reward Functions@>

Batch Active Preference-Based Learning of Reward Functions

This paper builds on a trend of recent papers that try to learn human preferences, not through demonstrations of optimal behavior, but through a human expressing a preference over two possible trajectories, which has both pragmatic advantages (re limits of human optimality) and theoretic ones (better ability to extrapolate a reward function). Here, the task is framed as: we want to send humans batches of paired trajectories to rank, but which ones? Batch learning is preferable to single-sample active learning because it's more efficient to update a network after a batch of human judgments, rather than after each single one. This adds complexity to the problem because you'd prefer to not have a batch of samples that are individually high-expected-information, but which are redundant with one another. The authors define an information criterion (basically the examples about which we're most uncertain of the human's judgment) and then pick a batch of examples based on different heuristics for getting a set of trajectories with high information content that are separated from each other in feature space.


This is a result I find impressive, primarily because of its interest in abiding by sensible real-world constraints: it's easier for humans to label successful end states than to demonstrate a series of actions, and the number of queries made was similarly pragmatically low.

End-to-End Robotic Reinforcement Learning without Reward Engineering

Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, Sergey Levine

This paper demonstrates an approach that can learn to perform real world robotics tasks based not on example trajectories (states and actions) but just a small number (10) of pixel-level images of goal states showing successful task completion. Their method learns a GAN-like classifier to predict whether a given image is a success, continually adding data sampled from the still-learning policy to the set of negative examples, so the model at each step needs to further refine its model of success. The classifier, which is used as the reward signal in learning the policy, also makes use of a simple active learning approach, choosing the state its classifier is most confident is success and querying a human about it on fixed intervals, ultimately using less than 75 queries in all cases.


The general point here, which comes up pretty often, is that any information you get about "what humans want" is going to be specific to the context in which you elicit that information. This post makes that point when the information you get is the actions that people take. Some other instances of this point:

    - [Inverse Reward Design](https://arxiv.org/abs/1711.02827) notes that a human-provided reward function should be treated as _specific to the training environment_, instead of as a description of good behavior in all possible environments.
    - [CP-Nets](https://www.cs.toronto.edu/~cebly/Papers/CPnets.pdf) are based on the point that when a human says "I want X" it is not a statement that is meant to hold in all possible contexts. They propose very weak semantics, where "I want X" means "holding every other aspect of the world constant, it would be better for X to be present than for it not to be present".
    - [Wei Dai's point](https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety) ([AN #37](https://mailchi.mp/6eb8190e723f/alignment-newsletter-37)) that humans likely have adversarial examples, and we should not expect preferences to generalize under distribution shift.
    - Stuart Armstrong and Paul Christiano have made or addressed this point in many of their posts.

Conditional revealed preference

When backing out preferences by looking at people's actions, you may find that even though they say they are optimizing for X, their actions are better explained as optimizing for Y. This is better than relying on what they say, at least if you want to predict what they will do in the future. However, all such inferences are specific to the current context. For example, you may infer that schools are "about" dealing with authoritarian work environments, as opposed to learning -- but maybe this is because everyone who designs schools doesn't realize what the most effective methods of teaching-for-learning are, and if they were convinced that some other method was better for learning they would switch to that. So, in order to figure out what people "really want", we need to see not only what they do in the current context, but also what they would do in a range of alternative scenarios.


I'm not familiar with this field, but it seems like they have identified a different set of problems with and solutions for goal inference, and it would be useful to see how they apply to IRL. Perhaps the explicit landmark inference leads to more hierarchy in goals? Maybe the unique landmark heuristic is not captured in the standard Boltzmann-rational assumption in IRL? I'd also be interested to see if we can apply IRL algorithms to the plan recognition dataset and get good performance.

Heuristic Approaches for Goal Recognition in Incomplete Domain Models

The planning community works on algorithms that can plan given a _symbolic_ definition of the environment, how actions affect the environment, and the goal state; analogous to reinforcement learning. The task of inverting the optimal behavior to infer the goal is called goal recognition or plan recognition (analogous to inverse reinforcement learning). This paper looks at goal recognition where the models of the world are incomplete, so that there are _possible_ preconditions and effects of actions. They extract potential _landmarks_ from the plan, which are things (facts or actions) that must happen in order to achieve the goal, and then suggest two heuristics for how to use the landmarks to rank among possible goals.


I'm excited to see approaches to reward learning that use information from natural language -- that definitely seems like a rich source of information we have not made use of yet. That said, there are a lot of moving parts in this system and the parts that analyze natural language impose a lot of structure, to the extent that it feels like a domain-specific language instead of natural language. (I'm not confident about this, it's hard to tell from the paper.) I also don't understand the experiments. They say that the sparse reward couldn't train policies with episode length 3 -- sparse reward is a hard problem, but it isn't _that_ hard, so I must be misunderstanding something here.

Reward Learning from Narrated Demonstrations

Hsiao-Yu Fish Tung, Adam W. Harley, Liang-Kang Huang, Katerina Fragkiadaki

This paper learns and optimizes rewards given demonstrations of behavior along with a description of the behavior in natural language. Their dataset is a set of videos of humans demonstrating a task and describing it with natural language (such as "the orange is in the bowl"). They combine several techniques to use this dataset to teach a robot. First, using speech recognition, they get a transcript of the natural language aligned with the video. They use object detectors to figure out what things are present in the image, and a syntactic parser to figure out the subject and object of the sentence, and match up these two results to figure out which objects in the image the natural language refers to, and extract their spatial features. They then train a classifier to take the spatial features and detecting whether it has achieved the goal, conditioned on the natural language description of the task. Now that they have a reward function (1 at a goal state, 0 otherwise) they can train a robot using DQN, though to get this to work they infer 3D object configurations from 2D images and use distance to the goal as a shaped reward.


AI Alignment Podcast: Human Cognition and the Nature of Intelligence

Joshua Greene's lab has two research directions. The first is how we combine concepts to form thoughts: a process which allows us to understand arbitrary novel scenarios (even ones we don't think ever occurred). He discusses some of his recent reseach, which uses brain imaging to infer what's happening when humans think about compound concepts. While Joshua considers the combinatorial nature of thought to be important, he argues that to build AGI, it's necessary to start with "grounded cognition" in which representations are derived from perception and physical action, rather than just learning to manipulate symbols (like language).

Joshua also works on the psychology and neuroscience of morality. He discusses his recent work in which participants are prompted to consider Rawls' Veil of Ignorance argument (that when making decisions affecting many people, we should do so as if we don't know which one we are) and then asked to evaluate moral dilemmas such as trolley problems. Joshua argues that the concept of impartiality is at the core of morality, and that it pushes people towards more utilitarian ideas (although he wants to rebrand utilitarianism as "deep pragmatism" to address its PR problems).


SafeAI 2019: Proceedings of the AAAI Workshop on Artificial Intelligence Safety 2019

Imitation Learning from Imperfect Demonstration

Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, Masashi Sugiyama


Learning User Preferences via Reinforcement Learning with Spatial Interface Valuing


I found the categorization useful for understanding the differences between views on human values, which can be quite varied and hard to compare.

What AI Safety Researchers Have Written About the Nature of Human Values

This post categorizes theories of human values along three axes. First, how complex is the description of the values? Second, to what extent are "values" defined as a function of behavior (as opposed to being a function of eg. the brain's algorithm)? Finally, how broadly applicable is the theory: could it apply to arbitrary minds, or only to humans? The post then summarizes different positions on human values that different researchers have taken.


On the Utility of Model Learning in HRI

Rohan Choudhury, Gokul Swamy, Dylan Hadfield-Menell, Anca Dragan


This is a cool example of how huge amounts of existing unlabeled video data might be utilised. The task they attempt is significantly more complex than those in other similar work (such as [this paper](https://arxiv.org/abs/1805.11592) which learns to play Atari games from Youtube videos); however, this also makes it difficult to judge how well the learned policy performed, and how much potential it has to transfer into the real world.

Learning from Demonstration in the Wild

Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu, João Gomes, Supratik Paul, Frans A. Oliehoek, João Messias, Shimon Whiteson

This paper learns traffic trajectories from unsupervised data by converting traffic camera footage into a Unity scene simulation, using that simulation to generate pseudo-LIDAR readings for each "expert trajectory", and then training an agent to imitate them using a variant of generative adversarial imitation learning (GAIL).


One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks

Tianhe Yu, Pieter Abbeel, Sergey Levine, Chelsea Finn


Shared autonomy is an interesting setting because the human is still necessary in order to actually perform the task, whereas in typical reward learning settings, once you have learned the reward function and the AI is performing well, the human does not need to be present in order to execute a good policy.

Shared Autonomy via Deep Reinforcement Learning

In shared autonomy, an AI system assists a human to complete a task. The authors implement shared autonomy in a deep RL framework by simply extending the state with the control input from the human, and then learning a policy that chooses actions given the extended state. They show that the human-AI team performs better than either one alone in the Lunar Lander environment.


The key insight that rewards are _not_ invariant to additions of a constant when you have variable-length episodes is useful and I'm glad that it's been pointed out, and a solution proposed. However, the experiments are really strange -- in one case (Figure 4, HalfCheetah) their algorithm outperforms the expert (which has access to the true reward), and in another (Figure 5, right) the blue line implies that using a uniformly zero reward lets you achieve around a third of expert performance (!!).

Addressing Sample Inefficiency and Reward Bias in Inverse Reinforcement Learning

Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, Jonathan Tompson

Deep IRL algorithms typically work by training a discriminator that distinguishes between states and actions from the expert from states and actions from the learned policy, and extracting a reward function from the discriminator. In any environment where the episode can end after a variable number of timesteps, this assumes that the reward is zero after the episode ends. The reward function from the discriminator often takes a form where it must always be positive, inducing a survival incentive, or a form where it must always be negative, inducing a living cost. For example, [GAIL](https://arxiv.org/abs/1606.03476)'s reward is always positive, giving a survival incentive. As a result, _without any reward learning at all_ GAIL does better on Hopper than behavioral cloning, and fails to learn on a reaching or pushing task (where you want to do the task as quickly as possible, so you want the living cost). To solve this, they learn an "absorbing state reward", which is a reward given after the episode ends -- this allows the algorithm to learn for itself whether it should have a survival incentive or living cost.

They also introduce a version that keeps a replay buffer of experience and uses an off-policy algorithm to learn from the replay buffer in order to improve sample efficiency.


International Conference on Innovative Techniques and Applications of Artificial Intelligence 2018

Task-Embedded Control Networks for Few-Shot Imitation Learning

Stephen James, Michael Bloesch, Andrew J. Davison


Few-Shot Goal Inference for Visuomotor Learning and Planning

Annie Xie, Avi Singh, Sergey Levine, Chelsea Finn


Video Imitation GAN: Learning control policies by imitating raw videos using generative adversarial reward estimation

Subhajit Chaudhury, Daiki Kimura, Tu-Hoa Pham, Asim Munawar, Ryuki Tachibana


Proceedings of the AAAI Workshop on Artificial Intelligence Safety 201

Inspiration Learning through Preferences


Active Inverse Reward Design

Sören Mindermann, Rohin Shah, Adam Gleave, Dylan Hadfield-Menell

Inverse Reward Design (IRD) introduced the idea that a hand-designed reward function should be treated as an _observation_ about the intended reward function. Any particular hand-designed reward function (called a _proxy reward_) is likely if it incentivizes good behavior in the training environment, as measured by the true reward function. In this paper, instead of asking the reward designer to choose a proxy reward from the space of all possible reward functions, the designer is presented with a small subset of possible reward functions and asked to choose the best option from those. The subset is chosen such that the answer will be maximally informative about the true reward (in expectation). This is an easier query for the reward designer to answer, and can convey more information about the true reward function. To see this, we can imagine pairing each possible reward function with the trajectory that it incentivizes in the training environment. Then original IRD gets information about the best such trajectory, which could correspond to multiple true reward functions, whereas active IRD can get information about the best trajectory in any subset of trajectories, and so can get more information in total. In some cases, that extra information can narrow down the space of possible reward functions to a single true reward function. The paper discusses two kinds of queries that can be asked (discrete and continuous), and several optimizations for actually computing the best query to ask (which can be computationally intensive). The technique is demonstrated on a contextual bandits problem and gridworlds.


Sample-Efficient Imitation Learning via Generative Adversarial Nets


I only skimmed through the math, so I don't understand the paper well enough to have a good opinion on it. The overall objective of having more risk-sensitivity seems useful for safety. That said, I do find the VNM utility theorem compelling, and it suggests that risk aversion is a bad strategy. I currently resolve this by saying that while the VNM theorem is true, if you want to optimize expected reward over a long time horizon in an environment with high-downside actions but not high-upside actions, even if you are maximizing expected utility you would not take low-probability-of-high-downside actions. (Here a high-downside action is one that causes something like death/episode termination.) Since humans are (probably) scope-insensitive with respect to time, it becomes important for humans to have a heuristic of risk aversion in order to actually maximize expected utility in practice. I'd be interested in seeing experiments with current (risk neutral) RL algorithms in long-horizon environments with actions with high downside, and see if they automatically learn behavior that we would call "risk-averse".

Take this with a grain of salt -- it's a lot more speculative than most of my opinions, which can already be quite speculative. Most of the steps in that argument are handwavy intuitions I have that aren't based on any research that's been done (though I haven't looked for any such research). Though you can think of the argument for focusing on long-term AI safety at all as an instance of this idea, where the argument is that our risk-aversion heuristic is only sufficient for timescales on the orders of human lifetimes, not for cosmic timescales, and so we should explicitly be more risk-averse and focus on reducing existential risk.

Risk-Sensitive Generative Adversarial Imitation Learning

Jonathan Lacotte, Mohammad Ghavamzadeh, Yinlam Chow, Marco Pavone

This paper extends GAIL to perform imitation learning where we try to optimize a policy for the mean reward collected under the constraint that the policy is no more risky than the expert policy. Since we don't know the true cost function, we have to approximate this problem with another problem where we infer the cost function as well, and evaluate the risk profile relative to the inferred cost function. The algorithm ends up looking very similar to the original GAIL algorithm, where the gradient updates change in order to include terms dependent on the conditional value-at-risk (CVaR). They evaluate against GAIL and RAIL (another risk-sensitive imitation learning algorithm) and find that their method performs the best on the Hopper and Walker Mujoco environments.


Entropic Regret I: Deterministic MDPs


One thing to note about this paper is that it is an incredible work of scholarship; it fluently cites research across a variety of disciplines, including AI safety, and provides a useful organizing framework for many such papers. If you need to do a literature review on inverse reinforcement learning, this paper is a good place to start.

Inverse Decision Modeling: Learning Interpretable Representations of Behavior

Daniel Jarrett*, Alihan Hüyük*, Mihaela van der Schaar

There’s lots of work on learning preferences from demonstrations, which varies in how much structure they assume on the demonstrator: for example, we might consider them to be <@Boltzmann rational@>(@Modeling Interaction via the Principle of Maximum Causal Entropy@) or [risk sensitive](https://rss2017.lids.mit.edu/static/papers/62.pdf), or we could try to <@learn their biases@>(@Learning biases and rewards simultaneously@). This paper proposes a framework to encompass all of these choices: the core idea is to model the demonstrator as choosing actions according to a _planner_; some parameters of this planner are fixed in advance to provide an assumption on the structure of the planner, while others are learned from data. This also allows them to separate beliefs, decision-making, and rewards, so that different structures can be imposed on each of them individually.

The paper provides a mathematical treatment of both the forward problem (how to compute actions in the planner given the reward, think of algorithms like value iteration) and the backward problem (how to compute the reward given demonstrations, the typical inverse reinforcement learning setting). They demonstrate the framework on a medical dataset, where they introduce a planner with parameters for flexibility of decision-making, optimism of beliefs, and adaptivity of beliefs. In this case they specify the desired reward function and then run backward inference to conclude that, with respect to this reward function, clinicians appear to be significantly less optimistic when diagnosing dementia in female and elderly patients.


One way this paper differs from my intuitions is that it finds that assuming Boltzmann rationality performs very poorly if the demonstrator is in fact systematically suboptimal. I would have instead guessed that Boltzmann rationality would do okay -- not as well as in the case where there is no misspecification, but only a little worse than that. (That’s what I found in <@my paper@>(@Learning biases and rewards simultaneously@), and it makes intuitive sense to me.) Some hypotheses for what’s going on, which the lead author agrees are at least part of the story:

1. When assuming Boltzmann rationality, you infer a distribution over reward functions that is “close” to the correct one in terms of incentivizing the right behavior, but differs in rewards assigned to suboptimal behavior. In this case, you might get a very bad log loss (the metric used in this paper), but still have a reasonable policy that is decent at acquiring true reward (the metric used in my paper).

2. The environments we’re using may differ in some important way (for example, in the environment in my paper, it is primarily important to identify the goal, which might be much easier to do than inferring the right behavior or reward in the autonomous driving environment used in this paper).

Human irrationality: both bad and good for reward inference

Lawrence Chan, Andrew Critch, Anca Dragan

Last summary, we saw a framework for inverse reinforcement learning with suboptimal demonstrators. This paper instead investigates the qualitative effects of performing inverse reinforcement learning with a suboptimal demonstrator. The authors modify different parts of the Bellman equation in order to create a suite of possible suboptimal demonstrators to study. They run experiments with exact inference on random MDPs and FrozenLake, and with approximate inference on a simple autonomous driving environment, and conclude:

1. **Irrationalities can be helpful for reward inference**, that is, if you infer a reward from demonstrations by an irrational demonstrator (where you know the irrationality), you often learn _more_ about the reward than if you inferred a reward from optimal demonstrations (where you know they are optimal). Conceptually, this happens because optimal demonstrations only tell you about what the best behavior is, whereas most kinds of irrationality can also tell you about preferences between suboptimal behaviors.

2. **If you fail to model irrationality, your performance can be very bad**, that is, if you infer a reward from demonstrations by an irrational demonstrator, but you assume that the demonstrator was Boltzmann rational, you can perform quite badly.


NeurIPS 2021 Track Datasets and Benchmarks

While the authors do a good job advocating for the problem of preference-based RL, I'm less convinced their particular benchmark is a large step forward. In particular, it seems the main contribution is not a suite of tasks, but rather a collection of different ways to add irrationality to the teacher oracle. The main takeaway of this paper is that current algorithms don't seem to perform well when the teacher can make mistakes, but this is quite similar to having a misspecified reward function. Beyond that criticism, the experiments support the areas suggested for future work.

B-Pref: Benchmarking Preference-Based Reinforcement Learning

Kimin Lee, Laura Smith, Anca Dragan, Pieter Abbeel

Deep RL has become a powerful method to solve a variety of sequential decision tasks using a known reward function for training. However, in practice, rewards are hard to specify making it hard to scale Deep RL for many applications. Preference-based RL provides an alternative by allowing a teacher to indicate preferences between a pair of behaviors. Because the teacher can interactively give feedback to an agent, preference-based RL has the potential to help address this limitation of Deep RL. Despite the advantages of preference-based RL it has proven difficult to design useful benchmarks for the problem. This paper introduces a benchmark (B-Pref) that is useful for preference-based RL in various locomotion and robotic manipulation tasks.

One difficulty with designing a useful benchmark is that teachers may have a variety of irrationalities. For example, teachers might be myopic or make mistakes. The B-Pref benchmark addresses this by emphasizing measuring performance under a variety of teacher irrationalities. They do this by providing various performance metrics to introduce irrationality into otherwise deterministic reward criteria. While previous approaches to preference-based RL work well when the teacher responses are consistent, experiments show they are not robust to feedback noise or teacher mistakes. Experiments also show that how queries are selected has a major impact on performance. With these results, the authors identify these two problems as areas for future work.


This seems like a reasonable approach. It has a similar ethos as Boltzmann rationality. In Boltzmann rationality, it seems like all you need to do is model the demonstrator as having some noise but still being more likely to choose higher-reward actions, and that’s enough to get decent performance; similarly here you just need to model different demonstrators as applying different amounts of Gaussian noise to the optimal policy and that’s enough to distinguish good from bad.

Note that, while the experimental results are good, the paper doesn’t have experiments with real human demonstrations; as we saw in the previous summary these can often be quite different (in ways that matter) from machine-generated demonstrations.

VILD: Variational Imitation Learning with Diverse-quality Demonstrations

Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, Masashi Sugiyama

We saw in the previous summary that existing methods struggle to cope with datasets of demonstrations of mixed quality. This paper aims to tackle exactly this problem. They consider a model in which there are k demonstrators with varying levels of quality. Each demonstrator is modeled as computing an action Boltzmann-rationally and then applying some Gaussian noise; the standard deviation of the Gaussian noise differs across the demonstrators (with higher standard deviation corresponding to lower quality).

They use variational inference to derive an algorithm for this problem that infers the reward function as well as an optimal policy to go along with it. In addition, they oversample data from the demonstrations that the model thinks are high quality in order to get more informative gradients. (They use an importance sampling correction in order to keep the gradient estimate unbiased.)

Their experiments on machine-generated data show significant improvement over existing imitation learning algorithms, both in the case where we synthetically add Gaussian noise (matching the model) and when we add time-signal-dependent (TSD) noise (in which case the model is misspecified).


I found the experimental comparisons to be a bit misleading. If you compare with the results in the original ValueDICE and SQIL paper the algorithms are closer in performance than this paper implies. It's also not clear that you need to use the soft-Bellman operator especially in the continuous-control setting which was what ValueDICE originally focused on. However, overall, non-adversarial methods are generally more stable so I found this paper to be a good contribution.

IQ-Learn: Inverse soft-Q Learning for Imitation

Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, Stefano Ermon

A popular way to view imitation learning is as a distribution matching problem. In this approach, the goal is to have the imitator induce a state-action distribution that closely matches that of the expert. Methods such as <@GAIL@>(@Generative Adversarial Imitation Learning@) and <@Value-DICE@>(@Imitation Learning via Off-Policy Distribution Matching@) propose adversarial methods, similar to GANs, to carry out the distribution matching. However, such methods can be difficult to train due to the difficulty of solving saddle-point problems. In this paper, the authors present a non-adversarial method that allows distribution matching to be carried out in a fully offline and non-adversarial fashion. They do this by building on Value-DICE and introducing a soft-Bellman operator which allows the saddle-point problem to be reduced to estimating a Q-function. In fact, the authors show this reduction is related to off-policy RL algorithms with the reward set to zero. In experiments, the method is shown to be competitive with other state-of-the-art methods in both the offline and image-based setting.


The paper is organized well and I found the examples to be interesting in their own right. On the other hand, binary classification is a fairly restrictive setting and IDT in this paper seems to require access to class posterior probabilities. These probabilities generally are not easy to estimate. Moreover, if you have access to that function it seems you could elicit the loss function with exponentially fewer human observations by sorting/sub-sampling the class posterior values. Despite these shortcomings, I'm interested to see how this work can be extended further.

Uncertain Decisions Facilitate Better Preference Learning

Human preference learning has been studied from various perspectives such as inverse reinforcement learning (IRL) and active learning. However, the IRL problem is underspecified, that is, even with access to the full behavioral policy, you cannot uniquely determine the preferences that led to that policy. Meanwhile, active learning often has a **description-experience gap**: the stated preferences in response to a question in active learning may not be the same as the preferences that would be revealed from demonstrations.

In this work, the authors study an alternative paradigm known as inverse decision theory (IDT) that aims to learn a loss function for binary classification using strictly observational data while returning unique solutions. (Such a loss function effectively specifies how good correct predictions are and how bad incorrect predictions are.) The authors show that preferences can be uniquely determined whenever there is uncertainty in the classification problem. This happens because we need observations predicting classes at different levels of certainty to identify a transition point where we switch from predicting one class over another. In contrast, without uncertainty, we won’t be able to precisely identify that threshold. The authors then strengthen this result by showing it holds even in cases where the underlying decision rule is sub-optimal.

The authors argue that since learning could be done efficiently in this setting, IDT could have broader applicability. For example, one application to fairness could be to collect a set of decisions from a trained classifier, split them across groups (e.g. race or gender), and compare the inferred loss functions to detect bias in the trained classifier.


[Identifiability in inverse reinforcement learning](https://arxiv.org/abs/2106.03498) has the same motivation and studies a very similar setting, but has a few different results. It's also easier to read if you're not as familiar with MaxEnt methods.

Reward Identification in Inverse Reinforcement Learning

Kuno Kim, Kirankumar Shiragur, Shivam Garg, Stefano Ermon

As mentioned in the previous summary, a major challenge with inverse reinforcement learning is that rewards are unidentifiable: even given perfect knowledge of the policy, we cannot recover the reward function that produces it. This is partly for boring reasons like “you can add a constant to a reward function without changing anything”, but even if you exclude those kinds of reasons, others remain. For example, since every policy is optimal for the constant reward function, the zero reward function can rationalize any policy.

For this reason, the authors instead focus on the case where we assume the policy is a solution to the maximum entropy RL objective (you can think of this as Boltzmann rationality, if you’re more familiar with that). The solution to MaxEnt RL for a zero reward is a uniformly random policy, so the zero reward no longer rationalizes every policy. Perhaps rewards are identifiable in this case?

(You might have noticed that I neglected the question of whether the MaxEnt RL model was better than the regular RL model in cases that we care about. As far as I can tell the paper doesn’t address this. But if they did so, perhaps they might say that in realistic situations we are dealing with boundedly-rational agents, and Boltzmann rationality / MaxEnt RL is a common model in such situations.)

Well, we still need to deal with the “additive constant” argument. To address this, the authors define two reward functions to be equivalent if they agree up to an additive constant. There are actually two versions of this: “trajectory equivalence” means that they agree on the rewards for all feasible trajectories, while “state-action equivalence” means that they agree on the rewards for all state-action pairs. Correspondingly, “weak identifiability” means that you can identify rewards up to trajectory equivalence, while “strong identifiability” means you can identify them up to state-action equivalence. Strong identifiability implies weak identifiability, since if you know the rewards on state-action pairs, that determines the reward for any given trajectory.

All deterministic MDPs are weakly identifiable under the MaxEnt RL model, since in this case a trajectory τ is selected with probability p(τ) proportional to exp(r(τ)), so the probability p(τ) can then be inverted to get r(τ). However, stochastic MDPs need not be weakly identifiable. Imagine an MDP in which no matter what you do, you are teleported to a random state. In such an MDP, the agent has no control over the trajectory, and so the MaxEnt RL objective will choose a uniformly random policy, no matter what the reward is, and so the reward must be unidentifiable.

Now the question is, assuming you have weak identifiability (i.e. you can infer r(τ)), when do you also have strong identifiability (i.e. you can infer r(s, a))? Intuitively, there needs to be a sufficient “diversity” of feasible trajectories τ, that cover a wide variety of possible (s, a) pairs, so that you can use the r(τ) values to infer the r(s, a) values. The authors prove a sufficient condition called “coverage”: there exists some timestep T, such that for every state there is some feasible trajectory that reaches that state at timestep T. (They also require the horizon to be at least 2T.) Coverage can be a fairly easy property to have; for example, if you can get to any state from any other state in some number of steps, then all you need is a single self-loop somewhere in the MDP that allows you to “waste time” so that you reach the desired state at exactly timestep T (instead of reaching too early).


Hierarchy seems to be a very important tool that humans use, so I'm glad to see work on it. Currently, the algorithm is very computationally expensive, and can only be applied in small domains right now, and requires the options to be specified ahead of time, but it does lead to a benefit on the environments they consider, despite the inevitable misspecification from having to hardcode the options. I would be very interested to see an extension to high-dimensional data where the options are learned (analogous to [Meta-Learning Shared Hierarchies](https://blog.openai.com/learning-a-hierarchy/) for hierarchical RL). Not only would this be more realistic, it could perform better because the options would be learned, not hardcoded.

Exploring Hierarchy-Aware Inverse Reinforcement Learning

One heuristic that humans use to deal with bounded computation is to make plans hierarchically, building long-term plans out of slightly smaller building blocks. How can we incorporate this knowledge into an IRL algorithm? This paper extends [Bayesian IRL](https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-416.pdf) to the setting where the demonstrator has access to a set of _options_, which are (to a first approximation) policies that can be used to achieve some subgoal. Now, when you are given a trajectory of states and actions, it is no longer clear which options the demonstrator was using to generate that trajectory. The authors provide an algorithm that can enumerate all the options that are consistent with the trajectory, and assign probabilities to them according to the Boltzmann-rational model. They evaluate on a taxi driver gridworld often used in hierarchical planning, as well as on real human data from a game called Wikispeedia.


We can look at this from an alignment perspective -- the child is giving the AI system a misspecified reward, relative to the parent's goal of "provide good suggestions that do not have inappropriate content". While the researchers solve it using contextual bandits, it could be interesting to consider how AI alignment approaches could deal with this situation.

Using Contextual Bandits with Behavioral Constraints for Constrained Online Movie Recommendation

IBM researchers train AI to follow code of ethics

Parents want movie recommendation systems not to recommend particular kinds of movies to children, but we would also like the recommendation system to suggest movies that the children will actually like. Researchers solved this problem by first learning a model for what kinds of movies should not be recommended, and then combined that with a contextual bandit model that learns online from the child's data to provide good suggestions that follow the parent's constraints.


Generative Adversarial Imitation from Observation

Faraz Torabi, Garrett Warnell, Peter Stone


Ultimately my guess is that if you want to predict human judgments well, you need to train against human judgments, rather than the proxy task of distinguishing between human and agent behavior. That being said, I do think these proxy tasks can serve as valuable pretraining objectives, or as auxiliary objectives that help to improve sample efficiency.

Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation

Sam Devlin*, Raluca Georgescu*, Ida Momennejad*, Jaroslaw Rzepecki*, Evelyn Zuniga*, Gavin Costello, Guy Leroy, Ali Shaw, Katja Hofmann

Since rewards are hard to specify, we are likely going to have to train AI agents using human feedback. However, human feedback is particularly expensive to collect, so we would like to at least partially automate this using reward models. This paper looks at one way of building such a reward model: training a classifier to distinguish between human behavior and agent behavior (i.e. to be the judge of a Turing Test). This is similar to the implicit or explicit reward model used in adversarial imitation learning algorithms such as <@GAIL@>(@Generative Adversarial Imitation Learning@) or <@AIRL@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@).

Should we expect these classifiers to generalize, predicting human judgments of how human-like a trajectory is on all possible trajectories? This paper conducts a user study in order to answer the question: specifically, they have humans judge several of these Turing Tests, and see whether the classifiers agree with the human judgments. They find that while the classifiers do agree with human judgments when comparing a human to an agent (i.e. the setting on which the classifiers were trained), they do not agree with human judgments when comparing two different kinds of artificial agents. In fact, it seems like they are _anti-correlated_ with human judgments, rather than simply having no correlation at all -- only one of the six classifiers tested does better than chance (at 52.5%), the median is 45%, and the worst classifier gets 22.5%. (Note however that the sample size is small, I believe n = 40 though I’m not sure.)


[Paper: Learning What To Do by Simulating the Past](https://arxiv.org/abs/2104.03946)

[Thesis: Extracting and Using Preference Information from the State of the World](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-210.html)

David Lindner, Rohin Shah, Pieter Abbeel, Anca Dragan

Since the state of the world has already been optimized for human preferences, it can be used to infer those preferences. For example, it isn’t a coincidence that vases tend to be intact and on tables. An agent with an understanding of physics can observe that humans haven’t yet broken a particular vase, and infer that they care about vases not being broken.

<@Previous work@>(@Learning Preferences by Looking at the World@) provides an algorithm, RLSP, that can perform this type of reasoning, but it is limited to small environments with known dynamics and features. In this paper (on which I am an author), we introduce a deep variant of the algorithm, called Deep RLSP, to move past these limitations. While RLSP assumes known features, Deep RLSP learns a feature function using self-supervised learning. While RLSP computes statistics for all possible past trajectories using dynamic programming, deep RLSP learns an inverse dynamics model and inverse policy to _simulate_ the most likely past trajectories, which serve as a good approximation for the necessary statistics.

We evaluate the resulting algorithm on a variety of Mujoco tasks, with promising results. For example, given a single state of a HalfCheetah balancing on one leg, Deep RLSP is able to learn a (noisy) policy that somewhat mimics this balancing behavior. (These results can be seen [here](https://sites.google.com/view/deep-rlsp).)


Paper: Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification

Recursive Classification: Replacing Rewards with Examples in RL

Benjamin Eysenbach, Sergey Levine, Ruslan Salakhutdinov

Previous work has suggested learning a reward model from examples of successfully solving the task. This paper suggests that rather than a two stage process of learning a reward model and then optimizing it using RL, we can instead directly learn a policy from the examples by building an equivalent of Bellman backups that apply directly to examples (rather than having to go through intermediate rewards). Their experiments show that this works well.


Four Motivations for Learning Normativity

We’ve <@previously seen@>(@Learning Normativity: A Research Agenda@) desiderata for agents that learn normativity from humans: specifically, we would like such agents to:

1. **Learn at all levels:** We don’t just learn about uncertain values, we also learn how to learn values, and how to learn to learn values, etc. There is **no perfect loss function** that works at any level; we assume conservatively that Goodhart’s Law will always apply. In order to not have to give infinite feedback for the infinite levels, we need to **share feedback between levels**.
2. **Learn to interpret feedback:** Similarly, we conservatively assume that there is **no perfect feedback**; so rather than fixing a model for how to interpret feedback, we want feedback to be **uncertain** and **reinterpretable**.
3. **Process-level feedback:** Rather than having to justify all feedback in terms of the consequences of the agent’s actions, we should also be able to provide feedback on the way the agent is reasoning. Sometimes we’ll have to judge the entire chain of reasoning with **whole-process feedback**.

This post notes that we can motivate these desiderata from multiple different frames:

1. _Outer alignment:_ The core problem of outer alignment is that any specified objective tends to be wrong. This applies at all levels, suggesting that we need to **learn at all levels**, and also **learn to interpret feedback** for the same reason. **Process-level feedback** is then needed because not all decisions can be justified based on consequences of actions.
2. _Recovering from human error:_ Another view that we can take is that humans don’t always give the right feedback, and so we need to be robust to this. This motivates all the desiderata in the same way as for outer alignment.
3. _Process-level feedback:_ We can instead view process-level feedback as central, since having agents doing the right type of _reasoning_ (not just getting good outcomes) is crucial for inner alignment. In order to have something general (rather than identifying cases of bad reasoning one at a time), we could imagine learning a classifier that detects whether reasoning is good or not. However, then we don’t know whether the reasoning of the classifier is good or not. Once again, it seems we would like to **learn at all levels**.
4. _Generalizing learning theory:_ In learning theory, we have a distribution over a set of hypotheses, which we update based on how well the hypotheses predict observations. **Process-level feedback** would allow us to provide feedback on an individual hypothesis, and this feedback could be **uncertain**. **Reinterpretable feedback** on the other hand can be thought of as part of a (future) theory of meta-learning.


Here, the task definition continues to be the reward function, and the human demonstration is used to help the agent effectively optimize the reward function. Such agents are still vulnerable to misspecified reward functions -- in fact, the agent discovers a bug in the emulator that wouldn't have happened if it was trying to imitate the human. I would still expect the agent to be more human-like than one trained with standard RL, since it only learns the environment near the human policy.

Learning Montezuma’s Revenge from a Single Demonstration

Montezuma's Revenge is widely considered to be one of the hardest Atari games to learn, because the reward is so sparse -- it takes many actions to reach the first positive reward, and if you're using random exploration, it will take exponentially many actions (in N, the number of actions till the first reward) to find any reward. A human demonstration should make the exploration problem much easier. In particular, we can start just before the end of the demonstration, and train the RL agent to get as much score as the demonstration. Once it learns that, we can start it at slightly earlier in the demonstration, and do it again. Repeating this, we eventually get an agent that can perform the whole demonstration from start to finish, and it takes time linear in the length of the demonstration. Note that the agent must be able to generalize a little bit to states "around" the human demonstration -- when it takes random actions it will eventually reach a state that is similar to a state it saw earlier, but not exactly the same, and it needs to generalize properly. It turns out that this works for Montezuma's Revenge, but not for other Atari games like Gravitar and Pitfall.


Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning

Valerie Chen, Abhinav Gupta, Kenneth Marino

It is particularly challenging for RL agents to perform hierarchical tasks when there is only a sparse reward. One natural piece of feedback in this setting is instructions in natural language specifying the different subtasks needed to solve the task. In particular, this paper assumes we have access to a dataset of human demonstrations paired with natural language instructions for each subtask that they complete.

We then have an architecture that first generates the language instruction for the current subtask given the final task and the current state, and then takes a low-level action computed from the current state and the language instruction. This is trained via imitation learning on the human demonstrations.

Using a small Minecraft-inspired gridworld, the authors show that the language generation is crucial for good generalization: if the agent is trained on “cobblestone block” and “iron ingot”, then it is able to generalize to cobblestone ingot, _as long as_ it was trained to generate the language instruction as well. Intuitively, the combinatorial structure of language leads to better generalization than direct imitation on low-level actions.


My understanding is the MEM only handles the six natural language instructions used in the StarCraft minigame, and so is roughly equivalent to training six classifiers using the hardcoded programs to generate datasets. Thus, these two papers ultimately boil down to “decompose the task into six steps, train classifiers for these six steps, and then do RL where the reward function gives positive reward every time a particular step is marked as complete”.

However, this is primarily because the authors had to ground the natural language instructions themselves. If we could instead leverage a pretrained model which already grounds natural language, such as [CLIP](https://openai.com/blog/clip/), then it seems like this approach could in fact save a lot of human effort in specifying what the AI system should do.

A Narration-based Reward Shaping Approach using Grounded Natural Language Commands

Nicholas Waytowich, Sean L. Barton, Vernon Lawhern, Garrett Warnell

One way to specify what an AI system should do is to simply specify it in natural language. If we have some way to map natural language instructions to states, then we could turn natural language into a reward function and use RL to optimize it.

This paper proposes specifying a task by breaking it down into a sequence of steps to be completed. Given a mapping from natural language to states, they define a reward function that gives a positive reward every time the mapping detects that the agent has completed the next stage in the sequence of steps. They show that this outperforms vanilla reinforcement learning on a win/loss reward function in a StarCraft minigame.

For the mapping of language to states, the authors use a mutual embedding model (MEM) they developed in [previous work](https://arxiv.org/abs/1906.02671). The core idea is to write down programs that identify states matching a particular natural language instruction, use this to generate a dataset of states and the corresponding natural language instruction, and then training a model to map the natural language instructions to be “close to” the mappings of the states (which are produced by a CNN).


Learning Rewards from Linguistic Feedback

Theodore R. Sumers, Mark K. Ho, Robert D. Hawkins, Karthik Narasimhan, Thomas L. Griffiths

This paper proposes another approach to reinforcement learning using natural language. After the agent plays an episode, we can ask a human for feedback in natural language. We then take their response, figure out what features of the environment the response mentions, and then use sentiment analysis to determine how to update the weights on the features. For sentiment analysis we can use an off-the-shelf classifier; the hard part is in determining the relevant environment feature vectors:

1. **Evaluative** feedback is feedback about the trajectory the agent produced, for example “good job”, so we can just use the features of this trajectory.
2. **Imperative** feedback specifies what the agent should have done, e.g. “you should have gone to the top right corner”. In this case, we must find the features consistent with the given instruction.
3. **Descriptive** feedback provides feedback directly about the reward, for example “yellow objects are bad”. In this case, we use a feature vector that has a 1 for every feature mentioned (in this case, the feature for yellow objects) and 0 everywhere else.

Types 2 and 3 require some domain knowledge in order to write down programs that map language to the relevant features. The environment the authors used was simple enough that they were able to do this.

Once we have the feature vector f and the sentiment s, we perform a Bayesian update on our weight distribution. This is similar to the way we perform Bayesian updates on the reward distribution upon seeing a human action as evidence, as in <@Bayesian IRL@>(@Bayesian Inverse Reinforcement Learning@) or <@reward-rational implicit choice@>(@Reward-rational (implicit) choice: A unifying formalism for reward learning@).

This model so far performs reasonably well. By adding a couple of heuristics inspired by pragmatics (e.g. assuming that features that aren’t mentioned aren’t decision-relevant), they reach approximately human-level performance.


Probing Emergent Semantics in Predictive Agents via Question Answering

Imitating Interactive Intelligence

Interactive Agents Group, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Stephen Clark, Andrew Dudzik, Petko Georgiev, Aurelia Guy, Tim Harley, Felix Hill, Alden Hung, Zachary Kenton, Jessica Landon, Timothy Lillicrap, Kory Mathewson, Alistair Muldal, Adam Santoro, Nikolay Savinov, Vikrant Varma, Greg Wayne, Nathaniel Wong, Chen Yan, Rui Zhu

While <@existing@>(@Learning to Follow Language Instructions with Adversarial Reward Induction@) <@work@>(@Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text@) has trained agents to follow natural language instructions, it may be the case that achieving AGI requires more interactivity: perhaps we need to train agents to both give and follow instructions, or engage in a full dialogue, to accomplish tasks in a 3-D embodied environment. This paper makes progress on this goal.

The authors introduce a 3-D room environment in which agents can interact with objects and move them around, leading to a combinatorial space of possible high-level actions. So far the authors have only worked on question-answering (e.g. “what is the color of the chair?”) and instruction-following (e.g. “please lift up the purple object”), but they hope to eventually also work on dialogue and play.

They collect demonstrations of games between humans in which one human is given a goal, and then is asked to give a natural language instruction. The other human sees this instruction and must then execute it in the environment. The authors then use various kinds of imitation learning algorithms to learn a policy that can both set instructions and execute them. They also train models that can evaluate whether a particular trajectory successfully completes the goal or not.

The authors show that the learned policies are capable of some generalization -- for example, if during training they remove all rooms containing orange ducks (but don’t remove other orange objects, or other colors of duck), the resulting policies are still able to handle rooms containing orange ducks.


Learning Normativity: A Research Agenda

To build aligned AI systems, we need to have our AI systems learn what to do from human feedback. However, it is unclear how to interpret that feedback: any particular piece of feedback could be wrong; economics provides many examples of stated preferences diverging from revealed preferences. Not only would we like our AI system to be uncertain about the interpretation about any particular piece of feedback, we would also like it to _improve_ its process for interpreting human feedback. This would come from human feedback on the meta-level process by which the AI system learns. This gives us _process-level feedback_, where we make sure the AI system gets the right answers _for the right reasons_.

For example, perhaps initially we have an AI system that interprets human statements literally. Switching from this literal interpretation to a Gricean interpretation (where you also take into account the fact that the human chose to say this statement rather than other statements) is likely to yield improvements, and human feedback could help the AI system do this. (See also [Gricean communication and meta-preferences](https://www.alignmentforum.org/posts/8NpwfjFuEPMjTdriJ/gricean-communication-and-meta-preferences), [Communication Prior as Alignment Strategy](https://www.alignmentforum.org/posts/zAvhvGa6ToieNGuy2/communication-prior-as-alignment-strategy), and [multiple related CHAI papers](https://www.alignmentforum.org/posts/zAvhvGa6ToieNGuy2/communication-prior-as-alignment-strategy?commentId=uWBFsKK6XFbL4Hs4z).)

Of course, if we learn _how_ to interpret human feedback, that too is going to be uncertain. We can fix this by “going meta” once again: learning how to learn to interpret human feedback. Iterating this process we get an infinite tower of “levels” of learning, and at every level we assume that feedback is not perfect and the loss function we are using is also not perfect.

In order for this to actually be feasible, we clearly need to share information across these various “levels” (or else it would take infinite time to learn across all of the levels). The AI system should not just learn to decrease the probability assigned to a single hypothesis, it should learn what _kinds_ of hypotheses tend to be good or bad.


**On feedback types:** It seems like the scheme introduced here is relying quite strongly on the ability of humans to give good process-level feedback _at arbitrarily high levels_. It is not clear to me that this is something humans can do: it seems to me that when thinking at the meta level, humans often fail to think of important considerations that would be obvious in an object-level case. I think this could be a significant barrier to this scheme, though it’s hard to say without more concrete examples of what this looks like in practice.

**On interaction:** I’ve previously <@argued@>(@Human-AI Interaction@) that it is important to get feedback _online_ from the human; giving feedback “all at once” at the beginning is too hard to do well. However, the idealized algorithm here does have the feedback “all at once”. It’s possible that this is okay, if it is primarily process-level feedback, but it seems fairly worrying to me.

**On desiderata:** The desiderata introduced in the first post feel stronger than they need to be. It seems possible to specify a method of interpreting feedback that is _good enough_: it doesn’t exactly capture everything, but it gets it sufficiently correct that it results in good outcomes. This seems especially true when talking about process-level feedback, or feedback one meta level up -- as long as the AI system has learned an okay notion of “being helpful” or “being corrigible”, then it seems like we’re probably fine.

Often, just making feedback uncertain can help. For example, in the preference learning literature, Boltzmann rationality has emerged as the model of choice for how to interpret human feedback. While there are several theoretical justifications for this model, I suspect it is successful simply because it makes feedback uncertain: if you want to have a model that assigns higher likelihood to high-reward actions, but still assigns some probability to all actions, it seems like you end up choosing the Boltzmann model (or something functionally equivalent). Note that there is work trying to improve upon this model, such as by [modeling humans as pedagogic](https://papers.nips.cc/paper/2016/file/b5488aeff42889188d03c9895255cecc-Paper.pdf), or by <@incorporating a notion of similarity@>(@LESS is More: Rethinking Probabilistic Models of Human Behavior@).

So overall, I don’t feel convinced that we need to aim for learning at all levels. That being said, the second post introduced a different argument: that the method does as well as we “could” do given the limits of human reasoning. I like this a lot more as a desideratum; it feels more achievable and more tied to what we care about.

Recursive Quantilizers II

This post gives an example scheme inspired by the previous post. Like [iterated amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd), it defines an ideal (analogous to <@HCH@>(@Humans Consulting HCH@)), and then an approximation to it that could be computed in practice.

Like HCH, we imagine a tree of systems that improve as we increase the depth of the tree. However, the nodes in the tree are question-answering (QA) _systems_, rather than direct questions. Given a few QA systems from a lower level, we construct a QA system at a higher level by asking one low-level QA system “what’s a good safe distribution over QA systems”, and a different low-level QA system “what’s a good metric that we can use to judge QA systems”.  We then use [quantilization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) ([AN #48](https://mailchi.mp/3091c6e9405c/alignment-newsletter-48)) to select better-performing QA systems, without optimizing too hard and falling prey to Goodhart’s Law. In the infinite limit, this should converge to a stable equilibrium.

By having the tree reason about what good safe distributions are, and what good metrics are, we are explicitly improving the way that the AI system learns to interpret feedback (this is what the “good metric” is meant to evaluate), thus meeting the desiderata from the previous post.

To implement this in practice, we do something similar to iterated amplification. Iterated amplification approximates depth-limited HCH by maintaining a model that can answer _arbitrary_ questions (even though each node is a single question); similarly here we maintain a model that has a _distribution_ over QA systems (even though each node is a single QA system). Then, to sample from the amplified distribution, we sample two QA systems from the current distribution, ask one for a good safe distribution and the other for a good metric, and use quantilization to sample a new QA system given these ingredients. We use distillation to turn this slow quantilization process into a fast neural net model.

Considering the problem of <@inaccessible information@>(@Inaccessible information@), the hope is that, as we amplify the QA system, we will eventually be able to approve of some safe reasoning process about inaccessible information. If this doesn’t happen, then it seems that no human reasoning could approve of reasoning about that inaccessible information, so we have done as well as possible.


Understanding Learned Reward Functions

Eric J. Michaud, Adam Gleave, Stuart Russell

This paper investigates what exactly learned reward functions have learned, through the use of interpretability techniques. The authors hope that it will be easier to identify misalignment by directly interpreting learned reward functions as opposed to policies trained on them, as it seems plausible that reward functions will stay relatively similar in complexity, even when the policies become more complex as AI systems become more capable. Specifically, the authors look at:

1. Saliency maps, which plot the gradient of the reward with respect to each pixel, intuitively quantifying “how important is this pixel to the reward”
2. Occlusion maps, which show how much the reward changes if a certain area of the image is blurred
3. Counterfactual inputs, in which the authors manually craft input images to see what the learned reward function outputs.

In a simple gridworld where the agent must find the goal, the authors coded the reward function “1 if the agent moves to a previously visible goal location, else 0”, but they show that the learned reward is instead “0 if there is a currently visible goal location, else 1”. These are identical in the training environment, where there is always exactly one goal location (that the agent may be standing on, in which case that location is not visible). However, if there are changes at test time, e.g. multiple goal locations, then the learned reward will diverge from the true reward.

They then apply a similar methodology to Atari. They find that if the score is not hidden, then the learned reward model will simply check whether the score pixels are changing to detect reward -- _unless_ the score pixels change at a later time than reward is accrued, in which case this is not a viable strategy. They thus suggest that future reward learning work on Atari should ensure that the score is removed from the screen. Another important direction would be to develop new interpretability techniques that are specific to reward functions, e.g. to handle the fact that reward functions are invariant to potential shaping.


Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning

Xinlei Pan, Eshed Ohn-Bar, Nicholas Rhinehart, Yan Xu, Yilin Shen, Kris M. Kitani


Multi-agent Inverse Reinforcement Learning for General-sum Stochastic Games

Xiaomin Lin, Stephen C. Adams, Peter A. Beling


The derivation of the method is interesting in its own right since it also offers an alternative derivation of GAIL. However, the main assumption of this paper is that the agent policy can be decomposed into an expert and non-expert mixture. This seems like a strong assumption and makes me skeptical of the theoretical analysis. Nevertheless, the experimental results do indicate that RIL-Co is able to outperform other naive approaches. I'm also surprised this paper has no reference to DART, which directly analyzes imitation learning in the noise-injection setting. Specifically, given that these experts are deterministic and we have a significant number of demonstrations, I'd expect that a reasonable amount of noise should be directly removable with regular behavioral cloning. Yet, there is no comparison with behavioral cloning.

<@GAIL@>(@Generative Adversarial Imitation Learning@)

Robust Imitation Learning from Noisy Demonstrations

Voot Tangkaratt, Nontawat Charoenphakdee, Masashi Sugiyama

One weakness of vanilla imitation learning is that it struggles to handle demonstrations from sub-optimal experts. Let’s consider a simplified setting where the sub-optimal experts are modeled as optimal policies with injected gaussian noise. Ideally, the agent would learn to separate the noise from the true optimal policy.

This paper proposes an algorithm that is able to do this separation. The authors assume that the sub-optimal demonstrations and the learned agent policy can both be decomposed into a mixture of expert-policy and noise distributions. The main insight is that we can then learn a single classifier to distinguish noisy data from expert data; this classifier can then be used to define a reward function for an RL agent.

One issue is that since there is no ground truth for what is expert vs. noise, the classifier has to be trained on its own predictions, which can lead to overconfidence via positive feedback loops. To stabilize training, the authors train two models concurrently (co-training); each model is used to create training data for the other model. The authors call this approach RIL-Co. The experimental results show their algorithm RIL-Co is able to perform better than GAIL and other algorithms in the noisy regime.


The EMPATHIC Framework for Task Learning from Implicit Human Feedback

Yuchen Cui*, Qiping Zhang*, Alessandro Allievi, Peter Stone, Scott Niekum, W. Bradley Knox

A problem with learning from human feedback is that human feedback is quite expensive to collect. Can we instead learn from the facial expressions that humans automatically make anyway? This paper shows that the answer is yes: they first record human reactions while watching an autonomous agent, and use that to train a model that predicts reward given human reactions. They then transfer this model to a new task.


While I found the motivation and description of this paper somewhat unclear or misleading (Maia seems to me to be identical to behavior cloning, in which case it would not just be a “connection”), the experiments they run are pretty cool and it was interesting to see the pretty stark differences between models trained on a performance metric and models trained to imitate humans.

Aligning Superhuman AI and Human Behavior: Chess as a Model System

Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg, Ashton Anderson

Current AI systems are usually focused on some well-defined performance metric. However, as AI systems become more intelligent, we would presumably want to have humans learn from and collaborate with such systems. This is currently challenging since our superintelligent AI systems are quite hard to understand and don’t act in human-like ways.

The authors aim to study this general issue within chess, where we have access both to superintelligent AI systems and lots of human-generated data. (Note: I’ll talk about “ratings” below; these are not necessarily ELO ratings and should just be thought of as some “score” that functions similarly to ELO.) The authors are interested in whether AI systems play in a human-like way and can be used as a way of understanding human gameplay. One particularly notable aspect of human gameplay is that there is a wide range in skill: as a result we would like an AI system that can make predictions conditioned on varying skill levels.

For existing algorithms, the authors analyze the traditional Stockfish engine and the newer Leela (an open-source version of <@AlphaZero@>(@AlphaZero: Shedding new light on the grand games of chess, shogi and Go@)). They can get varying skill levels by changing the depth of the tree search (in Stockfish) or changing the amount of training (in Leela).

For Stockfish, they find that _regardless of search depth_, Stockfish action distributions monotonically increase in accuracy as the skill of the human goes up -- even when the depth of the search leads to a Stockfish agent with a similar skill rating as an amateur human. (In other words, if you take a low-ELO Stockfish agent and treat it as a predictive model of human players, it isn’t a great predictive model ever, but it is best at predicting human experts, not human amateurs.) This demonstrates that Stockfish plays very differently than humans.

Leela on the other hand is somewhat more human-like: when its rating is under 2700, its accuracy is highest on amateur humans; at a rating of 2700 its accuracy is about constant across humans, and above 2700 its accuracy is highest on expert humans. However, its accuracy is still low, and the most competent Leela model is always the best predictor of human play (rather than the Leela model with the most similar skill level to the human whose actions are being predicted).

The authors then develop their own method, Maia. They talk about it as a “modification of the AlphaZero architecture”, but as far as I can tell it is simply behavior cloning using the neural net architecture used by Leela. As you might expect, this does significantly better, and finally satisfies the property we would intuitively want: the best predictive model for a human of some skill level is the one that was trained on the data from humans at that skill level.

They also investigate a bunch of other scenarios, such as decisions in which there is a clear best action and decisions where humans tend to make mistakes, and find that the models behave as you’d expect (for example, when there’s a clear best action, model accuracy increases across the board).


The experiments in this paper are a bit underwhelming. While they run a large number of experiments, ONAIL only occasionally outperforms value-dice consistently in the HalfCheetah environment. The authors justify this by noting that ONAIL wasn't regularized. Additionally, the policies are initialized with behavior cloning, something that value-dice doesn't require. However, the theoretical insight on iterative tightening is interesting, and together with the recent work on value-dice indicates that the design space of imitation learning algorithms is far from being exhausted.

Non-Adversarial Imitation Learning and its Connections to Adversarial Methods

Viewing imitation learning as a distribution matching problem has become more popular in recent years (see <@Value-Dice@>(@Imitation Learning via Off-Policy Distribution Matching@) / <@I2L@>(@State-only Imitation with Transition Dynamics Mismatch@)). However, the authors in this paper argue that such methods are unstable due to their formulation as saddle-point problems which means they have weak convergence guarantees due to the assumption that the policy is slowly updated. In this paper, the authors reformulate <@Adversarial IRL@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@) as a non-adversarial problem allowing for much stronger convergence guarantees to be proved. In particular, the authors derive a lower-bound on the discrimination reward which allows for larger policy updates and then introduce a method to iteratively tighten this bound. They also build on prior work for value-dice and derive a soft actor-critic algorithm (ONAIL) that they evaluate on a variety of control tasks.


It has <@previously been argued@>(@The easy goal inference problem is still hard@) that in order to do better than the demonstrator, you need to have a model of how the demonstrator makes mistakes. In this work, that model is something like, “while running A* search, the demonstrator may fail to find all the states, or may find a suboptimal path before an optimal one”. This obviously isn’t exactly correct, but is hopefully moving in the right direction.

Note that in the domains that the paper evaluates on, the number of possible goals is fairly small (at most 20), presumably because of computational cost. However, even if we ignore computational cost, it’s not clear to me whether this would scale to a larger number of goals. Conceptually, this algorithm is looking for the most likely item out of the set of (optimal demonstrations and plausible suboptimal or failed demonstrations). When the number of goals is low, this set is relatively small, and the true answer will likely be the clear winner. However, once the number of goals is much larger, there may be multiple plausible answers. (This is similar to the fact that since neural nets encode many possible algorithms and there are multiple settings that optimize your objective, usually instead of getting the desired algorithm you get one that fails to transfer out of distribution.)

Online Bayesian Goal Inference for Boundedly-Rational Planning Agents

Tan Zhi-Xuan, Jordyn L. Mann, Tom Silver, Joshua B. Tenenbaum, Vikash K. Mansinghka

Typical approaches to learning from demonstrations rely on assuming that the demonstrator is either optimal or noisily optimal. However, this is a pretty bad description of actual human reasoning: it is more accurate to say we are _boundedly-rational planners_. In particular, it makes more sense to assume that our plans are computed from a noisy process. How might we capture this in an algorithm?

This paper models the demonstrator as using a bounded probabilistic [A* search](https://en.wikipedia.org/wiki/A*_search_algorithm) to find plans for achieving their goal. The planner is also randomized to account for the difficulty of planning: in particular, when choosing which state to “think about” next, it chooses randomly with higher probability for more promising states (as opposed to vanilla A* which always chooses the most promising state).

The search may fail to find a plan that achieves the goal, in which case the demonstrator follows the actions of the most promising plan found by A* search until no longer possible (either an action leads to a state A* search hadn’t considered, or it reaches the end of its partial plan). Thus, this algorithm can assign significant probability to plans that fail to reach the goal.

The experiments show that this feature allows their SIPS algorithm to infer goals even when the demonstrator fails to reach their goal. For example, if an agent needs to get two keys to unlock two doors to get a blue gem, but only manages to unlock the first door, the algorithm can still infer that the agent’s goal was to obtain the blue gem.

I really like that this paper is engaging with the difficulty of dealing with systematically imperfect demonstrators, and it shows that it can do much better than Bayesian IRL for the domains they consider.


This seems right to me -- we can also see this by looking at the various paradoxes found in the philosophy of ethics, which involve taking everyday moral intuitions and finding extreme situations in which they conflict, and it is unclear which moral intuition should “win”.

"Go west, young man!" - Preferences in (imperfect) maps

This post argues that by default, human preferences are strong views built upon poorly defined concepts, that may not have any coherent extrapolation in new situations. To put it another way, humans build mental maps of the world, and their preferences are defined on those maps, and so in new situations where the map no longer reflects the world accurately, it is unclear how preferences should be extended. As a result, anyone interested in preference learning should find some incoherent moral intuition that other people hold, and figure out how to make it coherent, as practice for the case we will face where our own values will be incoherent in the face of new situations.


I'm excited by this improvement, since now we can actually solve non-trivial CIRL games -- one of the games they solve has around 10 billion states. With this we can run experiments with real humans, which seems really important, and the paper does mention a very preliminary pilot study run with real humans.

An Efficient, Generalized Bellman Update For Cooperative Inverse Reinforcement Learning

Dhruv Malik, Malayandi Palaniappan, Jaime F. Fisac, Dylan Hadfield-Menell, Stuart Russell, Anca D. Dragan

Previously, Cooperative Inverse Reinforcement Learning (CIRL) games were solved by reducing them to a POMDP with an exponentially-sized action space, and then solving with POMDP algorithms that are exponential in the size of the action space, leading to a doubly-exponential algorithm. This paper leverages the fact that the human has perfect information to create a modified Bellman update that still computes the optimal policy, but no longer requires an exponential action space. The modified Bellman update works with the human's policy, and so we can now swap in more accurate models of the human, including eg. noisy rationality (whereas previously the human had to be exactly optimal). They show huge speedups in experiments, and discuss some interesting qualitative behavior that arises out of CIRL games -- for example, sometimes the human _waits_ instead of making progress on the task, because it is a good signal to the robot of what the human wants.


The intuition behind this method makes a lot of sense to me, but I wish the experiments were clearer in showing how the method compares to other methods. They show that, on Cartpole and Acrobat, they can match the results of behavioral cloning with 50,000 state-action pairs using 50,000 state observations and 200 environment interactions, but I don't know if behavioral cloning actually needed that many state-action pairs. Similarly, I'm not sure how much environment interaction would be needed if you inferred a reward function but not the dynamics, since they don't compare against such a method. I'm also unclear on how hard it is to assign transitions to latent actions -- they only test on MDPs with at most 3 actions, it's plausible to me that with more actions it becomes much harder to figure out which hidden action a state transition should correspond to.

Imitating Latent Policies from Observation

Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker, Charles L. Isbell

Typically in imitation learning, we assume that we have access to demonstrations that include the actions that the expert took. However, in many realistic settings we only have access to state observations (eg. driving videos). In this setting, we could still infer a reward function and then use reinforcement learning (RL) to imitate the behavior, but this would require a lot of interaction with the environment to learn the dynamics of the environment. Intuitively, even demonstrations with only states and no actions should give us a lot of information about the dynamics -- if we can extract this information, then we would need much less environment interaction during RL. (For example, if you watch a friend play a video game, you only see states, not actions; yet you can infer a lot about the game rules and gameplay.) The key idea is that each action probably causes similar effects on different states. So, they create a model with hidden action nodes z, and use the state observations to learn a policy P(z | s) and dynamics s' = g(s, z) (they assume deterministic dynamics). This is done end-to-end with neural nets, but essentially the net is looking at the sequence of states and figuring out how to assign actions z to each s (this is P(z | s)), such that we can learn a function g(s, z) that outputs the next observed state s'. Once this is trained, intuitively g(s, z) will already have captured most of the dynamics, and so now we only require a small number of environment actions to figure out how the true actions a correspond to the hidden actions z -- concretely, we train a model P(a | s, z). Then, in any state s, we first choose the most likely hidden action z* according to P(z | s), and then the most likely action a* according to P(a | s, z*).


In one of the experiments, the baseline of running IRL from scratch performed second best, beating out two other methods of meta-learning. I'd guess that this is because both MandRIL and standard IRL benefit from assuming the maxent IRL distribution over trajectories (which I believe is how the demonstrations were synthetically generated), whereas the other two meta learning baselines do not have any such assumption, and must learn this relationship.

Learning a Prior over Intent via Meta-Inverse Reinforcement Learning

Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, Chelsea Finn

For complex rewards, such as reward functions defined on pixels, standard IRL methods require a large number of demonstrations. However, many tasks are very related, and so we should be able to leverage demonstrations from one task to learn rewards for other tasks. This naturally suggests that we use meta learning. The authors adapt [MAML](https://arxiv.org/abs/1703.03400) to work with maximum entropy IRL (which requires differentiating through the MaxEnt IRL gradient). They evaluate their approach, called MandRIL, on a navigation task whose underlying structure is a gridworld, but the state is represented as an image so that the reward function is nonlinear and requires a convnet.