I liked this paper because it offers a novel solution to a common concern with the adversarial approach. Namely, GAN approaches often train discriminators that overpower the generator leading to mode collapse. In the RL setting, it seems natural to leave agent generated trajectories unlabeled since we don't have any sort of ground truth for whether or not agent trajectories are successful. For example, it might be possible to perform a task in a way that's different than is shown in the demonstrations. In this case, it makes sense to try and propagate feedback to the larger unlabeled agent trajectory data set indirectly. Presumably, this wasn't previously possible because positive-unlabeled learning has only recently been generalized to the deep learning setting. **After reading this paper, my broad takeaway is that semi-supervised methods are starting to reach the point where they have potential to further progress in imitation learning.**

Positive-Unlabeled Reward Learning

The problem with learning a reward model and training an agent on the (now fixed) model is that the agent can learn to exploit errors in the reward model. Adversarial imitation learning seeks to avoid this by training a discriminator reward model with the agent: the discriminator is trained via supervised learning to distinguish between expert trajectories and agent trajectories, while the agent tries to fool the discriminator. However, this effectively treats the agent trajectories as negative examples — even once the agent has mastered the task. What we would really like to do is to treat the agent trajectories as unlabeled data. This is an instance of _semi-supervised learning_, in which a classifier has access to a small set of labeled data and a much larger collection of unlabeled data. In general, the common approach is to propagate classification information learned using labels to the unlabeled dataset. The authors apply a recent algorithm for positive-unlabeled (PU) learning, and show that this approach can improve upon both GAIL and supervised reward learning.


This paper presents a interesting novel alternative to inverse reinforcement learning and does a good job of acknowledging potential objections. Deciding whether and how to store information about the past seems like an important problem that inverse reinforcement learning has to reckon with. My main concern with norm inference, which the paper mentions, is that optimizing over all possible propositions is in practice extremely slow. I don't anticipate that norm inference will be a performance-tractable strategy unless a lot of computation power is available.

**Rohin's opinion:** The idea of "norms" used here is very different from what I usually imagine, as in e.g. <@Following human norms@>. Usually, I think of norms as imposing a constraint upon policies rather than defining an optimal policy, (often) specifying what not to do rather than what to do, and being a property of groups of agents, rather than of a single agent. (See also [this comment](https://www.alignmentforum.org/posts/eBd6WvzhuqduCkYv3/following-human-norms#ujma2pWoH7ibhdog2).) The "norms" in this paper don't satisfy any of these properties: I would describe their norm inference as performing IRL with history-dependent reward functions, with a strong inductive bias towards "logical" reward functions (which comes from their use of Linear Temporal Logic). Note that some inductive bias is necessary, as without inductive bias history-dependent reward functions are far too expressive, and nothing could be reasonably learned. I think despite how it's written, the paper should be taken not as a denouncement of IRL-the-paradigm, but a proposal for better IRL algorithms that are quite different from the ones we currently have.

Norms, Rewards, and the Intentional Stance: Comparing Machine Learning Approaches to Ethical Training

Daniel Kasenberg, Thomas Arnold, Matthias Scheutz

This paper argues that _norm inference_ is a plausible alternative to inverse reinforcement learning (IRL) for teaching a system what people want. Existing IRL algorithms rely on the _Markov assumption_: that the next state of the world depends only on the previous state of the world and the action that the agent takes from that state, rather than on the agent’s entire history. In cases where information about the past matters, IRL will either fail to infer the right reward function, or will be forced to make challenging guesses about what past information to encode in each state. By contrast, _norm inference_ tries to infer what (potentially temporal) propositions encode the reward of the system, keeping around only past information that is relevant to evaluating potential propositions. The paper argues that norm inference results in more interpretable systems that generalize better than IRL -- systems that use norm inference can successfully model reward-driven agents, but systems that use IRL do poorly at learning temporal norms.


I think the problem of providing human guidance to agents is a core problem in alignment, and I am glad to see more discussion of that problem. I generally think that this type of broad overview is very valuable for communicating research to those who just want a broad overview of the field and don’t need to know the individual details of each paper. However, I would appreciate if there were more quantitative comparisons of the tradeoffs between different paradigms. The introduction mentions sample efficiency and the large effort required for human labelling, which made me hope for theoretical or empirical comparisons of the different methods with regards to sample efficiency and labelling effort. Since this was lacking, it also left me unclear on what motivated their suggested research directions. Personally, I would be much more excited to pursue a research direction if there were quantitative results showing particular failure modes or negative characteristics of current approaches that motivated that particular approach.

**Rohin's opinion:** This seems like a great survey paper and I like their proposed future directions, especially on learning from different kinds of human guidance, and on improving methods of teaching. While it does seem useful to have datasets of human guidance in order to compare algorithms, this prevents researchers from making improvements by figuring out new forms of guidance not present in the dataset. As a result, I'd be more excited about benchmarks that are evaluated by how much time it takes for Mechanical Turkers to train an agent to complete the task. Admittedly, it would be costlier in both time and money for researchers to do such an evaluation.

Leveraging Human Guidance for Deep Reinforcement Learning Tasks

Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H. Ballard, Peter Stone

A core problem in RL is the communication of our goals and prior knowledge to an agent. One common approach to this is imitation learning: the human provides example demonstrations of a task, and the agent learns to mimic them. However, there are some limitations to this approach, such as requiring the human to be capable of the task. This paper outlines five different modalities from which agents can learn: evaluations, preferences, hierarchical feedback, observations, and attention (for example, where humans are looking while solving a task). It then suggests future research directions.

For this summary, I will focus on the future research directions, but you can read the full paper to understand existing approaches.  The first issue is that datasets of human guidance are difficult to capture and depend on many specific factors of the individuals providing guidance. As a result, the paper suggests creating standard datasets to save effort and enable fair comparisons. The second direction is to better understand how humans should teach agents. The literature currently emphasizes progress in learning methods, but improved teaching methods may be just as valuable when learning from human guidance. The last is unifying learning across different input modalities; ideally an agent would be able to learn from many different types of human guidance over different phases of its learning.


It is good to see research on mainstream NLProc/ML tasks that includes discussions on challenges, failure modes and relevance to the broader motivating goals of AI research.

The work opens up interesting avenues within OpenAI's alignment agenda, for example learning a diversity of preferences (A OR B), or a hierarchy of preferences (A AND B) sequentially without catastrophic forgetting.

In order to scale, we would want to generate automated labelers through semi-supervised reinforcement learning, to derive the most gains from every piece of human input. The robustness of this needs further empirical and conceptual investigation before we can be confident that such a system can work to form a hierarchy of learners, e.g. in amplification.

**Rohin's opinion:** One thing I particularly like here is that the evaluation is done by humans. This seems significantly more robust as an evaluation metric than any automated system we could come up with, and I hope that more people use human evaluation in the future.

Paper: Fine-Tuning Language Models from Human Preferences

Fine-Tuning GPT-2 from Human Preferences

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving

This blog post and its [associated paper](https://arxiv.org/abs/1909.08593) describes the results of several text generation/continuation experiments, where human feedback on initial/older samples was used in the form of a reinforcement learning reward signal to finetune the base 774-million parameter <@GPT-2 language model@>(@Better Language Models and Their Implications@). The key motivation here was to understand whether interactions with humans can help algorithms better learn and adapt to human preferences in natural language generation tasks.

They report mixed results. For the tasks of continuing text with positive sentiment or physically descriptive language, they report improved performance above the baseline (as assessed by external examiners) after fine-tuning on only 5,000 human judgments of samples generated from the base model. The summarization task required 60,000 samples of _online_ human feedback to perform similarly to a simple baseline, lead-3 - which returns the first three sentences as the summary - as assessed by humans.

Some of the lessons learned while performing this research include 1) the need for better, less ambiguous tasks and labelling protocols for sourcing higher quality annotations, and 2) a reminder that "bugs can optimize for bad behaviour", as a sign error propagated through the training process to generate "not gibberish but maximally bad output". The work concludes on the note that it is a step towards scalable AI alignment methods such as debate and amplification.


Overall, I think this is potentially a straightforward and clever trick for giving your imitation learner more informative data to learn off of. I have two main questions. First off, I'd have loved to see D-REX compared directly to T-REX, to get a sense of how much you lose from this approximate ranking strategy rather than a more ground truth one. And, secondly, I'd have appreciated a bit more justification of their assumption that noisier actions will consistently lead to a worse policy, in ways that capture reward information. This doesn't seem obviously untrue to me, I'd just love some more intuition on why we can get additional information about underlying reward just by adding noise.

Ranking-Based Reward Extrapolation without Rankings

Daniel S. Brown, Wonjoon Goo, Scott Niekum

A while back, these authors released the <@T-REX paper@>(@Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations@), where they showed that providing ranked sets of trajectories, rather than one single optimal trajectory, lets you learn a more accurate reward that can outperform the demonstrator. This ability to outperform the demonstrator is rooted in the ability to extrapolate predicted reward outside of demonstrated points, and that ability to extrapolate comes from the fact that ranked trajectories provide more information about relative reward values. This paper is a fairly straightforward extension of that one, and asks: can we get similar benefits without requiring humans to actually rank trajectories? The authors argue that they can replicate T-REX's ability to outperform the demonstrator by simply learning a behaviorally cloned policy off of a single (potentially sub-optimal) demonstrator, and making that policy gradually worse by adding more noise to it. This model is called D-REX, for Disturbance-based Reward EXtrapolation. They then make an assumption that more noise in the policy corresponds to less reward, and use that as a ranking scheme to throw into the existing T-REX algorithm.


Although this only got published recently, it’s work I did over a year ago. I’m no longer very optimistic about <@ambitious value learning@>(@What is ambitious value learning?@), and so I’m less excited about its impact on AI alignment now. In particular, it seems unlikely to me that we will need to infer all human values perfectly, without any edge cases or uncertainties, which we then optimize as far as possible. I would instead want to build AI systems that start with an adequate understanding of human preferences, and then learn more over time, in conjunction with optimizing for the preferences they know about. However, this paper is more along the former line of work, at least for long-term AI alignment.

I do think that this is a contribution to the field of inverse reinforcement learning -- it shows that by using an appropriate inductive bias, you can become more robust to (cognitive) biases in your dataset. It’s not clear how far this will generalize, since it was tested on simulated biases on simple environments, but I’d expect it to have at least a small effect. In practice though, I expect that you’d get better results by providing more information, as in <@T-REX@>(@Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations@).

On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference

Learning biases and rewards simultaneously

Rohin Shah, Noah Gundotra, Pieter Abbeel, Anca D. Dragan

Typically, inverse reinforcement learning assumes that the demonstrator is optimal, or that any mistakes they make are caused by random noise. Without a model of _how_ the demonstrator makes mistakes, we should expect that <@IRL would not be able to outperform the demonstrator@>(@The easy goal inference problem is still hard@). So, a natural question arises: can we learn the systematic mistakes that the demonstrator makes from data? While there is an <@impossibility result@>(@Humans can be assigned any values whatsoever…@) here, we might hope that it is only a problem in theory, not in practice.

In this paper, my coauthors and I propose that we learn the cognitive biases of the demonstrator, by learning their planning algorithm. The hope is that the cognitive biases are encoded in the learned planning algorithm. We can then perform bias-aware IRL by finding the reward function that when passed into the planning algorithm results in the observed policy. We have two algorithms which do this, one which assumes that we know the ground-truth rewards for some tasks, and one which tries to keep the learned planner “close to” the optimal planner. In a simple environment with simulated human biases, the algorithms perform better than the standard IRL assumptions of perfect optimality or Boltzmann rationality -- but they lose a lot of performance by using an imperfect differentiable planner to learn the planning algorithm.


I am a little cautious to make a strong statement about the importance of this paper, since I don't have as much domain knowledge in cognitive science as I do in machine learning, but overall this "treat your theoretical model like a generative model and sample from it" idea seems like an elegant and plausibly more broadly extensible way of incorporating theoretical priors alongside real data.

Cognitive Model Priors for Predicting Human Decisions

David D. Bourgin*, Joshua C. Peterson*, Daniel Reichman, Thomas L. Griffiths, Stuart J. Russell

Human decision making is notoriously difficult to predict, being a combination of expected value calculation and likely-not-fully-enumerated cognitive biases. Normally we could predict well using a neural net with a ton of data, but data about human decision making is expensive and scarce. This paper proposes that we pretrain a neural net on lots of data simulated from theoretical models of human decision making and then finetune on the small real dataset. In effect, we are using the theoretical model as a kind of prior, that provides the neural net with a strong inductive bias. The method achieves better performance than existing theoretical or empirical methods, without requiring feature engineering, both on existing datasets and a new, larger dataset collected via Mechanical Turk.


I appreciate papers like these that connect peoples intuitions between different areas (like imitation learning and distributional difference measures). It does seem like this would even more strongly lead to lack of ability to outperform the demonstrator, but that's honestly more a critique of imitation learning more generally than this paper in particular.

Imitation Learning as f-Divergence Minimization

Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, Siddhartha Srinivasa

This paper frames imitation learning through the lens of matching your model's distribution over trajectories (or conditional actions) to the distribution of an expert policy. This framing of distribution comparison naturally leads to the discussion of f-divergences, a broad set of measures including KL and Jenson-Shannon Divergences. The paper argues that existing imitation learning methods have implicitly chosen divergence measures that incentivize "mode covering" (making sure to have support anywhere the expert does) vs mode collapsing (making sure to only have support where the expert does), and that the latter is more appropriate for safety reasons, since the average between two modes of an expert policy may not itself be a safe policy. They demonstrate this by using a variational approximation of the reverse-KL distance as the divergence underlying their imitation learner.


Workshop on Self-Supervised Learning at ICML 2019

This idea definitely gets points for being a clear and easy-to-implement heuristic, though I worry it may have trouble with videos that  don't match its goal-directed assumption.

Perceptual Values from Observation

This paper proposes a technique for learning from raw expert-trajectory observations by assuming that the last state in the trajectory is the state where the goal was achieved, and that other states have value in proportion to how close they are to a terminal state in demonstration trajectories. They use this as a grounding to train models predicting value and action-value, and then use these estimated values to determine actions.


Seems like potentially a straightforward and clever modification to a typical reward learning structure, but a bit unclear how much of the performance relative to GAIL and BCO derives from T-REX's access to suboptimal demonstrations and subtrajectories giving it more effective training data. It does intuitively seem that adding examples of what poor performance looks like, rather than just optimal performance, would add useful informative signal to training. On a personal level, I'm curious if one implication of an approach like this is that it could allow a single set of demonstration trajectories to be used in reward learning of multiple distinct rewards, based on different rankings being assigned to the same trajectory based on the reward the ranker wants to see demonstrated.

**Rohin's opinion:** It's pretty interesting that the [Deep RL from Human Preferences](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/) approach works even with off-policy trajectories. It seems like looking at the _difference_ between good and bad trajectories gives you more information about the true reward that generalizes better. We saw similar things in our work on <@Active Inverse Reward Design@>.

Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations

Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, Scott Niekum

This paper claims to demonstrate a technique by which an agent learning from a demonstrator's actions can learn to outperform that demonstrator on their true reward, rather than, in the way of imitation learning or behavioral cloning, just mimicking the demonstrator under the assumption that the demonstrator's performance is optimal (or at least near-optimal). The key structural innovation of the paper is to learn using pairs of ranked trajectories and learn a neural network-based reward function based on correctly predicting which will be higher. This allows the model to predict what actions will lead to higher and lower reward, and to extrapolate that relationship beyond the best demonstration. When an agent is then trained using this reward model as it's ground truth reward, it's shown to be capable of outperforming the demonstrator on multiple tested environments, including Atari. An important distinction compared to some prior work is the fact that these rankings are collected in an off-policy manner, distinguishing it from [Deep RL from Human Preferences](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/) where rankings are requested on trajectories generated as an agent learns.


I really like the focus on figuring out how humans actually provide feedback in practice; it makes a lot of sense that we provide reinforcement signals that reflect the advantage function rather than the reward function. That said, I wish the evaluation had more complex tasks, and had involved human trainers who were not authors of the paper -- it might have taken an hour or two of human time instead of 10-15 minutes, but would have been a lot more compelling.

Before continuing, I recommend reading about Simulated Policy Learning in Video Models below. As in that case, I think that you get sample efficiency here by getting a lot of "supervision information" from the pixels used to train the VAE, though in this case it's by learning useful features rather than using the world model to simulate trajectories. (Importantly, in this setting we care about sample efficiency _with respect to human feedback_ as opposed to environment interaction.) I think the techniques used there could help with scaling to more complex tasks. In particular, it would be interesting to see a variant of deep COACH that alternated between training the VAE with the learned control policy, and training the learned control policy with the new VAE features. One issue would be that as you retrain the VAE, you would invalidate your previous control policy, but you could probably get around that (e.g. by also training the control policy to imitate itself while the VAE is being trained).

Deep Reinforcement Learning from Policy-Dependent Human Feedback

Dilip Arumugam, Jun Ki Lee, Sophie Saskin, Michael L. Littman

One obvious approach to human-in-the-loop reinforcement learning is to have humans provide an external reward signal that the policy optimizes. [Previous work](https://arxiv.org/abs/1701.06049) noted that humans tend to _correct_ existing behavior, rather than providing an "objective" measurement of how good the behavior is (which is what a reward function is). They proposed Convergent Actor-Critic by Humans (COACH), where instead of using human feedback as a reward signal, they use it as the _advantage function_. This means that human feedback is modeled as specifying how good an action is relative to the "average" action that the agent would have chosen from that state. (It's an average because the policy is stochastic.) Thus, as the policy gets better, it will no longer get positive feedback on behaviors that it has successfully learned to do, which matches how humans give reinforcement signals.

This work takes COACH and extends it to the deep RL setting, evaluating it on Minecraft. While the original COACH had an eligibility trace that helps "smooth out" human feedback over time, deep COACH requires an eligibility replay buffer. For sample efficiency, they first train an autoencoder to learn a good representation of the space (presumably using experience collected with a random policy), and feed these representations into the control policy. They reward entropy so that the policy doesn't commit to a particular behavior, making it responsive to feedback, but select actions by always picking the action with maximal probability (rather than sampling from the distribution) in order to have interpretable, consistent behavior for the human trainers to provide feedback on. They evaluate on simple navigation tasks in the complex 3D environment of Minecraft, including a task where the agent must patrol the perimeter of a room, which cannot be captured by a state-based reward function.


I found this theorem quite unintuitive when I first encountered it, despite it being two lines long, which is something of a testament to how annoying and tricky misspecification can be. One way I interpret the empirical result is that the wider the probability distributions of our assumptions, the more robust they are to misspecification. A literal robot assumes that the human can take any near-optimal trajectory, whereas a pedagogic robot assumes that the human takes very particular near-optimal trajectories that best communicate the reward. So, the literal robot places probability mass over a larger space of trajectories given a particular reward, and does not update as strongly on any particular observed trajectory compared to the pedagogic robot, making it more robust.

Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning

In [Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137), we assume a two-player game with a human and a robot where the robot doesn't know the reward R, but both players are trying to maximize the reward. Since one of the players is a human, we cannot simply compute the optimal strategy and deploy it -- we are always making some assumption about the human, that may be misspecified. A common assumption is that the human is playing optimally for the single-player version of the game, also known as a literal human. The robot then takes the best response actions given that assumption. Another assumption is to have a _pedagogic_ human, who acts as though the robot is interpreting her literally. The robot that takes the best response actions with this assumption is called a pedagogic or pragmatic robot.

However, any assumption we make about the human is going to be misspecified. This paper looks at how we can be robust to misspecification, in particular if the human could be literal or pedagogic. The main result is that the literal robot is more robust to misspecification. The way I think about this is that the literal robot is designed to work with a literal human, and a pedagogic human is "designed" to work with the literal robot, so unsurprisingly the literal robot works well with both of them. On the other hand, the pedagogic robot is designed to work with the pedagogic human, but has no relationship with the literal robot, and so should not be expected to work well. It turns out we can turn this argument into a very simple proof: (literal robot, pedagogic human) outperforms (literal robot, literal human) since the pedagogic human is designed to work well with the literal robot, and (literal robot, literal human) outperforms (pedagogic robot, literal human) since the literal robot is designed to work with the literal human.

They then check that the theory holds in practice. They find that the literal robot is better than the pedagogic robot _even when humans are trying to be pedagogic_, a stronger result than the theory predicted. The authors hypothesize that even when trying to be pedagogic, humans are more accurately modeled as a mixture of literal and pedagogic humans, and the extra robustness of the literal robot means that it is the better choice.


Unsupervised Visuomotor Control through Distributional Planning Networks

Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, Chelsea Finn


IROS 2018 Workshop on Machine Learning in Robot Motion Planning

Risk-Aware Active Inverse Reinforcement Learning

Daniel S. Brown, Yuchen Cui, Scott Niekum


Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow

Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, Sergey Levine

Adversarial learning techniques require a delicate balance between the generator and the discriminator. If the discriminator is too weak, it cannot tell the difference between generated samples and true samples, and it cannot provide a learning signal for the generator. If the discriminator is too strong, small changes to the generator are not going to fool the discriminator, and so again the gradient is uninformative. This paper proposes to control the power of the discriminator using an _information bottleneck_.

Instead of providing data points directly to the discriminator, the data points are first encoded into a new representation, and the discriminator must work with the new representation. The representation is learned to be helpful for the discriminator under the constraint of an upper bound on the mutual information between the representation and the original data points. The choice of upper bound determines how much information the discriminator is allowed to access, which in turn determines how powerful the discriminator is.

They apply this idea to imitation learning (GAIL), inverse reinforcement learning (AIRL), and image generation (GANs), and find that it improves results.


This paper explores an important idea: correcting poorly-specified instructions using human-in-the-loop feedback. The second task in particular is a nice toy example of iterative preference clarification. I'm not sure whether their meta-learning approach is directly relevant to safety, particularly because each correction is only "in scope" for a single episode, and also only occurs after a bad attempt has finished. However, the broad idea of correction-based learning seems promising.

Guiding Policies with Language via Meta-Learning

The authors train an agent to perform tasks specified in natural language, with a "correction" after each attempt (also in natural language). They formulate this as a meta-learning problem: for each instruction, several attempt-correction cycles are allowed. Each attempt takes into account previous attempts to achieve the same instruction by passing each previous trajectory and its corresponding correction through a CNN, then using the mean of all outputs as an input to a policy module.

In their experiments, all instructions and corrections are generated automatically, and test-time performance is evaluated as a function of how many corrections are allowed. In one experiment, the tasks is to navigate rooms to reach a goal, where the correction is the next subgoal required. Given 4 corrections, their agent outperforms a baseline which was given all 5 subgoals at the beginning of the task. In another experiment, the task is to move a block to an ambiguously-specified location, and the corrections narrow down the target area; their trained agent scores 0.9, as opposed to 0.96 for an agent given the exact target location.


BabyAI: First Steps Towards Grounded Language Learning With a Human In the Loop

Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, Yoshua Bengio


Learning under Misspecified Objective Spaces

Andreea Bobu, Andrea Bajcsy, Jaime F. Fisac, Anca D. Dragan

What can you do if the true objective that you are trying to infer is outside of your hypothesis space? The key insight of this paper is that in this scenario, the human feedback that you get will likely not make sense for _any_ reward function in your hypothesis space, which allows you to notice when this is happening. This is operationalized using a Bayesian model in which a latent binary variable represents whether or not the true objective is in the hypothesis space. If it is, then the rationality constant β will be large (i.e. the human appears to be rational), whereas if it is not, then β will be small (i.e. the human appears to be noisy). The authors evaluate with real humans correcting the trajectory of a robotic arm.


Learning Task Specifications from Demonstrations

Marcell Vazquez-Chanlatte, Susmit Jha, Ashish Tiwari, Mark K. Ho, Sanjit A. Seshia


Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Ruiqi Zhong, Kristy Lee, Zheng Zhang, Dan Klein

<@Large language models@>(@Language Models are Few-Shot Learners@) can be prompted to perform classification tasks. However, you may not want to simply phrase the prompt as a question like “Does the following tweet have positive or negative sentiment?” because in the training set such questions may have been followed by something other than an answer (for example, an elaboration of the question, or a denial that the question is important), and the model may end up choosing one of these alternatives as the most likely completion.

The natural solution is to collect a question-answering dataset and finetune on it. The core idea of this paper is that we can convert existing NLP classification datasets into a question-answering format, which we can then finetune on. For example, given a dataset for movie review classification (where the goal is to predict whether a review is positive or negative), we produce questions like “Is the review positive?” or “Does the user find this movie bad?” The entire classification dataset can then be turned into question-answer pairs to train on.

The authors do this for several datasets, producing 441 question types in total. They then finetune the 0.77B parameter T5 model on a training set of questions and evaluate it on questions that come from datasets not seen during training. Among other things, they find:

1. Their model does better than [UnifiedQA](https://arxiv.org/abs/2005.00700), which was also trained for question answering using a similar idea.
2. Pretraining is very important: performance crashes if you “finetune” on top of a randomly initialized model. This suggests that the model already “knows” the relevant information, and finetuning ensures that it uses this knowledge appropriately.
3. If you ensemble multiple questions that get at the same underlying classification task, you can do better than any of the questions individually.
4. It is possible to overfit: if you train too long, performance does decrease.


I’m particularly interested in cases where, after crossing a certain size or capability threshold, models become capable of transferring knowledge between domains, for example:

1. Intuitively, the goal of this paper is to get the model to follow the general rule “understand the semantic content of the instruction and then follow it”. Models only become able to successfully generalize this rule from training tasks to heldout tasks somewhere in the 8B - 68B range.
2. In the previous paper, the 0.77B model was able to successfully generalize the rule “answer questions well” from training tasks to heldout tasks. Presumably some smaller model would not have been able to do this.
3. <@Last week’s highlight@>(@Program Synthesis with Large Language Models@) showed that the 137B model was able to transfer knowledge from _code execution_ to _program synthesis_, while the 8B model was unable to do this.

Notably, the only major difference in these cases is the size of the model: the training method and dataset are the same. This seems like it is telling us something about how neural net generalization works and/or how it arises. I don’t have anything particularly interesting to say about it, but it seems like a phenomenon worth investigating in more detail.

Finetuned Language Models Are Zero-Shot Learners

Jason Wei*, Maarten Bosma*, Vincent Y. Zhao*, Kelvin Guu*, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, Quoc V. Le

This paper applies the approach from the previous paper on a much larger 137B parameter model to produce a model that _follows instructions_ (rather than just _answering questions_). Since they are focused on instruction following, they don’t limit themselves to classification tasks: they also want to have generative tasks, and so include e.g. summarization datasets. They also generate such tasks automatically by “inverting” the classification task: given the label y, the goal is to generate the input x. For example, for the movie review classification dataset, they might provide the instruction “Write a negative movie review”, and then provide one of the movie reviews classified as negative as an example of what the model should write in that situation.

A natural approach to classification with a language model is to ask a question like “Is this movie review positive?” and then checking the probability assigned to “Yes” and “No” and returning whichever one was higher. The authors note that this can be vulnerable to what we might call “probability splitting” (analogously to [vote splitting](https://en.wikipedia.org/wiki/Vote_splitting)). Even if the correct answer is “Yes”, the model might split probability across “Yes”, “Yup”, “Definitely”, “Absolutely”, etc such that “No” ends up having higher probability than “Yes”. To solve this problem, in classification questions they add a postscript specifying what the options are. During finetuning, the model should quickly learn that the next word is always chosen from one of these options, and so will stop assigning probability to other words, preventing probability splitting.

They find that the finetuned model does much better on held-out tasks than the original model (both evaluated zero-shot). The finetuned model also beats zero-shot GPT-3 on 19 of 25 tasks, and few-shot GPT-3 on 10 of 25 tasks. The finetuned model is always used zero-shot; unfortunately they don’t report results when using the finetuned model in a few-shot setting.

They also study the impact of instruction tuning over various model sizes. At every model size, instruction tuning helps significantly on the tasks that were seen during finetuning, as you would expect. However, when considering tasks that were _not_ seen during finetuning, instruction tuning actually _hurts_ performance up to models with 8B parameters, and only helps for the 68B and 137B models (where it raises performance by about 15 percentage points on average across heldout tasks).


I like these sorts of empirical benchmark papers; it feels so much easier to learn what works from such papers (relative to reading the papers in which the algorithms were introduced). This paper in particular was also especially clear and easy to read; my summary of the results is in large part just a restatement of Section 5 of the paper.

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martín-Martín

As you might expect from the title, this paper tests imitation learning and offline RL algorithms on a benchmark of robotic manipulation tasks in which the agent must learn to perform the task from human demonstrations. Most of the experiments were done in simulation, but they did do a final training run on a real robot using hyperparameters chosen in simulation, to demonstrate that their preferred algorithms could work in such a setting as well. Some findings I found particularly interesting:

1. It is important to have models with memory: behavioral cloning (BC) does significantly better on human demonstrations when it is training an RNN model (which has memory), especially on longer-horizon tasks. This is presumably because the humans providing the demonstrations chose actions based not only on the current state but also what had happened in the past, i.e. they were non-Markovian. To test this hypothesis, we could look at machine-generated demonstrations, where you get demonstrations from an expert agent trained using RL, which I _think_ are guaranteed to be Markovian by construction. Unfortunately, we can only get reasonable RL experts on the shorter-horizon tasks where the effect is less pronounced; in these cases BC-RNN still outperforms BC without the RNN, weakly suggesting that it isn’t just about Markovian vs. non-Markovian data.

2. Offline RL algorithms work quite well on the machine-generated data, but don’t work very well on human demonstrations. It isn’t particularly clear why this is the case.

3. In addition, offline RL struggles when used on datasets where the demonstrations are of mixed quality; in comparison BC-RNN does quite well.

4. Policy selection is a challenging problem: in these settings, the training objective (e.g. predict the expert actions) is usually not the thing you actually care about (e.g. did you successfully pick up the cup). Ideally, you would evaluate many model checkpoints throughout the training process on the metric you actually care about and then choose the one that performs best. If you instead select the model checkpoint that achieved the lowest validation loss, performance on the correct metric can decrease by 50-100%; if you always use the last checkpoint (i.e. at the end of training), performance can decrease by 10-30%. This demonstrates that it is important to choose the right model during training – but there’s no clear way to do this, as often the evaluation of a policy is non-trivial.

5. The observation space (e.g. pixel observations vs. observations of joint angles and forces) and hyperparameters (e.g. learning rate) both matter quite a lot. For example, adding information about end effectors can drop performance by 49-88% (presumably due to overfitting).

6. For complex tasks, more data provides significant improvements.


I really like this sort of empirical analysis: it seems incredibly useful for understanding what does and doesn’t work.

Note that I haven’t looked deeply into their results and analysis, and am instead reporting what they said on faith. (With most papers I at least look through the experiments to see if the graphs tell a different story or if there were some unusual choices not mentioned in the introduction, but that was a bit of a daunting task for this paper, given how many experiments and graphs it had.)

What Matters for Adversarial Imitation Learning?

Manu Orsini*, Anton Raichuk*, Léonard Hussenot*, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, Marcin Andrychowicz

This paper takes adversarial imitation learning algorithms (think <@GAIL@>(@Generative Adversarial Imitation Learning@) and <@AIRL@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@)) and tests the effect of various hyperparameters, including the loss function, the discriminator regularization scheme, the discriminator learning rate, etc. They first run a large, shallow hyperparameter sweep to identify reasonable ranges of values for the various hyperparameters, and then run a larger hyperparameter sweep within these ranges to get a lot of data that they can then analyze. All the experiments are done on two continuous control benchmarks: the MuJoCo environments in OpenAI Gym and manipulation environments from Adroit.

Obviously they have a lot of findings, and if you spend time working with adversarial imitation learning algorithms, I’d recommend reading through the full paper, but the ones they highlight are:
1. Even though some papers have proposed regularization techniques that are specific to imitation learning, standard supervised learning techniques like dropout work just as well.
2. There are significant differences in the results when using synthetic demonstrations vs. human demonstrations. (A synthetic demonstration is one provided by an RL agent trained on the true reward.) For example, the optimal choice of loss function is different for synthetic demos vs. human demos. Qualitatively, human demonstrations are not Markovian and are often multimodal (especially when the human waits and thinks for some time: in this case one mode is “noop” and the other mode is the desired action).


Prompting: Better Ways of Using Language Models for NLP Tasks

Since the publication of <@GPT-3@>(@Language Models are Few-Shot Learners@), many papers have been written about how to select the best prompt for large language models to have them solve particular tasks of interest. This post gives an overview of this literature. The papers can be roughly divided into two approaches: first, we have discrete prompts, where you search for a sequence of words that forms an effective prompt; these are “discrete” since words are discrete. Second, we have soft prompts, where you search within the space of embeddings of words for an embedding that forms an effective prompt; since embeddings are vectors of real numbers they are continuous (or “soft”) and can be optimized through gradient descent (unlike discrete prompts).


I’m a big fan of thinking about how normative information is transferred from us to our agents -- I frequently ask myself questions like “how does the agent get the information to know X”, where X is something normative like “wireheading is bad”.

In the case of large neural nets, I generally like assistance games as an analysis tool for thinking about how such AI systems should behave at deployment time, for the reasons outlined in the podcast. It’s less clear what the framework has to say about what should be done about training time, when we don’t expect to have a human in the loop (or we expect that to be a relative minority of our training data).

To be clear, this should be taken as an endorsement of thinking about assistance games: my point is just that (according to me) it is best to think of them in relation to deployment, not training. A framework doesn’t have to apply to everything in order to be well worth thinking about.

AXRP Episode 8 - Assistance Games

As with most other podcasts, I will primarily link you to my past summaries of the papers discussed in the episode. In this case they were all discussed in the special issue [AN #69](https://mailchi.mp/59ddebcb3b9a/an-69-stuart-russells-new-book-on-why-we-need-to-replace-the-standard-model-of-ai) on Human Compatible and the various papers relevant to it. Some points that I haven’t previously summarized:

1. The interviewee thinks of assistance games as an _analytical tool_ that allows us to study the process by which humans convey normative information (such as goals) to an AI system. Normally, the math we write down takes the objective as given, whereas an assistance game uses math that assumes there is a human with a communication channel to the AI system. We can thus talk mathematically about how the human communicates with the AI system.

2. This then allows us to talk about issues that might arise. For example, <@assistive bandits@>(@The Assistive Multi-Armed Bandit@) considers the fact that humans might be learning over time (rather than starting out as optimal).

3. By using assistance games, we build the expectation that our AI systems will have ongoing oversight and adaptation directly into the math, which seems significantly better than doing this on an ad hoc basis (as is currently the case). This should help both near-term and long-term systems.

4. One core question is how we can specify a communication mechanism that is robust to misspecification. We can operationalize this as: if your AI system is missing some relevant features about the world, how bad could outcomes be? For example, it seems like demonstrating what you want (i.e. imitation learning) is more robust than directly saying what the goal is.

5. One piece of advice for deep learning practitioners is to think about where the normative information for your AI system is coming from, and whether it is sufficient to convey what you want. For example, large language models have trillions of parameters, but only hundreds of decisions inform the choice of what data to train them on -- is that enough? The language we train on has lots of normative content -- does that compensate?

6. Dylan says: “if you’re interested in doing this type of work and you thought this conversation was fun and you’d like to have more conversations like it with me, I’ll invite you to [apply to MIT’s EECS PhD program](https://gradapply.mit.edu/eecs/apply/login/?next=/eecs/) next year and mention me in your application.”


IEEE Robotics and Automation Letters 2021

This seems like an interesting point to be thinking about. The part I'm most interested in is whether it is true that small changes in the neural net parameters must lead to small changes in the resulting trajectory. It seems plausible to me that this is true for small neural nets but ends up becoming less true as neural nets become larger and data becomes more diverse. In our running example, if the neural net was implementing some decision process that considered both left and right as options, and then "chose'' to go right, then it seems plausible that a small change to the weights could cause it to choose to go left instead, allowing gradient descent to switch across trajectory homotopy classes with a small nudge to model parameters.

Transfer Reinforcement Learning across Homotopy Classes

Zhangjie Cao*, Minae Kwon*, Dorsa Sadigh

Suppose a robot walks past a person and it chooses to pass them on the right side. Imagine that we want to make the robot instead pass on the left side, and our tool for doing this was to keep nudging the robot's trajectory until it did what we wanted. In this case, we're screwed: there is no way to ``nudge'' the trajectory from passing on the right to passing on the left without going through a trajectory that crashes straight into the person.

The core claim of this paper is that the same sort of situation applies to finetuning for RL agents. Suppose we train an agent for one task where there is lots of data, and then we want to finetune it to another task. Let's assume that the new task is in a different _homotopy class_ than the original task, which roughly means that you can't nudge the trajectory from the old task to the new task without going through a very low reward trajectory (in our example, crashing into the person). However, finetuning uses gradient descent, which nudges model parameters; and intuitively, a nudge to model parameters would likely correspond to a nudge to the trajectory as well. Since the new task is in a different homotopy class, this means that gradient descent would have to go through a region in which the trajectory gets very low reward. This is not the sort of thing gradient descent is likely to do, and so we should expect finetuning to fail in this case.

The authors recommend that in such cases, we first train in a simulated version of the task in which the large negative reward is removed, allowing the finetuning to ``cross the gap''. Once this has been done, we can then reintroduce the large negative reward through a curriculum -- either by gradually increasing the magnitude of the negative reward, or by gradually increasing the number of states that have large negative reward. They run several robotics experiments demonstrating that this approach leads to significantly faster finetuning than other methods.


I really like the fact that we’re studying a fuzzy task directly, and using human evaluation to determine how well the model performs. (Though note that this is not the first benchmark to do so.)

TuringAdvice: A Generative and Dynamic Evaluation of Language Use

Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi, Yejin Choi

There are two main ways in which current NLP models are evaluated: quality of generations (how sensible the generated language looks), and correctness (given some crisp question or task, does the model output the right answer). However, we often care about using models for tasks in which there is no literally correct answer. This paper introduces an evaluation method for this setting: TuringAdvice. Models are presented with a situation in which a human is asking for advice, and the model must provide a helpful response. To score models, the resulting responses are compared against good human responses. The model’s response is successful if its advice is at least as helpful to the advice-seeker as human-written advice.

The authors collect a dataset of situations from Reddit, and for the human-written advice they take the most upvoted top-level comment on the post. A finetuned T5 model achieves a score of 14%, while prompted GPT-3 achieves a score of 4%. In contrast, taking the _secondmost_ upvoted top-level comment would give a score of 41%, and a model that gave advice about as good as the typical best advice from a human would get 50%. The paper also presents several qualitative failures in which the models seem to have significant misunderstandings of the situation (though I can’t tell how cherrypicked these are).


Imitative Generalisation (AKA 'Learning the Prior')

This post explains a simplified version of the scheme in [Learning the prior](https://www.alignmentforum.org/posts/SL9mKhgdmDKXmxwE4/learning-the-prior) ([AN #109](https://mailchi.mp/ee62c1c9e331/an-109teaching-neural-nets-to-generalize-the-way-humans-would)) with an image classification example.

A key issue for distributional shift is that neural nets assign significant “probability” to “crazy” hypotheses. Imagine that we want to train a neural net to classify dog breeds, and in our training dataset D all huskies are on snow, but on the test dataset D’ they may also be on grass. Then a neural net is perfectly happy with the hypothesis “if most of the bottom half of the image is white, then it is a husky”, whereas humans would see that as crazy and would much prefer the hypothesis “a husky is a large, fluffy, wolf-like dog”, _even if they don’t know what a husky looks like_.

Thus, we might say that the human “prior” over hypotheses is much better than the corresponding neural net “prior”. So, let’s optimize our model using the human prior instead. In particular, we search for a hypothesis such that 1) humans think the hypothesis is likely (high human prior), and 2) the hypothesis leads humans to make good predictions on the training dataset D. Once we have this hypothesis, we have humans make predictions using that hypothesis on the test distribution D’, and train a model to imitate these predictions. We can then use this model to predict for the rest of D’. Notably, this model is now being used in an iid way (i.e. no distribution shift).

A key challenge here is how to represent the hypotheses that we’re optimizing over -- they need to be amenable to ML-based optimization, but they also need to be interpretable to humans. A text-based hypothesis would likely be too cumbersome; it is possible that neural-net-based hypotheses could work if augmented by interpretability tools that let the humans understand the “knowledge” in the neural net (this is similar in spirit to <@Microscope AI@>(@Chris Olah’s views on AGI safety@)).

For more details on the setup, see the full post, or my [previous summary](https://mailchi.mp/ee62c1c9e331/an-109teaching-neural-nets-to-generalize-the-way-humans-would).


DERAIL: Diagnostic Environments for Reward And Imitation Learning

Pedro Freire, Adam Gleave, Sam Toyer, Stuart Russell

Most deep RL algorithms are quite sensitive to implementation and hyperparameters, and this applies to imitation learning as well. So, it would be useful to have some simple sanity checks that an algorithm works well, before throwing algorithms at challenging benchmarks trying to beat the state of the art. This paper presents a suite of simple environments that each aim to test a single aspect of an algorithm, in a similar spirit to unit testing.

For example, RiskyPath is a very simple four-state stochastic MDP, in which the agent can take a long, safe path to the reward, or a short, risky path. While it is always better in expectation to take the safer path, properly reasoning about stochasticity can be subtle, and some published algorithms, like <@Maximum Entropy IRL@>(@Maximum Entropy Inverse Reinforcement Learning@), always choose the risky path (this can be fixed by using <@causal entropy@>(@Modeling Interaction via the Principle of Maximum Causal Entropy@)). By isolating the issue, RiskyPath can be used as a quick test to detect this behavior in new algorithms.

The paper also presents a case study in tuning an implementation of [Deep RL from Human Preferences](https://deepmind.com/blog/learning-through-human-feedback/), in which a sparse exploration task suggested that the comparison queries were insufficiently diverse to guarantee stability.


Bayesian Inverse Reinforcement Learning

Unlike many other methods, [Bayesian Inverse Reinforcement Learning](https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-416.pdf) produces a _posterior distribution_ over the reward functions that would explain the observed demonstrations. This distribution can be used for e.g. planning in a risk-averse manner. It works by starting with some randomly chosen reward function, and then repeating the following steps:

1. Perturb the reward function randomly
2. Solve for the optimal policy for that reward function
3. Use the learned policy to see how likely the demonstrations would be for the reward function
4. Use the likelihood to determine whether to take this new reward function, or return to the old one.

(This is the application of a standard MCMC sampling algorithm to the likelihood model used in IRL.)


I think I broadly agree with Daniel’s first two points in support of the task-based approach, and I was somewhat talking past him during the podcast. I generally _do_ agree that individual AI systems should be specialized to particular tasks or domains, and should not be “generically improving one’s life”. I agree with Daniel that at least outwardly it seems like most of the AI alignment field seems to be about building AI systems that can generically optimize your entire life, or even more ambitiously, the lot of humanity; I also agree that this is weird and probably not the right thing to do.

My optimism about helpfulness is not predicated on an idea that we’ll build AI systems that are generically trying to make all aspects of your life better: I do think that we still want our AI systems to be domain-specific, such as (say) a financial advisor AI system. The idea is more that if we can design domain-general _techniques_ that allow us to train domain-specific _systems_ that are trying to be helpful, that seems like it would be a solution to the AI alignment problem (the problem of how to prevent an AI from adversarially optimizing against its user).

AXRP 2: Learning Human Biases

After talking about <@my paper on learning biases@>(@Learning biases and rewards simultaneously@) (for which I refer you to the linked blog post and past AN summary), Daniel and I talked about the implications of inverse reinforcement learning for AI safety, and in particular how we would want AI systems to be architected at a high level.

My position was that we want intelligent AI systems to be trying to help their users: they are explicitly interacting with humans in order to clarify what they should do, perhaps by explicitly asking questions, or by watching other human decisions and making inferences about what humans must care about. (However, this isn’t the vast majority of what they do; it is probably significantly less than one-fifth of “everything they do”.)

In contrast, Daniel would prefer for a superintelligent AI system to be pursuing a well-defined task, such as “build a thriving city”. He has three reasons for this:

1. When our goal is to build AI systems that can pursue a relatively well-defined task, it is much easier for us to tell whether we are succeeding, and we can be much clearer about what it is we are trying to accomplish.
2. We can increase the difficulty of well-specified tasks over time, rising in tandem with the capabilities of AI systems. In contrast, if our AI system is supposed to generically make our life better, that seems like a fixed task that is fairly difficult and requires quite a high minimum threshold of capabilities.
3. It seems easier to tell whether your AI system has built a good city, than to tell whether an AI system has generically improved your life.

In the podcast, I don’t think I really engaged properly with the first two points, so I’ll talk about that in the opinion. I did disagree with the third point -- I don’t see why it should be harder to evaluate whether my life has been generically improved; for example, I expect that we are capable of telling apart good and bad personal assistants.

Daniel also asked why it helps to aim for “AI systems that are trying to help you” -- how has that made the problem any simpler? It seems to me that the notion of “helpfulness” is domain-independent: once you have the concept of being helpful, it can be applied in different domains. One hopes that we could then train lots of AI systems that are specialized to particular domains, but all of them are still trying to be helpful.


I really liked the insight: because IRL provides information about the maximum of a set, it is generally safe to underestimate that set (i.e. work with a subset), but is not safe to overestimate that set (i.e. work with a superset). It’s simple and intuitive but can plausibly provide significant benefits in practice.

I Know What You Meant: Learning Human Objectives by (Under)estimating Their Choice Set

<@Misspecification in reward learning@>(@Model Mis-specification and Inverse Reinforcement Learning@) can be quite bad, and seems nearly inevitable to happen. The key insight of this paper is that we can mitigate its effects by ensuring that we err on the side of _underestimating_ the demonstrator’s capabilities.

Consider inverse reinforcement learning (IRL), where we get demonstrations of good behavior. In practice, there are some demonstrations that humans can’t give: for example, when teleoperating a complex robot arm, humans might find it challenging to move a coffee cup without tilting it. Ideally, we would estimate the set of possible trajectories the demonstrator could have given, known as their choice set, and only model them as noisily rational across trajectories from that set.

However, we won’t perfectly estimate this choice set, and so there will be some misspecification. If we overestimate the demonstrator’s capabilities, for example by assuming they could move the coffee cup perfectly straightly, then since that _isn’t_ the demonstration we get we would infer that the human couldn’t have cared about keeping the cup upright. However, if we underestimate the demonstrator’s capabilities, there’s no such issue.

If we make the theoretical simplification that the demonstrator chooses the actual best trajectory out of their choice set, then we can prove that in the case of underestimation, you will always assign as much probability to the true reward function as you would if you had the correct choice set. (Intuitively, this is because for reward r, if the trajectory is optimal under the true choice set, then it must also be optimal under the underestimated choice set.)

Okay, but how do we ensure we have underestimated the choice set? This paper suggests that we augment the demonstrations that we do observe. For example, we can take the real demonstration, and inject noise into it, along the lines of <@D-REX@>(@Ranking-Based Reward Extrapolation without Rankings@). Alternatively, we can repeat actions -- the idea is that it is easier for a human to give consistent inputs than to change the actions constantly. Finally, we can make the demonstration sparser, i.e. reduce the magnitude of the actions (in the robotics setting).

The authors run experiments in simulated domains as well as with a user study and report good results.


With this paper, I conclude that IRL works for Mujoco-style control tasks. The performance of this method is similar to offline GAIL but is better justified and more stable. However, ultimately, I'm a bit skeptical of their claim that the method will generalize to other tasks. Results for GAIL/DAC are quite poor in Atari-like environments whereas pair-wise reward modeling seems to perform quite well. This would suggest a reward modeling approach would scale much better in more complicated settings.

Paper: Primal Wasserstein Imitation Learning

Imitation Learning in the Low-Data Regime

Robert Dadashi, Léonard Hussenot, Matthieu Geist, Olivier Pietquin

<@Non-Adversarial Imitation Learning@>(@Non-Adversarial Imitation Learning and its Connections to Adversarial Methods@) has become more popular recently due to the fact that GAN style architectures can be notoriously unstable during training. This paper makes a contribution by introducing an imitation learning strategy that relies on minimizing an upper bound on the Wasserstein distance between the imitator and expert state visitation distributions. The Wasserstein distance can be understood using the 'Earth Mover's Analogy'. In this interpretation, we view the distance as the cost of the most efficient transport strategy to move probability mass from the imitator distribution to the expert distribution. The advantage of such an approach is that the metric can be calculated in an offline way. If we calculate the distance for partial rollouts then we can create a dense, albeit non-stationary, reward for the imitator. In experiments, agents trained using the Wasserstein distance are able to learn control tasks using only a single trajectory.


What's most interesting about this paper (from an alignment perspective) is the increased sample efficiency of the learning of human preferences, by limiting the type of preferences that can be expressed to preferences over goal states in a goal-directed setting. While not all preferences could be captured this way, I think a large amount of them in a large number of settings could be - it might come down to creating a clever encoding of the task as goal-directed in a way an RL policy could learn.

Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery

Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, Sergey Levine

In reinforcement learning (RL), reward function specification is a central problem in training a successful policy. For a large class of tasks, we can frame the problem as goal-directed RL: giving a policy a representation of a goal (for example coordinates in a map, or a picture of a location) and training the policy to reach this goal. In this setting, the naive reward function would be to give a reward of 1 when the policy reaches the goal state (or very close to it), and a reward of 0 otherwise. However, this makes it difficult to train the correct policy, as it will need to explore randomly for a long time before finding the true reward. Instead, if we had a notion of distance within the environment, we could use the negative distance from the goal state as the reward function - this would give the policy good information about which direction it should be moving in, even if it hasn't yet found the reward.

This paper is about how to learn a distance function in an unsupervised manner, such that it's useful for shaping the reward of an RL policy. Given an environment without a reward function, and starting with a random goal-directed policy, they alternate between (1) choosing a state **s** to train the policy to reach, and (2) training a distance function **d(s*, s')** which measures the minimum number of environment steps it takes for the policy to reach a state **s*** from a different state **s'**. This distance function is trained with supervised learning using data collected by the policy acting in the environment, and is called the __Dynamical Distance__, as it measures the distance with respect to the environment dynamics and policy behaviour.

The key choice in implementing this algorithm is how states are chosen to train the policy (step 1). In the first implementation, the authors choose the state which is farthest from the current state or the starting state, to encourage better long-term planning and skills in the policy and better generalisation in the agent. In the second (and more relevant) implementation, the state is chosen from a selection of random states by a human who is trying to express a preference for a given goal state. This effectively trains the policy to be able to reach states which match humans preferences. This second method outperforms [Deep RL from Human Preferences](https://arxiv.org/abs/1706.03741) in terms of sample efficiency of human queries in learning human preferences across a range of locomotion tasks.


This paper is a great look at what reward learning would look like at scale. The most salient takeaways for me were that data quality becomes very important and having very large models does not mean that the reward can now be optimized arbitrarily.

Paper: Learning to summarize from human feedback

Learning to Summarize with Human Feedback

Nisan Stiennon*, Long Ouyang*, Jeff Wu*, Daniel M. Ziegler*, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano

OpenAI has been working on <@finetuning language models from human preferences@>(@Fine-Tuning GPT-2 from Human Preferences@). This blog post and paper show the progress they have made on text summarization in particular since their last release.

As a reminder, the basic setup is similar to that of [Deep RL from Human Preferences](https://deepmind.com/blog/learning-through-human-feedback/): we get candidate summaries by executing the policy, have humans compare which of two summaries is better, and use this feedback to train a reward model that can then be used to improve the policy. The main differences in this paper are:

1. They put in a lot of effort to ensure high data quality. Rather than having MTurk workers compare between summaries, they hire a few contractors who are paid a flat hourly rate, and they put a lot of effort into communicating what they care about to ensure high agreement between labelers and researchers.
2. Rather than collecting preferences in an online training setup, they collect large batches at a time, and run a relatively small number of iterations of alternating between training the reward model and training the policy. My understanding is that this primarily makes it simpler from a practical perspective, e.g. you can look at the large batch of data you collected from humans and analyze it as a unit.
3. They initialize the policy from a model that is first pretrained in an unsupervised manner (as in <@GPT-3@>(@Language Models are Few-Shot Learners@)) and then finetuned on the reference summaries using supervised learning.

On the Reddit task they train on, their summaries are preferred over the reference summaries (though since the reference summaries have varying quality, this does not imply that their model is superhuman). They also transfer the policy to summarize CNN / DailyMail news articles and find that it still outperforms the supervised model, despite not being trained at all for this setting (except inasmuch as the unsupervised pretraining step saw CNN / DailyMail articles).

An important ingredient to this success is that they ensure their policy doesn’t overoptimize the reward, by adding a term to the reward function that penalizes deviation from the supervised learning baseline. They show that if they put a very low weight on this term, the model overfits to the reward model and starts producing bad outputs.


Workshop on Incentives in Machine Learning, ICML 2020

According to me, the main insight of this paper is that it is both necessary and difficult to design mechanisms that incentivize principals to report not just the best thing to do, but a comparison amongst different alternatives. Within the formalism of paper, this is done by overriding a principal’s action unless it is a novel action, but I expect in practice we’ll do this in some other way (it seems rather unusual to imagine the agent overriding a human, I’d be surprised if that was how we ended up building our AI systems).

Multi-Principal Assistance Games

Arnaud Fickinger, Simon Zhuang, Dylan Hadfield-Menell, Stuart Russell

So far the work in the <@assistance games framework@>(@Human Compatible: Artificial Intelligence and the Problem of Control@) (previously called CIRL) has focused on the case where there is a single human and a single AI assistant. Once we have multiple humans (or _principals_, as the paper calls them), things get much trickier.

One problem is that we don’t know how to aggregate the values across different principals. Rather than taking a stance on the problem, this paper assumes that we have some mechanism that can combine reward functions in some reasonable way. It instead focuses on a second problem: while previously we could trust the human to report their preferences accurately (as the human and agent were aligned), when there are multiple principals whose preference will be aggregated, the principals have an incentive to misrepresent their preferences (which we’ll call non-straightforward play).

Let’s consider the case where the principals provide demonstrations, _and get reward for those demonstrations_. For now our agent will assume that the principals are playing straightforwardly, and so the agent simply infers their preferences, aggregates them, and optimizes the results. In this setting, if the agent will act far more often than the principals provide demonstrations (so that the reward of the demonstrations is almost irrelevant), we can apply the Gibbard-Satterthwaite theorem to show that any non-trivial mechanism will be vulnerable to non-straightforward play. In contrast, if the principals provide lots of demonstrations, while the agent only acts for a short period of time, then optimal principals primarily want to ensure their demonstrations are good, and so will be straightforward most of the time (provably). In the middle, the fact that principals get rewarded for demonstrations does help reduce non-straightforward play, but does not eliminate it.

Now let’s consider the case where the agent can design a mechanism. Here, when the principals are providing demonstrations, the agent can override their action choice with one of its own (a setting considered <@previously@>(@The Assistive Multi-Armed Bandit@)). Roughly speaking, the algorithm only executes a proposed human action if it hasn’t executed it before. By doing so, it incentivizes the principals to report second-best actions, and so on, giving the agent more information about the principals' utility functions. The mechanism incentivizes straightforward play, and is approximately efficient (i.e. there is an upper bound on the worst case social welfare achieved).


Since their theoretical results are only for optimal values, it's unclear whether starting from random initial policies will necessarily converge to these optimal values -- indeed, they make this point themselves, that they do not train to convergence as gradient descent cannot hope to find the global optimum for GAN-like non-convex loss functions. In light of that, it's not evident *why* their algorithms outperform the competition. Additionally, they do not report computational speed-up or wall-clock comparisons, which to me felt like the broad motivation behind this work. Nonetheless, the work illuminates new territory in adversarial imitation learning, provides positive evidence for a novel technique, and raises interesting questions for future work, such as how to learn robust reward functions via this method, or what kind of convergence properties can be expected.

Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization

Paul Barde*, Julien Roy*, Wonseok Jeon*, Joelle Pineau, Christopher Pal, Derek Nowrouzezahrai

This work aims to simplify algorithms for adversarial imitation learning by using a _structured_ discriminator, which is parameterised by the current generator and a learned policy. They prove that if so formulated, the policy that yields the optimal discriminator is exactly the same as the policy that generated the expert data, which is also precisely what we hope the generator will learn. As long as the discriminator's learned policy is parameterised correctly such that it can be sampled and evaluated, this eliminates the need for a reinforcement learning outer loop for policy improvement, as this learned policy can be substituted in for the generator's policy in the next training iteration. They empirically show the competitiveness of their method with state-of-the-art algorithms across a small but increasingly complex suite of tasks.


It's worth emphasizing that this isn't quite a pure IRL method. They use preferences over demonstrations in addition to the demonstrations themselves and so they have more information than would be available in a pure IRL context. However, it’s also worth emphasizing that (as the authors show) pixel-level features make it difficult to use IRL or GAIL to learn an imitation policy, which means I wasn’t expecting a pure IRL approach to work here. Conceptually, what's interesting about the Bayesian approach is that uncertainty in the reward distribution translates into confidence intervals on expected performance. This means that Bayesian REX is fairly robust to direct attempts at reward hacking due to the ability to directly measure overfitting to the reward function as high variance in the expected reward.

Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences

Daniel S. Brown, Russell Coleman, Ravi Srinivasan, Scott Niekum

Bayesian reward learning would allow for rigorous safety analysis when performing imitation learning. However, Bayesian reward learning methods are typically computationally expensive to use. This is because a separate MDP needs to be solved for each reward hypothesis. The main contribution of this work is a proposal for a more efficient reward evaluation scheme called Bayesian REX (see also an <@earlier version@>(@Deep Bayesian Reward Learning from Preferences@)). It works by pre-training a low-dimensional feature encoding of the observation space which allows reward hypotheses to be evaluated as a linear combination over the learned features. Demonstrations are ranked using pair-wise preference which is relativistic and thus conceptually easier for a human to evaluate. Using this method, sampling and evaluating reward hypotheses is extremely fast: 100,000 samples in only 5 minutes using a PC. Moreover, Bayesian REX can be used to play Atari games by finding a most likely or mean reward hypothesis that best explains the ranked preferences and then using that hypothesis as a reward function for the agent.


This is a cool idea! Even if you only have a dataset with ground truth (x, y) pairs, you could assume that h = y, and while you wouldn’t get benefit 1, you would still get benefit 2 above. If you constructed your dataset by the common method of getting a bunch of human predictions and defining y to be the modal prediction, then you could automatically construct h by having it be the average across the human predictions, and get both benefits above.

Learning to Complement Humans

Many current AI systems aim to assist humans in complex tasks such as medical diagnoses. Given that AI systems have a very different range of capabilities than humans, there has been a lot of interest in detecting “hard” examples and showing them to humans. This paper demonstrates how this can be done in an end-to-end way.

The authors assume they have access to an augmented supervised learning dataset of triples (x, y, h), where x is the input, y is the label, and h is the human prediction. A traditional approach would be to first train a model to predict y given x, and then come up with a new algorithm or model to predict when you should ask the human instead of querying the model. In contrast, they create a single model that first decides whether to look at h (for some fixed cost c), and then make a prediction given x (and h, if the model chose to look at it). They have two versions: a classic discriminative approach (very similar to e.g. image classifiers) and a decision-theoretic approach (where the model uses several probabilistic models and then calculates the value of information (VOI) of h to decide whether to query the human).

The end-to-end training confers two main benefits:
1. The models automatically learn to focus their learning capability on examples that are hard for humans.
2. The models ignore examples where they are going to ask a human anyway (rather than e.g. learning enough to make a 50% confident prediction).


<@Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning@>

Showing versus doing: Teaching by demonstration

Mark K. Ho, Michael Littman, James MacGlashan, Fiery Cushman, Joseph L. Austerweil

This paper creates and validates a model of _pedagogy_ as applied to reward learning. Typically, inverse reinforcement learning (IRL) algorithms assume access to a set of demonstrations that are created from an approximately _optimal_ policy. However, in practice, when people are asked to _show_ a task, they don't give the optimal trajectory; they give the trajectory that helps the learner best _disambiguate_ between the possible tasks. They formalize this by creating a model in two steps:

1. A literal or IRL robot is one which learns rewards under the model that the demonstrator is Boltzmann rational.
2. The pedagogic human shows trajectories in proportion to how likely a literal robot would think the true reward is upon seeing the trajectory.

They validate this model with user studies and find that it predicts human demonstrations well.


It's another example to compare to deep RL from human preferences.

Deep Reinforcement Learning from Human Preferences

Active Reward Learning

Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, Jan Peters

Reward learning on continuous control tasks, when using Relative Entropy Policy Search (REPS) for the policy. (This paper is from 2014, before deep learning was applied to robotics tasks.) Their setup is similar to deep RL from human preferences, where they have a reward predictor, which they then use to train a policy, which then generates trajectories and sends it back to the reward predictor. The reward predictor chooses a few trajectories and asks the expert about them; the expert provides a rating (a number), not a preference. This is then used to update the reward predictor.


This paper is a good example of reward learning, and it's interesting to compare and contrast this more principled method to Deep RL from Human Preferences. For example, while they maintain a distribution over rewards and choose the most useful query, deep RL from human preferences works with higher-dimensional reward functions where this would be too expensive, and so they instead train an ensemble of reward predictors and use disagreement between the reward predictors as a measurement of uncertainty.

Active Preference-Based Learning of Reward Functions

Dorsa Sadigh, Anca D. Dragan, Shankar Sastry, and Sanjit A. Seshia

In a continuous, dynamical system such as an autonomous car, how can we learn good behavior without hardcoding a reward function? It may be too hard to get demonstrations (humans could only show how they drive a car, not how they want the car to drive). The authors propose presenting the human with two trajectories and asking her to choose which one is better, and using this information to infer the reward.


Machine Theory of Mind

Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S.M. Ali Eslami, Matthew Botvinick


Policy Shaping: Integrating Human Feedback with Reinforcement Learning

Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L. Isbell and  Andrea L. Thomaz


This is a cool application of density-estimation via DualDICE. While the experiments are a bit weak, the fact that an off-policy method exists to do distribution-matching is interesting in its own right. Moreover, the method seems able to compete with both BC and GAIL-like methods which is intriguing.

Imitation Learning via Off-Policy Distribution Matching

Ilya Kostrikov, Ofir Nachum, Jonathan Tompson

One way to view imitation learning is as a distribution matching problem. In other words, the agent is rewarded based on how well it can imitate the state-distribution induced by the expert. In recent years, distribution matching via adversarial methods such as GAIL has become a popular approach to imitation learning. However, one weakness of these methods is that they require on-policy samples which means they require the agent to interact with the environment. In this paper, the authors present an off-policy method for distribution matching which can work without environment interaction. They do this by building on the prior work of DualDICE, a policy-agnostic method to estimate distribution ratios between agent and expert which can then be used to provide a reward to the agent. This allows the optimal policy to be estimated directly from demonstrations without any need for agent interaction. The authors run a few experiments and show that the method has comparable performance to behavioral cloning in the off-policy setting and adversarial methods in the on-policy setting.


The main contribution in this paper seems to be deriving a new bound so that AIRL can handle state-only imitation learning. The use of indirection via a buffer is also interesting and seems to be a good idea to provide stability in training. However, they did not do an ablation. Overall, it's aesthetically interesting that this paper is borrowing tricks, such as buffering and Wasserstein critic. Finally, the results seem promising, particularly for the sim-to-real problem. It would be interesting to see a follow-up to gauge whether or not I2L can help bridge this gap.

Paper: Learning Robust Rewards With Adversarial Inverse Reinforcement Learning

State-only Imitation with Transition Dynamics Mismatch

Most existing imitation learning algorithms rely on the availability of expert demonstrations that come from the *same* MDP as the one the imitator will be evaluated in. With the advent of <@adversarial inverse reinforcement learning (AIRL)@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@), it has become possible to learn general behaviors. However, algorithms such as <@GAIL@>(@Generative Adversarial Imitation Learning@) are capable of learning with just state-information, something that AIRL was not designed for. In this paper, the authors introduce indirect-imitation learning (I2L) to try and merge the benefits of both GAIL and AIRL. The basic sketch of the algorithm is to first use a generalization of AIRL to imitate demonstrations via a buffer distribution and then focus on moving that buffer closer to the expert's demonstration distribution using a Wasserstein critic, a smoother way to train GAN networks. By combining these two approaches, agents trained with I2L learn how to control Ant in regular gravity and can *generalize* to perform in simulations with differing parameters for gravity. For the suite of Gym continuous domains, they show consistent advantages for I2L over other algorithms such as GAIL, BCO, and AIRL when parameters such as friction, density, and gravity are changed.


I was surprised to see pure RL solutions rank in the leaderboard, given the limitations on compute and environment interactions. Notably though, while the second place team (pure imitation) got 42.41 points, the eighth place team (pure RL) only got 8.25 points.

More generally, I was excited to see an actual benchmark for techniques using human demonstrations: so far there hasn't been a good evaluation of such techniques. It does seem like Minecraft benefits a lot from hierarchy and action pruning, which we may not care about when evaluating algorithms.

The MineRL Competition on Sample-Efficient Reinforcement Learning Using Human Priors: A Retrospective

Stephanie Milani, Nicholay Topin, Brandon Houghton, William H. Guss, Sharada P. Mohanty, Oriol Vinyals, Noboru Sean Kuno

This paper reports on the results of the <@MineRL competition@>(@NeurIPS 2019 Competition: The MineRL Competition on Sample Efficient Reinforcement Learning using Human Priors@), in which participants had to train agents to obtain a diamond in Minecraft using a limited amount of compute, environment interactions, and human demonstrations. While no team achieved this task, one team did make it to the penultimate milestone: obtaining an iron pickaxe.

The top nine teams all used some form of action reduction: that is, they constrained their agents to only take a subset of all available actions, shaping the space in which the agent had to learn and explore. The top four teams all used some form of hierarchy in order to learn longer "options" that could then be selected from. The second place team used pure imitation learning (and so required _no_ environment interactions), while the eighth and ninth place teams used pure reinforcement learning (and so required _no_ human demonstrations).


This is very related to [Inverse Reward Design](https://arxiv.org/abs/1711.02827), where instead of maxmin planning we use risk-averse planning, and instead of considering all rewards compatible with an expert demonstration we consider all reward functions that are probable based on which reward function the designer wrote down.

Learning Safe Policies with Expert Guidance

Jessie Huang, Fa Wu, Doina Precup, Yang Cai

Expert demonstrations can be consistent with many possible reward functions. Instead of simply trying to mimic the demonstration, the authors consider all possible rewards that are consistent with the demonstration, and then maximize the worst reward, leading to safe behavior.


The basic idea of augmenting demonstration data by relabeling the goal is clever. While I understood HER suffered from being a breadth-first search algorithm, I hadn't considered the possibility that GAIL might be limited to only extracting direct information from the demonstrations. Generalizing GAIL so that it can reach arbitrary states allows for a smooth transition between learning from demonstrations and achieving the desired outcome.

Goal-conditioned Imitation Learning

Yiming Ding*, Carlos Florensa*, Mariano Phielipp, Pieter Abbeel

Goal-conditioned tasks are objectives that can be specified at the start of an episode. Specifically, the objective is set to encourage the agent to reach an arbitrary state in the environment. **This paper investigates using goal-conditioning to improve the performance of imitation learning algorithms.** The authors build off of prior work into Hindsight-Experience Replay (HER), a method that allows standard RL algorithms to learn from failure by relabeling final states as goal states. One drawback of HER is that the search process is breadth-first since we won't know which search directions are useful before we encounter the true goal state. This can complicate exploration. On the other hand, when we have access to expert demonstrations, such as in imitation learning, we can generally avoid breadth-first search and instead focus on copying the demonstrations using a method such as generative adversarial imitation learning (GAIL). However, with GAIL we evaluate entire agent trajectories as either similar/dissimilar from the expert demonstration. Yet, it's also true that we could view different points along the trajectory as sub-goals which greatly augment the demonstration data-set. Using this insight, the authors extend goal-conditioning to the imitation learning setting. The authors test their goal-conditioned algorithm on a variety of basic manipulation tasks and show that with the goal relabeling the task is learned faster and at a higher quality than with other approaches such as GAIL or HER.


I think this is an important critique: it seems to me that the hardest part of the three principles suggested in <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@) is the one that requires human preferences to be tied to human behavior. It seems quite hard to make an accurate and robust model for this, for reasons like the ones given in this post.

The two-layer model of human values, and problems with synthesizing preferences

This post points out a problem with the recent <@preference synthesis research agenda@>(@Research Agenda v0.9: Synthesising a human's preferences into a utility function@) (and presumably other value learning agendas as well): these agendas tend to require simple models of how human behavior, speech, or mental models relate to human preferences. However, in reality, it seems likely that the brain is a big learning machine without any innate "values", and what we experience as our conscious selves is a "strategy" chosen by this learning machine, and as such does not have a sensible interpretation as something that optimizes for "values". The author suggests that value learning agendas need to deal directly with the fact that there are these two "layers" in humans, and presents some preliminary thoughts that don't reach any particular conclusions.


Using vector fields to visualise preferences and make them consistent

This post proposes that we represent a person's preferences as follows: for every state, we have a vector whose direction specifies how the person would most like the state to change, and whose magnitude specifies the intensity of the preference. Under suitable conditions on the state space, this defines a vector field. Intransitive or circular preferences correspond to the [curl](https://en.wikipedia.org/wiki/Curl_(mathematics)) of the vector field. The authors propose that a consistent set of preferences can then be inferred by "removing the curl", e.g. by using the [Helmholtz decomposition](https://en.wikipedia.org/wiki/Helmholtz_decomposition).


I'm happy to see reward modeling being used to achieve new capabilities results, primarily because it may lead to more focus from the broader ML community on a problem that seems quite important for safety. Their reward sketching process is quite efficient and having more reward data from humans should enable a more faithful model, at least on tasks where humans are able to annotate accurately.

A Framework for Data-Driven Robotics

Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, Ziyu Wang

This paper presents a framework for using a mix of task-agnostic data and task-specific rewards to learn new tasks. The process is as follows:

 1. A human teleoperates the robot to provide a *demonstration*.  This circumvents the exploration problem, by directly showing the robot the relevant states.
 2. All of the robot's sensory input is saved to *NeverEnding Storage (NES)*, which stores data from all tasks for future use.
 3. Humans annotate a subset of the *NES* data via task-specific *reward sketching*, where humans draw a curve showing progress towards the goal over time (see paper for more details on their interface).
 4. The labelled data is used to train a *reward model*.
 5. The agent is trained using **all** the *NES* data, with the *reward model* providing rewards.
 6. At test-time, the robot continues to save data to the *NES*.

They then use this approach with a robotic arm on a few object manipulation tasks, such as stacking the green object on top of the red one. They find that on these tasks, they can annotate rewards at hundreds of frames per minute.


In general, I like the idea of 'translating' demonstrations from one domain into another. It's worth noting that there do exist methods for translating visual demonstrations into latent policies. I'm a bit surprised that we didn't see any comparisons with other adversarial methods like [GAIfO](https://arxiv.org/pdf/1807.06158.pdf), but I understand that those methods have high sample complexity so perhaps the methods weren't useful in this context. It's also important to note that these other methods would still require demonstration translation. Another criticism is that AVID is not fully autonomous since it relies on human feedback to progress between stages. However, compared to kinetic teaching or teleoperation, sparse feedback from a human overseer is a minor inconvenience.

Paper: AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos

Learning to Imitate Human Demonstrations via CycleGAN

Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, Sergey Levine

Most methods for imitation learning, where robots learn from a demonstration, assume that the actions of the demonstrator and robot are the same. This means that expensive techniques such as teleoperation have to be used to generate demonstrations. **This paper presents a method to engage in automated visual instruction-following with demonstrations (AVID) that works by translating video demonstrations done by a human into demonstrations done by a robot.** To do this, the authors use [CycleGAN](https://junyanz.github.io/CycleGAN/), a method to translate an image from one domain to another domain using unpaired images as training data. CycleGAN allows them to translate videos of humans performing the task into videos of the robot performing the task, which the robot can then imitate. In order to make learning tractable, the demonstrations had to be divided up into 'key stages' so that the robot can learn a sequence of more manageable tasks. In this setup, the robot only needs supervision to ensure that it's copying each stage properly before moving on to the next one. To test the method, the authors have the robot retrieve a coffee cup and make coffee. AVID significantly outperforms other imitation learning methods and can achieve 70% / 80% success rate on the tasks, respectively.


Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications


Maximum Causal Tsallis Entropy Imitation Learning

Kyungjae Lee, Sungjoon Choi, Songhwai Oh


Planning to Give Information in Partially Observed Domains with a Learned Weighted Entropy Model

Rohan Chitnis, Leslie Pack Kaelbling, Tomás Lozano-Pérez