It is worth noting that the authors use the <@Dopamine@>(@Introducing a New Framework for Flexible and Reproducible Reinforcement Learning Research@) framework for evaluating the model-free baselines, meaning that a slightly stunted version of Rainbow is used on an evaluation protocol different from the original publication without retuning hyperparameters. That said, DreamerV2 definitely performs at a level similar to Rainbow, which is significant progress in model-based RL. In particular, the fact that the reward can be inferred from the world model even without gradients flowing back from the reward suggests transferability of the world models to different tasks with the same underlying dynamics.

Paper: Mastering Atari with Discrete World Models

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, Jimmy Ba

Model-based reinforcement learning can have better sample efficiency, allows for smarter exploration strategies, and facilitates generalization between different tasks. Still, previous attempts at model-based RL on the Atari Benchmark like <@Dreamer@>(@Dream to Control: Learning Behaviors by Latent Imagination@) and <@SimPLe@>(@Simulated Policy Learning in Video Models@) were unable to compete with model-free algorithms in terms of final performance. This paper presents DreamerV2, a model-based algorithm that outperforms DQN and its variants -- including Rainbow -- in terms of both median human- or gamer-normalized performance and on mean world-record normalized performance on Atari after 200M environment steps, achieving roughly 35% on the latter (25% if algorithm performance is clipped to max out at 100% for each game).

DreamerV2 learns a recurrent state-space model that stochastically encodes frames and a hidden state into a latent variable and uses the hidden state to predict the next value of the latent variable. Frames and reward are then reconstructed using both the hidden state and the latent variable. A policy is obtained by actor-critic training on the latent state space, leveraging parallelization to train on 468B imagined samples. As DreamerV2 does not use MCTS, it requires 8x less wall clock time to train than the more complicated but better performing <@MuZero Reanalyze@>(@Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model@).  Unlike earlier approaches, DreamerV2 uses a vector of categorical latent variables rather than gaussians to enable better model predictions for dynamics with multiple distinct modes, as well as KL-balancing (scaling up the importance of the transition loss compared to the entropy regularizer on the latent variable). Ablations confirm that the image reconstruction loss is crucial for DreamerV2's performance and that both the use of discrete latent variables and KL-balancing lead to significant improvements. Interestingly, preventing the gradients for reward prediction from affecting the world model does not affect performance at all.


A common intuition that many researchers have is that specifying _what_ to do (the reward function) should be easier than specifying _how_ to do it (the policy). In practice, this _doesn't_ seem to be the case for deep learning, where imitation via inverse reinforcement learning (inferring a reward function and optimizing it) seems to be similar to imitation learning via behavior cloning ("copying" the policy). Similarly, this method seems broadly similar to meta learning algorithms like MAML and RL^2, though it does outperform them on one (probably carefully designed) transfer learning task.

What Can Learned Intrinsic Rewards Capture?

Zeyu Zheng*, Junhyuk Oh*, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado van Hasselt, David Silver, Satinder Singh

This paper studies whether a learned reward function can serve as a locus of knowledge about the environment, that can be used to accelerate training of new agents. In particular, such a learned intrinsic reward can help with test-time adaptation: in a novel environment, the intrinsic reward can quickly "tell" the agent e.g. where it should explore -- even if in the new environment the agent has a different action space, or uses a different learning algorithm (situations that meta learning would typically not be able to handle).

The authors create an algorithm that learns an intrinsic reward function, that when used to train a new agent over a "lifetime" (which consists of multiple episodes), leads to the best cumulative reward over the lifetime, using a meta-gradient approach. Experiments on gridworlds demonstrate that these learned intrinsic rewards: 1. switch between early exploration and later exploitation, 2. explore only for information that is relevant for optimal behavior, 3. capture invariant causal relationships, and 4. can anticipate and adapt to changes in the extrinsic reward within a lifetime.


At first glance, it would seem as though the idea of using an off-policy method to combine imitation and reinforcement learning is obvious. However, the implementation is complicated by the fact that we want the value functions being estimated by our agent to satisfy the optimality condition for the Bellman equation. Prior work, such as [Hester et al. 2018](https://aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16976/16682) uses n-step returns to help pre-training and make use of on-policy methods when performing RL. What I like about this paper is that they perform an ablation study and show that simple sequencing of imitation learning and RL algorithms isn't enough to get good performance. This means that combining the imitation and reinforcement objectives into a single loss function is providing a significant improvement over other methods.

Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Sparse Reward Environments

Vinicius G. Goecks, Gregory M. Gremillion, Vernon J. Lawhern, John Valasek, Nicholas R. Waytowich

This paper contributes to the effort of combining imitation and reinforcement learning to train agents more efficiently. The current difficulty in this area is that imitation and reinforcement learning proceed under rather different objectives which presents a significant challenge to updating a policy learned from a pure demonstration. A major portion of this difficulty stems from the use of so-called "on-policy" methods for training which require a significant number of environment interactions to be effective. In this paper, the authors propose a framework dubbed "Cycle-of-Learning" (CoL) that allows for the off-policy combination of imitation and reinforcement learning. This allows the two approaches to be combined much more directly which grounds the agent's policy in the expert demonstrations while simultaneously allowing for RL to fine-tune the policy. The authors show that CoL is an improvement over the current state of the art by testing their algorithm in several environments and performing an ablation study.


On net, I am quite happy to see work which argues in favour of reducing time spent hand-tuning and hand-crafting parts of a complex pipeline, and demonstrates the alternatives that currently exist to do so.

However, I feel the work did not fully compare the trade-off between tuning hyperparameters, and increasing the complexity of the pipeline by adding the adaptive components. I agree, though, that the latter is a one-time effort (per inductive bias), and is thus far more scalable than the former which needs to be repeated for each bias for every new task.

It would also be interesting to see how adaptive agents fare on problems where we care more about failures than successes, or if they are better/worse suited for safe exploration than baseline agents. My intuition is that adaptive internals of the agent cause it behave more noisily/unpredictably, and it may not fare as well as our current efforts for such problems.

**Rohin's opinion:** While it's certainly true that fewer inductive biases imply more general agents, it also usually means more compute and data requirements. For action repetition and learned discount factors, only one new parameter has to be learned, so it doesn't make much of a difference either way (and in fact performance on Atari doesn't change much). Clipped rewards do in fact learn faster than PopArt. I don't know why a recurrent network improves upon standard observation preprocessing for Atari -- perhaps initially RNNs were hard to train, and it became a de facto standard to use observation preprocessing, and no one checked about using recurrent networks later when RNNs became easier to train?

On Inductive Biases in Deep Reinforcement Learning

Matteo Hessel*, Hado van Hasselt*, Joseph Modayil, David Silver

The fewer inductive biases we use, the more general our algorithms will be. But how much does it really help to have fewer inductive biases? This paper replaces several hand-engineered components of an A2C agent with generic or adaptive variants.

Specifically, they compared: 1) reward clipping vs. reward normalization via <@PopArt@>(@Preserving Outputs Precisely while Adaptively Rescaling Targets@), 2) handpicked discount factor vs. online adaptive discounting via meta-learning, 3) fixed action repeats vs. learned action-commitment, and 4) standard Atari observation preprocessing vs. passing raw observations to a recurrent network. Over 57 Atari tasks, they found that the tuned algorithm outperformed the adaptive method only in (1). Performance was similar for (2) and (3), and the proposed method outperformed the baseline for (4). When the fully adaptive agent was compared to the vanilla agent (with heuristics designed for Atari) over 28 unseen continuous control tasks, the adaptive agent performed better in 14 of them, worse in one, and about the same in the rest, providing evidence that fewer inductive biases do lead to more general agents.


While the work in this paper is solid, the conclusions don't seem particularly surprising: everyone knows that deep RL is incredibly sample intensive (which straightforwardly implies inaccurate gradient estimates) and relies on many implementation tricks. I'm not familiar enough with PPO to know how surprising their last result is.

Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms?

Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry

This paper argues that policy gradient algorithms are very dependent on additional optimisations (such as value function clipping, reward scaling, etc), and that they operate with poor estimates of the gradient. It also demonstrates that the PPO objective is unable to enforce a trust region, and that the algorithm's empirical success at doing so is due to the additional optimisations.


I agree that having standard benchmarks is often useful for spurring progress in deep learning, and that this one will be useful. I'm somewhat concerned that the tasks the authors have selected (CartPole, HalfCheetah, etc) are too simple, and that the property they're measuring is more like robustness to peturbations than the sort of combinatorial generalisation discussed in [this paper] (http://arxiv.org/abs/1806.01261) from [last week's newsletter](https://mailchi.mp/c1f376f3a12e/alignment-newsletter-30). The paper would benefit from more clarity about what they mean by "generalisation".

Assessing Generalization in Deep Reinforcement Learning

Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, Dawn Song

This paper aims to create a benchmark for measuring generalisation in reinforcement learning. They evaluate a range of standard model-free algorithms on OpenAI Gym and Roboschool environments; the extent of generalisation is measured by varying environmental parameters at test time (note that these tasks are intended for algorithms which do not update at test time, unlike many transfer and multi-task learners). They distinguish between two forms of generalisation: interpolation (between values seen during training) and extrapolation (beyond them). The latter, which is typically much harder for neural networks, is measured by setting environmental parameters to more extreme values in testing than in training.


While I don’t think this paper is particularly novel, it usefully combines several ideas and provides easily-interpretable results.

CURIOUS: Intrinsically Motivated Multi-Task, Multi-Goal Reinforcement Learning

Cédric Colas, Pierre Fournier, Olivier Sigaud, Mohamed Chetouani, Pierre-Yves Oudeyer

This paper presents an intrinsically-motivated algorithm (an extension of Universal Value Function Approximators) which learns to complete multiple tasks, each parameterised by multiple "goals" (e.g. the locations of targets). It prioritises replays of tasks which are neither too easy nor too hard, but instead allow maximal learning progress; this also help prevent catastrophic forgetting by refocusing on tasks which it begins to forget.


Reinforcement Learning for Improving Agent Design

This paper explores what happens when you allow an RL agent to modify aspects of the environment; in this case, the agent's body. This allows you to learn asymmetric body designs that are better suited for the task at hand. There's another fun example of specification gaming -- the agent makes its legs so long that it simply falls forward to reach the goal.


I find Starcraft more interesting as a test-bed for deep RL than as a goal in itself. While the results of this paper are cool,  I doubt that its methods will scale well - in general, approaches which rely on a lot of human knowledge being hard-coded in don't tend to.

Note the similarity between the macros in this paper and the way that OpenAI Five could choose between different hand-coded combinations of items. However, the latter is only a small part of the game, whereas the former is much more extensive.

TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game

Peng Sun, Xinghai Sun, Lei Han, Jiechao Xiong, Qing Wang, Bo Li, Yang Zheng, Ji Liu, Yongsheng Liu, Han Liu, Tong Zhang

This paper showcases an RL agent which is able to defeat the built-in Starcraft AI (roughly at the level of the 30th-50th percentile of players). It do so by choosing between 165 hand-coded macro actions, which each correspond to an elementary task like producing a certain building. This avoids the necessity of learning unimportant details like exactly where the building should be placed, as well as difficult rules like the prerequisites for each building. The authors create a second agent which uses an expert system to choose actions in a hierarchical fashion, which performs at a similar level to the first. See also [Import AI](https://jack-clark.net/2018/09/25/import-ai-113-why-satellitesai-gives-us-a-global-eye-industry-pays-academia-to-say-sorry-for-strip-mining-it-and-kindred-researchers-seek-robot-standardization/).


The two properties that this paper highlights do seem important, and the fact that they can be varied in Space Fortress makes it a good benchmark for them.

I'm not convinced that the experimental work is particularly useful, though. It seems to reinforce the well-known point that shaped rewards can work well when they're shaped in sensible ways, and much less well otherwise.

Challenges of Context and Time in Reinforcement Learning: Introducing Space Fortress as a Benchmark

The authors note that most existing RL benchmarks (like Atari games) lack sharp context-dependence and temporal sensitivity. The former requires an agent to sometimes change strategies abruptly; the latter requires an agent's strategy to vary over time. Space Fortress is an arcade-style game which does have these properties, and which cannot be solved by standard RL algorithms, even when rewards are made dense in a naive way. However, when the authors shape the rewards to highlight the context changes, their agent achieves superhuman performance.


This paper feels incomplete; it has very little discussion of the reasons why meta-learning and adversarial training failed, and the data augmentation result is rather perfunctory.

Learning Invariances for Policy Generalization

Remi Tachet des Combes, Philip Bachman, Harm van Seijen

This paper compares three ways to induce generalisation in deep RL agents: data augmentation, meta-learning and adversarial training. They find that data augmentation boosts performance on a simple task from 2.8% to 99.8%, whereas the other two don't improve on the baseline.


The use of embeddings in OpenAI Five

This blog post discusses some information which has been released by OpenAI about the structure of the OpenAI Five networks. Starting from Dota API information, each unit, ability, item and modifier is converted to an embedding. During processing, there's some surprising use of max-pooling over those embeddings even when it doesn't seem appropriate. Actions are chosen depending on the dot product similarity between their embedding and the final LSTM output. In a previous version, the target for the chosen action was picked without conditioning on what the action actually was; this has changed in more recent versions.


If you find my summary thought-provoking, then yes.

[Beyond algorithmic equivalence: algorithmic noise](https://www.lesserwrong.com/posts/meG3Pai2YeRYcwPwS/beyond-algorithmic-equivalence-rewards) and [Beyond algorithmic equivalence: self-modelling](https://www.lesserwrong.com/posts/kmLP3bTnBhc22DnqY/beyond-algorithmic-equivalence-self-modelling)

Given human behavior, it is hard to figure out whether the behavior is generated because it is good for the human's values, or because the human is biased. One way to tell whether or not it is a bias is to look at the decision process going on inside the human's brain and see whether or not the human would remove the bias if possible.


The approach seems very empirically-minded and not theoretically justified, which makes me worry that we would need to run a lot of experiments to actually get this into a form that doesn't fail immediately. That said, we do need some way of dealing with human inconsistencies, and I think the ideas behind the framework (such as endorsements) are intriguing and useful for future work on this problem.

Resolving human values, completely and adequately

IRL and other value learning approaches make the assumption that the human has a consistent reward function, whereas in reality human values are "contradictory, underdefined, changeable, and manipulable". How can we use the output from algorithms like IRL given this? He proposes a framework where humans can have _values_ that can "endorse" rewards and other values, and defines a process that can take these endorsements and compute an overall reward function to optimize. The hope here is for the overall reward to be adequate, not perfect.


If you found my example interesting and worth thinking about, it is worth reading the post to see an attempt at formalizing it.

Why we want unbiased learning processes

Suppose we have a reward learning agent where we have designed the reward space so that "ask the human whether to do X" always has higher reward than "do X". The agent is now considering whether to ask the human to try heroin, or just give them heroin. If the agent gives them heroin, it will see their look of ecstasy and will update to have the reward function "5 for giving the human heroin, 7 for asking the human". If the agent asks the human, then the human will say "no", and the agent will update to have the reward function "-1 for giving the human heroin, 1 for asking the human". In both cases asking the human is the optimal action, yet the agent will end up giving the human heroin, since that gets reward 5, while asking the human gets reward 1. The post doesn't have this example, it has a formal model and an abstract example in that model that is isomorphic to the example I have.


A theory of human values

This post presents an outline of how to construct a theory of human values. First, we need to infer preferences and meta-preferences from humans who are in "reasonable" situations. Then we need to synthesize these into a utility function, by resolving contradictions between preferences, applying meta-preferences to preferences, and having a way of changing the procedures used to do the previous two things. We then need to argue that this leads to adequate outcomes -- he gives some simple arguments for this, that rely on particular facts about humans (such as the fact that they are scope insensitive).


Hierarchical system preferences and subagent preferences

Often, when looking at a hierarchical system, you can ascribe preferences to the system as a whole, as well as to individual parts or subagents of the system. Often, any divergence between the parts can be either interpreted as a difference between the goals of the parts, or a failure of rationality of one of the parts. The post gives a particular algorithm, and shows how based on the structure of the code of the subagent, we could either infer that the subagent is mistaken, or that it has different goals.

Now, we could infer meta-preferences by seeing how the system tends to self-modify -- perhaps we notice that it tends to amplify the "goals" of one particular subagent, in which case we can infer that it has a meta-preference for those goals. But without that, there's no correct answer to what the true goals are. In the post's own words: "In the absence of some sort of meta-preferences, there are multiple ways of establishing the preferences of a hierarchical system, and many of them are equally valid."


I like this theoretical analysis that shows what could go wrong with processes that learn preferences. I did find it a bit hard to connect the ideas in this post with concrete reward learning algorithms (such as inverse reinforcement learning) -- it seems plausible to me that if I properly understood what the formal definitions of unriggable and uninfluenceable meant in the IRL setting, I wouldn't view them as desirable.

[Reward function learning: the value function](https://www.lesswrong.com/posts/55hJDq5y7Dv3S4h49/reward-function-learning-the-value-function) and [Reward function learning: the learning process](https://www.lesswrong.com/posts/upLot6eG8cbXdKiFS/reward-function-learning-the-learning-process)

These posts introduce a theoretical framework for reward learning, where a reward learning algorithm is modeled as something that produces a probability distribution over reward functions given a history and current policy. With such a general notion of reward learning, it becomes hard to define the value function -- while we still want something like sum of expected rewards, it is no longer clear how to take an expectation over the reward function, given that the distribution over it can change over time. Most plausible ways of doing this lead to time-inconsistent decisions, but one works well. The second post turns to the learning process and analyzes properties that it would be nice to have. In the worst case, we can get quite pathological behavior, but of course we get to choose the learning algorithm so we can avoid worst-case behavior. In general, we would want our learning algorithm to be _unriggable_ and/or _uninfluenceable_, but this is not possible when learning from humans since different policies on the AI's part will lead to it learning different rewards.


The fact that humans are quite good at inferring preferences should give us optimism about value learning. In the [framework](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/h9DesGT3WT9u2k7Hr) of rationality with a mistake model, we are trying to infer the mistake model from the way that humans infer preferences about other humans. This sidesteps the impossibility result by focusing on the _structure_ of the algorithm that generates the policy. However, it still seems like we have to make some assumption about how the structure of the algorithm leads to a mistake model, or a model for what values are. Though perhaps we can get an answer that is principled enough or intuitive enough that we believe it.

Figuring out what Alice wants: non-human Alice

We know that if we have a potentially irrational agent, then inferring their preferences is [impossible](https://arxiv.org/abs/1712.05812) without further assumptions. However, in practice we can infer preferences of humans quite well. This is because we have very specific and narrow models of how humans work: we tend to agree on our judgments of whether someone is angry, and what anger implies about their preferences. This is exactly what the theorem is meant to prohibit, which means that humans are making some strong assumptions about other humans. As a result, we can hope to solve the value learning problem by figuring out what assumptions humans are already making and using those assumptions.


There's one possible view where you look at the impossibility result around inferring preferences, and think that value alignment is hopeless. I don't subscribe to this view, for basically the reasons given in this post -- while you can't infer preferences for arbitrary agents, it certainly seems possible for humans in particular.

That said, I would expect that we accomplish this by learning a model that implicitly knows how to think about human preferences, rather than by explicitly constructing particular normative assumptions that we think will lead to good behavior. Explicit assumptions will inevitably be [misspecified](https://www.alignmentforum.org/posts/gnvrixhDfG7S2TpNL/latent-variables-and-model-mis-specification) ([AN #32](https://mailchi.mp/8f5d302499be/alignment-newsletter-32)), which is fine if we can correct the misspecification in the future, but at least under the threat model of an AI system that prevents us from changing its utility function (which I believe is the threat model Stuart usually considers) this isn't an option available to us.

["Normative assumptions" need not be complex](https://www.alignmentforum.org/posts/QzsCrzGd4zkNwk9cd/normative-assumptions-need-not-be-complex), [Humans interpreting humans](https://www.alignmentforum.org/posts/eXiuk8Yjd7FsunhxM/humans-interpreting-humans) and [Anchoring vs Taste: a model](https://www.alignmentforum.org/posts/7gMkkEMQa78h8Bpmr/anchoring-vs-taste-a-model)

We have seen before that since humans are not perfectly rational, it is [impossible](https://www.alignmentforum.org/posts/ANupXf8XfZo2EJxGv/humans-can-be-assigned-any-values-whatsoever) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)) to deduce their preferences, even with a simplicity prior, without any additional assumptions. This post makes the point that those assumptions need not be complex -- for example, if we could look at the "source code" of an agent, and we can find one major part with the same type signature as a reward function, and another major part with the type signature of a planner, then we can output the first part as the reward function. This won't work on humans, but we can hope that a similarly simple assumption that bakes in a lot of knowledge about humans could allow us to infer human preferences.

Since we seem to be very capable of inferring preferences of other humans, we might want to replicate our normative assumptions. The key idea is that we model ourselves and others in very similar ways. So, we could assume that if H is a human and G another human, then G's models of H's preferences and rationality are informative of H's preferences and rationality.

Stuart then shows how we could apply this to distinguish between choices made as a result of anchoring bias vs. actual taste preferences. Suppose that in condition 1, our human H would pay $1 or $3 for the same bar of chocolate depending on whether they were anchored on $0.01 or $100, and in condition 2 they would pay $1 or $3 depending on whether the chocolate has nuts. Ideally, we'd call the first case a bias, and the second one a preference. But in both cases, H's choice was determined by access to some information, so how can we distinguish between them? If we have access to H's internal model, we might expect that in the nuts case the information about nuts passes through a world model that then passes it on to a reward evaluator, whereas in the anchoring case the world model throws the information away, but it still affects the reward evaluator through a side channel. So we could add the normative assumption that only information that goes through the world model can be part of preferences. Of course, we could imagine another agent where the anchoring information goes through the world model and the nuts goes through the side channel -- but this agent is not human-like.


I'm not quite sure what overall point Stuart is trying to make. However, if we're concerned that an agent might manipulate humans, I don't see why we should trust it to aggregate the data from many one-step hypotheticals, since "manipulation" could then occur using the many degrees of freedom involved in choosing the questions and interpreting the answers.

[One-step hypothetical preferences](https://www.alignmentforum.org/posts/i6hWWcKyxBPj7ELT6/one-step-hypothetical-preferences) and [A small example of one-step hypotheticals](https://www.lesswrong.com/posts/zo5K8QeDZiLicSCe6/a-small-example-of-one-step-hypotheticals)

We don't hold most of our preferences in mind at any given time - rather, they need to be elicited from us by prompting us to think about them. However, a detailed prompt could be used to manipulate the resulting judgement. In this post, Stuart discusses hypothetical interventions which are short enough to avoid this problem, while still causing a human to pass judgement on some aspect of their existing model of the world - for example, being asked a brief question, or seeing something on a TV show. He defines a one-step hypothetical, by contrast, as a prompt which causes the human to reflect on a new issue that they hadn't considered before. While this data will be fairly noisy, he claims that there will still be useful information to be gained from it.


If you previously were optimistic about inverse reinforcement learning and similar techniques because you thought they could infer the same notion of value that the brain computes, this seems like an important counterargument. However, it should be noted that the authors are _not_ arguing that the brain is not optimizing some specific notion of value: just that it is not _explicitly computing_ such a notion of value. (Similarly, policies learned by RL optimize the reward function, even though they need not explicitly calculate the reward of every state in order to choose an action.) So you could still hope that the brain is optimizing some notion of value that isn’t explicitly computed, and then use inverse RL to recover that notion of value.

The case against economic values in the brain

It has been common in the neuroeconomics literature to assume (based on past research) that the brain explicitly computes some notion of value in order to make choices. This paper argues that this is wrong: it is plausible that the brain does not in fact explicitly calculate values, and instead directly learns a policy that produces actions.


Rigging is a form of wireheading

Ideally in reward learning, we want our AI to be learning a fact about the world -- which reward it should be optimizing. However, for most proposals of reward learning, the AI's actions can also influence this "fact" about the world. In this case, the AI can wirehead by influencing the world so that it learns an easy-to-maximize reward. This is what Stuart calls "rigging" of the learning process.


It does seem like we should be able to learn the things that humans mostly agree on, and that this can help us a lot with inferring human preferences. I don't know if the goal is to use these models to infer broad human values, or something a lot simpler. Broad human values seems very unlikely to work, since you are trying to get to superhuman ability at knowing what humans want by mimicking human models (which are tautologically not superhuman).

Figuring out what Alice wants, parts [I](https://www.lesswrong.com/posts/so78tQvTCaBgw4bfL/figuring-out-what-alice-wants-part-i) and [II](https://www.lesswrong.com/posts/rcXaY3FgoobMkH2jc/figuring-out-what-alice-wants-part-ii)

Since it's not possible to infer human preferences without making some normative assumption about the human, we should try to learn the models that humans use to reason about each other that allow us to infer preferences of other humans. While we can't get access to these models directly, we can access fragments of them -- for example, whenever a person expresses regret, that can be taken as an mismatch between the model expectation and actual outcome. Part II goes through two example scenarios and what the internal human models might look like, and the challenges that arise in trying to learn them.


Some in the safety community have speculated that robustness to data shift (sometimes called "transfer learning" in the safety community) cannot be resolved only by leveraging more GPUs and more data. Also, it is argued that the difficulty in attaining data shift robustness suggests longer timelines. Both this paper and [Robustness properties of Facebook's ResNeXt WSL models](https://arxiv.org/abs/1907.07640) analyze the robustness of models trained on over 100 million to 1 billion images, rather than only training on ImageNet-1K's ~1 million images. Both papers show that data shift robustness greatly improves with more data, so data shift robustness appears more tractable with deep learning. These papers evaluate robustness using benchmarks collaborators and I created; they use [ImageNet-A](https://arxiv.org/abs/1907.07174), [ImageNet-C](https://arxiv.org/abs/1903.12261), and [ImageNet-P](https://arxiv.org/abs/1903.12261) to show that performance tremendously improves by simply training on more data. See [Figure 2](https://arxiv.org/pdf/1911.04252.pdf#page=6&zoom=100,0,88) of the Noisy Student paper for a summary of these three benchmarks. Both the Noisy Student and Facebook ResNeXt papers have problems. For example, the Noisy Student paper trains with a few expressly forbidden data augmentations which overlap with the ImageNet-C test set, so performance is somewhat inflated. Meanwhile, the Facebook ResNeXt paper shows that more data does not help on ImageNet-A, but this is because they computed the numbers incorrectly; I personally verified Facebook's ResNeXts and more data brings the ImageNet-A accuracy up to 60%, though this is still far below the 95%+ ceiling. Since [adversarial robustness can transfer to other tasks](https://arxiv.org/pdf/1901.09960.pdf), I would be surprised if robustness from these models could not transfer. These results suggest data shift robustness can be attained within the current paradigm, and that attaining image classifier robustness will not require a long timeline.

Self-training with Noisy Student improves ImageNet classification

Qizhe Xie, Eduard Hovy, Minh-Thang Luong, Quoc V. Le

Instead of summarizing this paper, I'll provide an opinion describing the implications of this and other recent papers.


I am quite excited about the focus on preventing low likelihood catastrophic events, particularly from the standpoint of existential risk reduction. The key assumption in this paper, that earlier in training the agent will fail in related ways but more frequently, seems plausible to me and in line with most of my experience training neural networks, and the experiments demonstrate a very large increase in efficiency.

I’d be interested to see theoretical analysis of what situations would make this assumption more or less likely in the context of more powerful future agents. For example, one situation where the failure modes might be distinct later in training is if an agent learns how to turn on a car, which then makes states where the agent has access to a car have significantly higher likelihood of catastrophic failures than they did before.

Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures

Jonathan Uesato*, Ananya Kumar*, Csaba Szepesvari*, Tom Erez, Avraham Ruderman, Keith Anderson, Krishmamurthy (Dj) Dvijotham, Nicolas Heess, Pushmeet Kohli

An important problem in safety-critical domains is accurately estimating slim probabilities of catastrophic failures: one in a million is very different from one in a billion. A standard Monte Carlo approach requires millions or billions of trials to find a single failure, which is prohibitively expensive. This paper proposes using agents from earlier in the training process to provide signals for a learned failure probability predictor. For example, with a Humanoid robot, failure is defined as the robot falling down. A neural net is trained on earlier agents to predict the probability that the agent will fall down from a given state. To evaluate the final agent, states are importance-sampled based on how likely the neural network believes they are to cause failure. This relies on the assumption that the failure modes of the final agent are similar to some failure mode of earlier agents. Overall, the approach reduces the number of samples required to accurately estimate the failure probability by multiple orders of magnitude.


I really like the analogy between programming and AI; a lot of my thoughts have been shaped by thinking about this analogy myself. I agree that the analogy implies that we are trying to solve problems that we've attacked before in a different context, but I do think there are significant differences now. In particular, with long-term AI safety we are considering a setting in which mistakes can be extremely costly, _and_ we can't provide a formal specification of what we want. Contrast this to traffic lights, where mistakes can be extremely costly but I'm guessing we can provide a formal specification of the safety constraints that need to be obeyed. To be fair, Pushmeet acknowledges this and highlights specification learning as a key area of research, but to me it feels like a qualitative difference from previous problems we've faced, whereas I think Pushmeet would disagree with that (but I'm not sure why).

<@Towards Robust and Verified AI: Specification Testing, Robust Training, and Formal Verification@>

Designing robust & reliable AI systems and how to succeed in AI

(As is typical for large content, I'm only summarizing the most salient points, and ignoring entire sections of the podcast that didn't seem as relevant.)

In this podcast, Rob delves into the details of Pushmeet's work on making AI systems _robust_. Pushmeet doesn't view AI safety and AI capabilities as particularly distinct -- part of building a good AI system is ensuring that the system is safe, robust, reliable, and generalizes well. Otherwise, it won't do what we want, so why would we even bother using it. He aims to improve robustness by actively searching for behaviors that violate the specification, or by formally verifying particular properties of the neural net. That said, he also thinks that one of the major challenges here is in figuring out the specification of what to verify in the first place.

He sees the problems in AI as being similar to the ones that arise in programming and computer security. In programming, it is often the case that the program that one writes down does not accurately match the intended specification, leading to bugs. Often we simply accept that these bugs happen, but for security critical systems such as traffic lights we can use techniques like testing, fuzzing, symbolic execution, and formal verification that allow us to find these failures in programs. We now need to develop these techniques for machine learning systems.

The analogy can go much further. Static analysis involves understanding properties of a program separately from any inputs, while dynamic analysis involves understanding a program with a specific input. Similarly, we can have "static" interpretability, which understands the model as a whole (as in [Feature visualization](https://distill.pub/2017/feature-visualization/)), or "dynamic" interpretability, which explains the model's output for a particular input. Another example is that the technique of abstract interpretation of programs is analogous to a particular method for verifying properties of neural nets.

This analogy suggests that we have faced the problems of AI safety before, and have made substantial progress on them; the challenge is now in doing it again but with machine learning systems. That said, there are some problems that are unique to AGI-type systems; it's just not the specification problem. For example, it is extremely unclear how we should communicate with such a system, which may have its own concepts and models that are very different from those of humans. We could try to use natural language, but if we do we need to ground the natural language in the way that humans do, and it's not clear how we could do that, though perhaps we could test if the learned concepts generalize to new settings. We could also try to look at the weights of our machine learning model and analyze whether it has learned the concept -- but only if we already have a formal specification of the concept, which seems hard to get.


A naive alignment strategy and optimism about generalization

We want to build an AI system that answers questions honestly, to the best of its ability. One obvious approach is to have humans generate answers to questions, select the question-answer pairs where we are most confident in the answers, and train an AI system on those question-answer pairs.

(I've described this with a supervised learning setup, but we don't have to do that: we could also [learn](https://deepmind.com/blog/learning-through-human-feedback/) from [comparisons](https://ai-alignment.com/optimizing-with-comparisons-c02b8c0d7877) between answers, and we only provide comparisons where we are confident in the comparison.)

What will the AI system do on questions where we _wouldn't_ be confident in the answers? For example, questions that are complex, where we may be misled by bad observations, where an adversary is manipulating us, etc.

One possibility is that the AI system learned the **intended policy**, where it answers questions honestly to the best of its ability. However, there is an **instrumental policy** which also gets good performance: it uses a predictive model of the human to say whatever a human would say. (This is "instrumental" in that the model is taking the actions that are instrumental to getting a low loss, even in the test environment.) This will give incorrect answers on complex, misleading, or manipulative questions -- _even if_ the model "knows" that the answer is incorrect.

Intuitively, "answer as well as you can" feels like a much simpler way to give correct answers, and so we might expect to get the intended policy rather than the instrumental policy. This view (which seems common amongst ML researchers) is _optimism about generalization_: we are hoping that the policy generalizes to continue to answer these more complex, misleading, manipulative questions to the best of its ability.

Are there reasons to instead be pessimistic about generalization? There are at least three:

1. If the answers we train on _aren't_ perfectly correct, the instrumental policy might get a _lower_ training loss than the intended policy (which corrects errors that humans make), and so be more likely to be found by gradient descent.
2. If the AI already needs to make predictions about humans, it may not take much "additional work" to implement the instrumental policy. Conversely, if the AI reasons at a different level of abstraction than humans, it may take a lot of "additional work" to turn correct answers in the AI's ontology into correct answers in human ontologies.
3. From [a followup post](https://www.alignmentforum.org/posts/SRJ5J9Tnyq7bySxbt/answering-questions-honestly-given-world-model-mismatches), the AI system might answer questions by translating its concepts to human concepts or observations, and then deduce the answer from those concepts or observations. This will systematically ignore information that the AI system understands that isn't represented in the human concepts or observations. (Consider the [example](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/) of the robot hand that only _looked_ like it was grasping the appropriate object.)

A possible fourth problem: if the AI system did the deduction in its own concepts and only as a final step translated it to human concepts, we might _still_ lose relevant information. This seems not too bad though -- it seems like we should at least be able to <@explain the bad effects of a catastrophic failure@>(@Can there be an indescribable hellworld?@) in human concepts, even if we can't explain why that failure occurred.

A [followup post](https://www.alignmentforum.org/posts/roZvoF6tRH6xYtHMF/avoiding-the-instrumental-policy-by-hiding-information-about) considers whether we could avoid the instrumental policy by <@preventing it from learning information about humans@>(@Thoughts on Human Models@), but concludes that while it would solve the problems outlined in the post, it seems hard to implement in practice.


Section 1.3 of [this version of the paper](https://mltheory.org/dg_short.pdf)

Distributional Generalization: A New Kind of Generalization

Suppose you train a classifier to distinguish between CIFAR-10 classes, except each airplane has a 30% chance of being mislabeled as a car. If you then train a model to achieve perfect accuracy on this badly labeled dataset, it will get 100% accuracy on the training set, and 97% of those labels will actually be correct (since 3% are mislabeled airplanes). Under the current paradigm, if we say that the model "generalizes", that means that it will also get 97% accuracy at test time (according to the actually correct labels). However, this doesn't tell us anything about what mistakes are made at test time -- is it still the case that 30% of airplanes are mislabeled as cars, or does the model also make mistakes on e.g. deer?

_Distributional generalization_ aims to make claims about situations like these. The core idea is to make claims about the full distribution of classifier outputs, rather than just the single metric of test accuracy.

Formally, we assume there is some distribution D, from which we can sample pairs of points (x, y), which generates both our train and test sets. Then, the train (resp. test) distribution of classifier outputs is (x, f(x)), with x coming from the train (resp. test) set. The train and test distributions of classifier outputs are the objects of study in distributional generalization. In particular, given a [0,1]-valued function on distributions (called a _test_ T), we say that the classifier _generalizes w.r.t T_ if T outputs similar values on the train and test distribution. (W.r.t means "with respect to".) For example, given a distribution, the _accuracy test_ checks how often the classifier's output is correct in expectation over that distribution. Generalization w.r.t the accuracy test is equivalent to the canonical notion of generalization.

Let's suppose that the classifier perfectly fits the training set, so that the train distribution of classifier outputs is the same as the original distribution D. Let's additionally suppose that the classifier generalizes with respect to the accuracy test, so that the classifier has perfect test accuracy. Then, the test distribution of classifier outputs will also be the same as the original distribution D, that is, all distributions are identical and there isn't much more to say. So, the interesting situations are when one of these two assumptions is false, that is, when either:
1. The classifier does not perfectly fit the training set, or
2. The classifier does not generalize w.r.t accuracy.
This paper primarily focuses on classifiers that _do_ perfectly fit the training set, but don't generalize w.r.t accuracy. One typical way to get this setting is to inject label noise (as in the mislabeled airplanes case), since this prevents the classifier from getting 100% test accuracy.

Speaking of which, let's return to our original example in which we add label noise by mislabeling 30% of airplanes as cars. Notice that, since the label noise is completely divorced from the classifier's input x, the best way for the classifier to minimize _test_ loss would be to always predict the true CIFAR-10 label, and then 3% of the time the true distribution will say "lol, jk, this airplane is actually a car". However, in practice, classifiers will also label approximately 30% of airplanes as cars in the test set as well! This incurs higher loss, because the 30% of airplanes that the classifier labels as cars must be independent of the 30% of airplanes that the true distribution labels as cars, which implies that the model disagrees with the true distribution 4.2% of the time; this is worse than the 3% it would get if it consistently labeled airplanes as airplanes. **Classifiers trained to interpolation are not Bayes-optimal in the presence of label noise.**

Okay, let's get back to distributional generalization. We already know the classifier does not generalize w.r.t accuracy. However, the fact that it still labels about 30% of airplanes as cars suggests a different kind of generalization. Recall that the train and test distributions of classifier outputs have the form (x, f(x)). Consider the feature L(x) that says whether x is an airplane or not. Then, if we replace (x, f(x)) with (L(x), f(x)), then this now looks identical between the train and test distributions! Specifically, this distribution places 7% on ("yes airplane", "airplane"), 3% on ("yes airplane", "car"), and 10% on ("no airplane", c) for every class c other than "airplane". An alternative way of stating this is that the classifier generalizes w.r.t all tests whose dependence on x factors through the feature L. (In other words, the test can only depend on whether x is an airplane or not, and cannot depend on any other information about x.)

The authors make a more general version of this claim they call _feature calibration_: for every feature L that _could_ be learned by the classifier, the classifier generalizes w.r.t all tests whose dependence on x factors through L. Note that they do not assume that the classifier _actually_ learns L: just that, if you hypothetically trained the classifier on a dataset of (x, L(x)), then it could learn that function near-perfectly.

They then provide evidence for this through a variety of experiments and one theorem:
- If you plug in the constant feature L(x) = 0 into the conjecture, it implies that classifiers should get the right class balance (i.e. if your distribution contains class 1 twice as often as class 0, then you predict class 1 twice as often as class 0 at test time). They demonstrate this on a rebalanced version of CIFAR-10, even for classifiers that generalize poorly w.r.t accuracy.
- When using a WideResNet (for which the true CIFAR-10 labels are learnable), if you add a bunch of structured label noise into CIFAR-10, the test predictions reflect that same structure.
- The same thing is true for decision trees applied to a molecular biology dataset.
- A ResNet-50 trained to predict attractiveness on the CelebA dataset (which does not generalize w.r.t accuracy) does satisfy feature calibration w.r.t "wearing lipstick", "heavy makeup", "blond hair", "male", and "eye-glasses". Note there is no label noise in this case.
- AlexNet predicts that the right fraction of dogs are Terriers, even though it mistakes which exact dogs are Terriers.
- The nearest-neighbor classifier provably satisfies feature calibration under relatively mild regularity conditions.

In an appendix, they provide preliminary experiments suggesting this holds _pointwise_. In our mislabeled airplane example, for a _specific_ airplane x from the test set, if you resample a training set (with the 30% mislabeling of airplanes) and retrain a classifier on that set, then there is a roughly 30% chance that that specific x will be misclassified as a car.

The authors then introduce another distributional generalization property: _agreement_. Suppose we have two classifiers f and g trained on independently sampled training sets. The agreement conjecture states that the test accuracy of f is equal to the expected probability that f agrees with g on the test distribution (loosely speaking, this is how often f and g make the same prediction for test inputs). The agreement property can also be framed as an instance of distributional generalization, though I won't go into the specific test here. The authors perform similar experiments as with feature calibration to demonstrate that the agreement property does seem to hold across a variety of possible classifiers.

Interestingly, these properties are _not_ closed under ensembling. In our mislabeled airplane example, every model will label 30% of airplanes as cars, but _which_ airplanes are mislabeled is independent across models. As a result, the plurality voting used in ensembles reduces the misclassification rate to 22%, which means that you no longer satisfy feature calibration. Consistent with this, the authors observe that neural network ensembles, random forests, and k-nearest neighbors all did not satisfy feature calibration, and tended to be closer to the Bayes-optimal solution (i.e. getting closer to being robust to label noise, in our example).

Summary of the summary: Let's look at the specific ways in which classifiers make mistakes on the test distribution. This is called distributional generalization. The paper makes two conjectures within this frame. _Feature calibration_ says that for any feature that a classifier could have learned, the distribution of its predictions, conditioned on that feature, will be the same at train and test time, including any mistakes it makes. _Agreement_ says that the test accuracy of a classifier is equal to the probability that, on some randomly chosen test example, the classifier's prediction matches that of another classifier trained on a freshly generated training set. Interestingly, while these properties hold for a variety of ML models, they do not hold for ensembles, because of the plurality voting mechanism.


While the problem presented in this paper isn’t particularly novel, I appreciated the framing of fragility of distributional shift as being caused by underspecification. I see concerns about <@inner alignment@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) as primarily worries about underspecification, rather than distribution shift more generally, so I’m happy to see a paper that explains it well.

That being said, the experiments with neural networks were not that compelling -- while it is true that the models had higher variance on the metrics testing robustness to distributional shift, on an absolute scale the variance was not high: even a standard deviation of 0.024 (which was an outlier) is not huge, especially given that the distribution is being changed.

Underspecification Presents Challenges for Credibility in Modern Machine Learning

Alexander D'Amour*, Katherine Heller*, Dan Moldovan*, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, D. Sculley

This paper explains one source of fragility to distributional shift, which the authors term _underspecification_. The core idea is that for any given training dataset, there are a large number of possible models that achieve low loss on that training dataset. This means that the model that is actually chosen is effectively arbitrarily chosen from amongst this set of models. While such a model will have good iid (validation) performance, it may have poor inductive biases that result in bad out-of-distribution performance.

The main additional prediction of this framing is that if you vary supposedly "unimportant" aspects of the training procedure, such as the random seed used, then you will get a different model with different inductive biases, which will thus have different out-of-distribution performance. In other words, not only will the out-of-distribution performance be worse, its _variance_ will also be higher.

The authors demonstrate underspecification in a number of simplified theoretical settings, as well as realistic deep learning pipelines. For example, in an SIR model of disease spread, when we only have the current number of infections during the initial growth phase, the data cannot distinguish between the case of having high transmission rate but low durations of infection, vs. a low transmission rate but high durations of infection, even though these make very different predictions about the future trajectory of the disease (the out-of-distribution performance).

In deep learning models, the authors perform experiments where they measure validation performance (which should be relatively precise), and compare it against out-of-distribution performance (which should be lower and have more variance). For image recognition, they show that neural net training has precise validation performance, with 0.001 standard deviation when varying the seed, but less precise performance on <@ImageNet-C@>(@Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations@), with standard deviations in the range of 0.002 to 0.024 on the different corruptions. They do similar experiments with medical imaging and NLP.


While I agree with the general thrust of this article, comparing image misclassification rates to rates of catastrophic failures in aviation seems a bit harsh. I am having difficulties imagining an aviation system that fails due to a single input that has been processed wrongly, even though the correlation between subsequent failures given similar inputs might mean that this is not necessary for locally catastrophic outcomes.

**Rohin's opinion:** My guess is that we’ll need to treat systems based primarily on neural nets similarly to operators. The main reason for this is that the tasks that AI systems will solve are usually not even well-defined enough to have a reliability rate like 0.99999997 (or even a couple of orders of magnitude worse). For example, human performance on image classification datasets is typically under 99%, not because humans are bad at image recognition, but because in many cases what the “true label” should be is ambiguous. For another example, you’d think “predict the next word” would be a nice unambiguous task definition, but then for the question “How many bonks are in a quoit?“, should your answer be [“There are three bonks in a quoit”](https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html) or [“The question is nonsense”](https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/)? (If you’re inclined to say that it’s obviously the latter, consider that many students will do something like the former if they see a question they don’t understand on an exam.)

Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance

Test, Evaluation, Verification, and Validation (TEVV) is an important barrier for AI applications in safety-critical areas. Current TEVV standards have very different rules for certifying _software_ and certifying _human operators_. It is not clear which of these processes should be applied for AI systems.

If we treat AI systems as similar to human operators, we would certify them ensuring that they pass tests of ability. This does not give much of a guarantee of robustness (since only a few situations can be tested), and is only acceptable for humans because humans tend to be more robust to new situations than software. This could be a reasonable assumption for AI systems as well: while systems are certainly vulnerable to adversarial examples, the authors find that AI performance degrades surprisingly smoothly out of distribution in the absence of adversaries, in a plausibly human-like way.

While AI might have some characteristics of operators, there are good reasons to treat it as software. The ability to deploy multiple copies of the same system increases the threat of correlated failures, which is less true of humans. In addition, parallelization can allow for more extensive testing that is typical for software TEVV. For critical applications, a common standard is that of Safety Integrity Levels (SILs), which correspond to approximate failure rates per hour. Current AI systems fail way more often than current SILs for safety-critical applications demand. For example an image recognition system would require an accuracy of 0.99999997 at 10 processed frames per second just to reach the weakest SIL used in aviation.

However, SILs are often used on multiple levels and it is possible to build a system with a strong SIL from weaker components by using redundant components that fail independently or by detecting failures sufficiently early, such that AI modules could still be used safely as parts of a system specifically structured to cope with their failures. For example, we can use out-of-distribution detection to revert to a safe policy in simple applications. However, this is not possible for higher levels of automation where such a policy might not be available.


I think this is an awesome paper which, among other things, points at potential research directions for increasing OOD robustness: training more, training more diversely, and training in self-supervised ways. I think it's pretty noteworthy that larger models don't increase OOD robustness in NLP (all else equal), because it implies that certain guarantees may be constrained entirely by training procedures.

Pretrained Transformers Improve Out-of-Distribution Robustness

Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, Dawn Song

One important metric for the performance of deep learning models is the extent to which they generalize to examples that are _out-of-distribution_ (OOD) from the original distribution on which they were trained. This ability is sometimes called out-of-distribution _robustness_. This paper examines the OOD robustness of several NLP models: a bag-of-words model, word embedding models that use word averages, LSTMs, or ConvNets, and several models that use pretrained bidirectional transformers (BERT).

The paper finds that:
- Pretrained transformers (BERT) are significantly more OOD robust.
- Pretrained transformers (BERT) are significantly better at _detecting_ when they've encountered an OOD example. Previous models do worse than random chance at detection.
- Larger models don't increase OOD robustness in NLP the way they seem to in computer vision.
- Model distillation (using a larger trained neural network to train a smaller neural network) reduces OOD robustness, suggesting that naive in-distribution tests for model distillation methods may mask later failures.
- More diverse data improves OOD robustness.

The paper hypothesizes that these pretrained models may perform better because they were pretrained on particularly diverse data, were trained on a large amount of data, and were trained with self-supervised objectives, which previous work has suggested improves OOD robustness and detection.


While I'm not sure to what extent models learn truly irrelevant information (see <@Adversarial Examples Are Not Bugs, They Are Features@>), it seems good to add an incentive against learning information that won't be useful for a downstream task, and the empirical results (especially of the next paper) suggest that it is providing some benefit.

The Conditional Entropy Bottleneck

While I've categorized this paper under robustness because it can apply to most forms of training, I'll talk about it specifically in the context of unsupervised learning (and in particular its relation to Contrastive Predictive Coding (CPC), summarized in the highlights).

One potential problem with deep learning is that there might be too _much_ information in the input, causing the model to learn spurious correlations that do not actually generalize well (see <@Causal Confusion in Imitation Learning@> as an example). The idea with CEB is to penalize the model for learning irrelevant information, using a form of _information bottleneck_.

We consider a setting where we want to learn a representation **Z** of some input data **X** in order to predict some downstream data **Y**. In CPC, **X** would be the inputs from time 1 to t, **Z** would be the latent representation **z_t**, and **Y** would be the future data **x_{t+k}**. Then, we want **Z** to capture the **minimum necessary information** needed for **Z** to predict **Y** as best as possible. The _necessary_ information is **I(Y; Z)**, that is, the mutual information between **Z** and **Y**: we want to maximize this to maximize our accuracy at predicting **Y**. Since **Y** depends on **X** and **Z** is computed from **X**, any information about **Y** must come through mutual information between **X** and **Z**. Maximizing just this **I(Y; Z)** term gives us Contrastive Predictive Coding.

However, we don't want to capture any extra irrelevant information (the minimality criterion), which means that **Z** shouldn't capture any _more_ information about **X** beyond what it captured to maximize **I(Y; Z)**. In information-theoretic terms, we want to _minimize_ **I(X; Z | Y)**. Thus, we have the CEB objective: minimizing **I(X; Z | Y) - γ I(Y; Z)**, where **γ** is a hyperparameter controlling the tradeoff between the two terms. The authors then use some fairly straightforward math to reduce the objective to simpler terms which can be bounded using variational approximations, leading to an algorithm that can work in practice.

The authors perform experiments on Fashion MNIST and CIFAR10 (where Y corresponds to the labels for the images, so we're in the supervised learning setting). Since the main benefit of CEB is to remove unnecessary information from the model, they evaluate adversarial robustness and out-of-distribution detection in addition to standard performance checks. They find that models trained with CEB perform better than ones trained with a variational information bottleneck, or ones trained with vanilla SGD.


CEB Improves Model Robustness

This empirical paper finds that ImageNet classifiers trained with the CEB objective (summarized above) are already somewhat adversarially robust, without having any decrease in accuracy, and without any adversarial training. Notably, since CEB does not rely on knowing the attack method ahead of time, its adversarial robustness generalizes to multiple kinds of attacks, whereas models that were adversarially trained tend to be fragile in the face of previously unseen attacks.


Call for Papers: ICML 2019 Workshop on Uncertainty and Robustness in Deep Learning

Topics of this workshop include out-of-distribution detection, calibration, robustness to corruptions, robustness to adversaries, etc. Submissions are due April 30th.


The semi-verified model seems highly useful for various safety-related scenarios including learning with [label](https://arxiv.org/abs/1803.09050) [corruption](https://arxiv.org/abs/1802.05300), poisoned input data, and minimal supervision.

Learning from Untrusted Data

Moses Charikar, Jacob Steinhardt, Gregory Valiant

This paper introduces semi-verified learning. Here a model learns from a verified or trusted dataset, and from an untrusted dataset which consists in a mixture of legitimate and arbitrary examples. For the untrusted dataset, it is not known which points are legitimate and which are not. This scenario can occur when data is scraped from the internet, recorded by unreliable devices, or gathered through [crowdsourcing](https://ai.googleblog.com/2017/04/federated-learning-collaborative.html). Concretely if a (possibly small) fraction of the scraped data is hand-labeled, then this could count as the trusted set, and the remaining data could be considered the untrusted set. This differs from semi-supervised learning where there are labeled and unlabeled task-relevant examples. Here there are trusted examples and examples which are untrusted (e.g., labels may be wrong, features may be out-of-distribution, examples may be malicious, and so on). See the full paper for theorems and an algorithm applicable to tasks such as robust density estimation.


Experimentally evaluating whether honesty generalizes

The highlighted post introduced the notion of optimism about generalization. On this view, if we train an AI agent on question-answer pairs (or comparisons) where we are confident in the correctness of the answers (or comparisons), the resulting agent will continue to answer honestly even on questions where we wouldn't be confident of the answer.

While we can't test exactly the situation we care about -- whether a superintelligent AI system would continue to answer questions honestly -- we _can_ test an analogous situation with existing large language models. In particular, let's consider the domain of unsupervised translation: we're asking a language model trained on both English and French to answer questions about French text, and we (the overseers) only know English.

We could finetune the model on answers to questions about grammar ("Why would it have been a grammatical error to write Tu Vas?") and literal meanings ("What does Defendre mean here?"). Once it performs well in this setting, we could then evaluate whether the model generalizes to answer questions about tone ("Does the speaker seem angry or sad about the topic they are discussing?"). On the optimism about generalization view, it seems like this should work. It is intentional here that we only finetune on two categories rather than thousands, since that seems more representative of the case we'll actually face.

There are lots of variants which differ in the type of generalization they are asking for: for example, we could finetune a model on all questions about French text and German text, and then see whether it generalizes to answering questions about Spanish text.

While the experiments as currently suggested probably won't show good generalization, a variant that could support it would be one in which we train for _plausibility_. In our original example, we finetune on correct answers for grammar and literal meanings, and then we _also_ finetune to have the model give _plausible_ answers to tone (i.e. when asked about tone, instead of saying "en colère means 'angry'", the model says "the author is angry, as we can see from the use of 'en colère'"). It seems possible that this combination leads to the model giving actually correct answers about tone, just because "honestly report your best guess" seems like the simplest policy that meets all of these criteria.


Evaluating the Robustness of Collaborative Agents

Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, A. D. Dragan, Rohin Shah

Assuming a well-specified reward function, we would like to evaluate robustness of an agent by looking at the average reward it obtains on a wide scenario of plausible test time inputs that it might get. However, the key challenge of robustness is that it is hard to specify the test distribution in advance, and we must work with the training distribution instead.

This paper (on which I am an author) proposes _measuring_ robustness using a suite of hand-designed _unit tests_. Just as a function is tested by having the programmer write down potential edge cases and checking for the expected behavior, AI developers can come up with a set of potential "edge case" situations (especially ones not likely to arise during training) and check whether the agent's behavior on these situations works well or not. Intuitively, since these unit tests are created separately from the training process, they may not have the same spurious correlations that could be present in the training data. Thus, they can serve as an evaluation of the robustness of the agent.

The authors built a test suite for <@Overcooked@>(@Collaborating with Humans Requires Understanding Them@), and use it to evaluate several techniques aimed to improve the robustness of agents trained to collaborate with humans.

For example, one technique is to start each episode from a state sampled randomly from a dataset of human-human gameplay, so that the agents learn how to handle a broader diversity of states. This technique _decreases_ the average _validation_ reward, and if that's all we look at, we would conclude that it did not work. However, the technique also _increases_ performance on the unit test suite, suggesting that in reality the technique does increase robustness, though it comes at the cost of reduced performance when playing with the particular set of partners that make up the validation distribution.


I think it is plausible that the final stages of AGI deployment are carried out by the US military, similar to how the Manhattan Project took on the final stages of nuclear bomb development. If this happens, it will be important for the DoD to recognize and address issues of AI safety very carefully. If the TEVV process does get updated in a meaningful way soon, this could be a great opportunity for the AI safety community to try to translate some of its research into practice as well as building credibility and trust with the DoD.

Separately from long-term AGI deployment, I think that the DoD could be a good partner for AI safety research, particularly regarding areas like adversarial training and robustness. My main concern is that confidentiality might lead to safety research being hidden and thus less useful than if it was carried out in the open. I don’t know enough about the details of working with the DoD to know how likely that is to be an issue, however.

Building Trust through Testing

Michèle A. Flournoy, Avril Haines, Gabrielle Chefitz

In order to deploy AI systems, especially in the military, we will need to have good reason to _trust_ them. However, under the current machine learning paradigm, there are several challenges to building such trust. These challenges are broken down into technological and bureaucratic challenges. Technological challenges include lack of robustness, lack of representative test sets, lack of interpretability, and complexity once ML is integrated into larger systems. Bureaucratic challenges include lack of coordination, recruiting talent, and coordination between DoD, the private sector, and academia.

To address these challenges, the authors suggest that DoD updates its testing, evaluation, verification, and validation (TEVV) process to handle AI systems. They make 11 recommendations. A few that I found particularly interesting are:
- Create an OSD coordinating body to lead on AI/ML TEVV and incentivize strong cooperation with the services.
- Develop industry / U.S. government TEVV standards and promote them internationally.
- Test, train, and certify human-machine teams through wargaming, simulation, and experimentation.
- Increase resources for and attention on adversarial testing and red-teaming.


I was surprised that the accuracies the paper presented after adversarially training on TextFooler-produced sentences still weren't very high-- BERT's after-attack accuracy on one dataset went from 11.5% to 18.7%, and on another went from 4.0% to 8.3%. The paper didn't give a detailed description of its retraining procedure, so this may just be because they didn't adversarially train as much as they could have.

**Rohin's opinion:** This is an instance of the general trend across domains where if you search in a black-box way around training or test inputs, you can relatively easily uncover examples where your model performs poorly. We've seen this with adversarial examples in image classification, and with <@adversarial@>(@Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures@) <@policies@>(@Adversarial Policies: Attacking Deep Reinforcement Learning@) in deep reinforcement learning.

Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment

Di Jin*, Zhijing Jin*, Joey Tianyi Zhou, Peter Szolovits

This paper presents TextFooler, an algorithm for generating adversarial text for natural language tasks with only black-box access to models. TextFooler tries to generate sentences that are grammatical and semantically similar to original input sentences but produce incorrect labels. It does this by identifying a small set of most important words in the original sentence, generating candidate synonyms for those words, and gradually replacing the important words in the sentence by testing which synonyms cause the model to mispredict or report the least confidence score.

TextFooler is tested on three state-of-the-art NLP models-- WordCNN, WordLSTM, and BERT, all trained to ~80 - 90% test accuracy. On a variety of text classification datasets, TextFooler reduces accuracy to below ~15% with less than ~20% of the words perturbed. Humans evaluating the generated sentences say they are approximately as grammatical as the original, have the same label as the original in ~90% of cases, and have a sentence similarity score to the original sentence of 0.9 on a 0 to 1 scale. The paper finds that generally, models with higher original accuracy have higher after-attack acuracy.

The authors retrain BERT from scratch using data produced by TextFooler and then attack it using TextFooler again. They find that the after-attack accuracy is higher and that attacks require more perturbed words.


Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks

Mingchen Li, Mahdi Soltanolkotabi and
Samet Oymak

Previous [empirical](https://arxiv.org/pdf/1901.09960.pdf#page=2&zoom=100,0,81) [papers](https://papers.nips.cc/paper/8094-generalized-cross-entropy-loss-for-training-deep-neural-networks-with-noisy-labels.pdf) have shown that finding ways to decrease training time greatly improves robustness to label corruptions, but to my knowledge this is the first theoretical treatment.


The biggest challenge I see with this area of research, at least in its application to powerful and general AI systems, is how you get the specification in the first place, so I'm glad to see "learning specifications" as one of the areas of interest.

If I take the view from this post, it seems to me that techniques like domain randomization, and more generally training on a larger distribution of data, would count as an example of the second type of research: it is a change to the training procedure that allows us to meet the specification "the agent should achieve high reward in a broad variety of environments". Of course, this doesn't give us any provable guarantees, so I'm not sure if the authors of the post would include it in this category.

Towards Robust and Verified AI: Specification Testing, Robust Training, and Formal Verification

Pushmeet Kohli, Krishnamurthy (Dj) Dvijotham, Jonathan Uesato, Sven Gowal, and the Robust & Verified Deep Learning group

This post highlights three areas of current research towards making robust AI systems. First, we need better evaluation metrics: rather than just evaluating RL systems on the environments they were trained on, we need to actively search for situations in which they fail. Second, given a specification or constraint that we would like to ensure, we can develop new training techniques that can ensure that the specifications hold. Finally, given a specification, we can use formal verification techniques to ensure that the model obeys the specification on all possible inputs. The authors also list four areas of future research that they are excited about: leveraging AI capabilities for evaluation and verification, developing publicly available tools for evaluation and verification, broadening the scope of adversarial examples beyond the L-infinity norm ball, and learning specifications.


Adversarial Examples Are a Natural Consequence of Test Error in Noise

Nicolas Ford, Justin Gilmer, Nicolas Carlini, Dogus Cubuk

While this was previously summarized in [AN #32](https://mailchi.mp/8f5d302499be/alignment-newsletter-32), this draft is much more readable.


Trustworthy Deep Learning Course

Jacob Steinhardt, Dawn Song, Trevor Darrell

This underway course covers topics in AI Safety topics for current deep learning systems. The course includes slides and videos.


AI Alignment Podcast: The Byzantine Generals’ Problem, Poisoning, and Distributed Machine Learning

Byzantine resilience is the ability of a system to operate successfully when some of its components have been corrupted, even if it's unclear which ones they are. In the context of machine learning, this is relevant to poisoning attacks in which some training data is altered to affect the batch gradient (one example being the activity of fake accounts on social media sites). El Mahdi explains that when data is very high-dimensional, it is easy to push a neural network into a bad local minimum by altering only a small fraction of the data. He argues that his work on mitigating this is relevant to AI safety: even superintelligent AGI will be vulnerable to data poisoning due to time constraints on computation, and the fact that data poisoning is easier than resilient learning.


ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, Wieland Brendel

This paper empirically demonstrates the outsized influence of textures in classification. To address this, they apply style transfer to ImageNet images and train with this dataset. Although training networks on a specific corruption tends to provide robustness only to that specific corruption, stylized ImageNet images supposedly lead to generalization to new corruption types such as uniform noise and high-pass filters (but not blurs).


Iterative Learning with Open-set Noisy Labels

Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, Shu-Tao Xia

Much previous research on corrupted learning signals deals with label corruption, but this CVPR 2018 paper considers learning with corrupted or irrelevant inputs. For example, they train a CIFAR-10 classifier on CIFAR-10 data mixed with out-of-class CIFAR-100 data; such a scenario can occur with flawed data curation or data scraping. They use a traditional anomaly detection technique based on the local outlier factor to weight training examples; the more out-of-distribution an example is, the less weight the example has in the training loss. This approach apparently helps the classifier cope with irrelevant inputs and recover accuracy.


Teaching ML to answer questions honestly instead of predicting human answers

This post presents an algorithm that aims to solve the second problem from the highlighted post. As a reminder, the second problem is that an AI system that already has to make predictions about humans might learn a policy that is just "say what humans would say", since that is simpler than learning another translation that maps its knowledge to human language (so that it can answer honestly to the best of its knowledge).

The core idea is to have a "simple" labeling process and a "complex" labeling process, where the complex process can catch errors from the simple process. We'll also assume we have a simple / complex dataset (labeled by the simple / complex processes respectively), where we are confident that all of the answers in the dataset are correct. This means that the complex dataset has more challenging questions than the simple one. The _simple / complex instrumental policy_ is the policy that predicts what the simple / complex labeling process would do.

As an inefficient first algorithm, we can train our model on answers from the simple process where we are confident in their correctness, and then simply check whether the model performs well on new, harder questions where the complex and simple processes disagree. If the model fails this check, we restart from scratch. Intuitively, when the model eventually passes the check, it has probably learned the intended policy, as that's the one which would generalize zero-shot to complex questions as well (and in particular, the simple instrumental policy would _not_ generalize in this way).

Of course, we need a more efficient algorithm than that. Instead of creating an algorithm where the intended policy would work but the instrumental policy wouldn't, could we instead change the inductive biases so that the intended policy is _likely_ while the instrumental policy is not? Our original worry was that the intended policy has to bake in two implementations of language -- one in its world model, and one when translating answers into human-understandable concepts. So we could instead try to train a model that learns language from the simple instrumental policy, but is also trained on the complex dataset. The hope would be that the intended policy can learn the second implementation of language "for free" from the simple instrumental policy, while still working on the complex dataset. The actual details are quite complex and I'm not going to go into them here.

[This post](https://www.alignmentforum.org/posts/gEw8ig38mCGjia7dj/answering-questions-honestly-instead-of-predicting-human) by Evan Hubinger points out some problems and potential solutions with the approach.


AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

Dan Hendrycks*, Norman Mu*, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, Balaji Lakshminarayanan

This paper introduces a data augmentation technique to improve robustness and uncertainty estimates. The idea is to take various random augmentations such as random rotations, produce several augmented versions of an image with compositions of random augmentations, and then pool the augmented images into a single image by way of an elementwise convex combination. Said another way, the image is augmented with various traditional augmentations, and these augmented images are "averaged" together. This produces highly diverse augmentations that have similarity to the original image. Unlike techniques such as AutoAugment, this augmentation technique uses typical resources, not 15,000 GPU hours. It also greatly improves generalization to unforeseen corruptions, and it makes models more stable under small perturbations. Most importantly, even as the distribution shifts and accuracy decreases, this technique produces models that can [remain calibrated under distributional shift](https://openreview.net/pdf?id=S1gmrxHFvB#page=8&zoom=100,144,298).


An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods

Sanghyuk Chun, Seong Joon Oh, Sangdoo Yun, Dongyoon Han, Junsuk Choe, Youngjoon Yoo

There are several small tricks to improve classification performance such as label smoothing, dropout-like regularization, mixup, and so on. However, this paper shows that many of these techniques have mixed and often negative effects on various notions of robustness and uncertainty estimates.


Improving Robustness of Machine Translation with Synthetic Noise

Vaibhav Vaibhav, Sumeet Singh, Craig Stewart, Graham Neubig

By injecting noise (such as typos, word omission, slang) into the training set of a machine translation model, the authors are able to improve performance on naturally occurring data. While this trick usually does not work for computer vision models, it can work for NLP models.


Redwood Research’s current project

This post introduces Redwood Research's current alignment project: to ensure that a language model finetuned on fanfiction never describes someone getting injured, while maintaining the quality of the generations of that model. Their approach is to train a classifier that determines whether a given generation has a description of someone getting injured, and then to use that classifier as a reward function to train the policy to generate non-injurious completions. Their hope is to learn a general method for enforcing such constraints on models, such that they could then quickly train the model to, say, never mention anything about food.


I was pleasantly surprised at how understandable the paper was, and they do a good job of looking at exactly what their method is doing and how it accomplishes the attack in different ways under different settings.

Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks

Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, Tom Goldstein

Demonstrates a data poisoning attack in which the adversary gets to choose a poison input to add to the training set, but does _not_ get to choose its label. The goal is to misclassify a single test instance as a specific base class. They achieve this by creating a poison input that looks like the base class in pixel space but looks like the test instance in feature space (i.e. the activations in the penultimate layer). The poison input will be labeled by humans as the base class, and then when the network is retrained with the original dataset and the new poisoned input(s), it will classify the poison input as the base class, and with it the test instance as well (since they have very similar features).


Learning Robust Rewards with Adversarial Inverse Reinforcement Learning

GAIL and Guided Cost Learning (GCL) both have the idea of learning a policy and a reward function simultaneously, in a GAN-like way (where the generator is the policy, and the discriminator has to distinguish between trajectories from the policy and the expert trajectories, which can then be interpreted as a reward function). In GAIL, the discriminator can't be interpreted as a reward function, and you only get a policy as output (which is why it is called imitation learning). However, if you enforce that the discriminator has to be of the form:
D(τ) = exp(f(τ)) / (exp(f(τ)) + π(τ))
Then you can show that if π is trained to maximize
R(τ) = log(1 - D(τ)) - log(D(τ))
Then R and π converge to the optimal reward and policy respectively.

This trajectory-centric formulation is called GAN-GCL (the GAN version of guided cost learning). It's main issue is that it works with trajectories and so is hard to optimize -- the gradients are very high variance. So, instead, we can work with individual state-action pairs instead of trajectories, just replacing every τ in the equations above with (s, a). This makes it more sample-efficient, and in this case f converges to the advantage function of the optimal policy. However, the advantage function induces a heavily entangled reward function, which rewards the RL agent for doing the action that the expert would have taken, without actually understanding the goal that the expert had. We would like to learn a disentangled reward, which they define as a reward function that leads to the optimal policy according to the true reward function even if the transition dynamics change.

Intuitively, since entanglement happens by rewarding the agent for taking the same action as the expert, we can do better by enforcing that the reward function only be a function of the state, so that it is forced to learn the actual goal rather than memorizing the actions that are good. They prove two theorems under the condition that the true reward is only a function of the state. First, the learned optimal reward function is fully disentangled if it is a function of only the state, assuming that the transition dynamics are "decomposable". Second, the reward function must be a function of only the state if it is fully disentangled.

Now the discriminator in the formulation above is either looking at the trajectory as a whole, or looking at the current action in order to see whether or not you are matching the expert demonstrations. Clearly we can't just make the discriminator a function of only the current state -- there's no way that could distinguish between policies, since it has no access to information about the actions that the policies took. However, we can instead separate the discriminator's f function into the reward term and a shaping term, and enforce that the shaping term does not change the optimal policy:
f(s, a, s') = g(s) + γh(s') − h(s)
(It happens to be the case that for any function h, adding γh(s') − h(s) to the reward function does not change the optimal policy.) Now, the discriminator gets information about the action taken by the policy by seeing the next state s' that resulted. Since γh(s') − h(s) does not change the optimal policy, g(s) should converge to an optimal reward function, while h(s) must then be the value function V(s) in order to have f(s, a, s') be the advantage function.

They run a bunch of experiments with recovering a reward function and then transferring it to a situation with different dynamics, and show that it works much better than any other algorithm. They also show that for direct imitation (no transfer required), it does about as well as GAIL.


Generative Adversarial Imitation Learning

To do imitation learning, you could try to learn the state-action correspondence that defines the expert policy via behavioral cloning (simply learn a model of action given state using supervised learning), but that's brittle. Specifically, this is learning the policy over single timesteps, so any errors in one timestep lead to the next timestep being slightly out of distribution, leading to worse errors, giving compounding errors. So, you can instead use IRL to find the cost function, and use RL to then find a good policy. But IRL is slow. They propose a fast method that gets directly to the policy, but it's as if you did IRL followed by RL.

They write down IRL as a maximization of a minimization that gives you a cost function, and then RL as a minimization over policies of the expected cost. Specifically, maxent RL(c) computes a good policy π given a cost function c by minimizing expected cost E[c] and maximizing entropy H(π), where H(π) is computed according to causal entropy, not regular entropy (see MaxCausalEnt from [AN #12](https://mailchi.mp/bcb2c6f1d507/alignment-newsletter-12)). IRL(π) finds a cost function c such that the expert policy is maximally better than the RL policy.

Let's assume that we're working with all cost functions c : S → A → Reals. This is a huge space, so let's throw in a convex regularizer ψ for the IRL problem. Then, they prove that RL(IRL(π)) can be expressed as a single minimization involving ψ*, the convex conjugate of ψ. We can make different choices of ψ give us different imitation learning algorithms that minimize this objective.

Let ρ(s, a) be the occupancy measure of a policy π for a state-action pair (s, a). (Occupancy measure is the sum of the discounted probabilities of taking the given state-action pair.) Then, the expected cost of a trajectory is the sum of ρ(s, a) c(s, a) over all s, a. In the special case where you don't do any regularization, the occupancy measure of the solution is guaranteed to be equal to the occupancy measure of the expert policy. (In the language of convex optimization, IRL and maxent RL are dual to each other, and strong duality applies.)

Of course, this all assumes that you have access to the full expert policy, whereas in practice you only have access to some expert demonstrations. So now, the question is how to choose a good ψ that deals with that issue. If you don't regularize at all, you get something that matches the estimated occupancy measure exactly -- which is not great, since there are likely many areas of state space where there are not enough samples to accurately match the occupancy measure, at least in the case with stochastic dynamics. On the other hand, previous algorithms are basically using a ψ that is infinity on cost functions outside of a particular class, and constant elsewhere, which means that they only work if the cost function is actually in the class of learnable functions.

So, they instead propose a ψ that is dependent on the expert data. It requires that the cost function be negative everywhere, and puts a high penalty on any cost function that puts high cost (i.e. close to zero) on the expert data. Note that this can represent any (bounded) reward function, so it has very good expressive power. They chose the particular form they did because when you work through the convex conjugate, you end up choosing π to minimize the Jensen-Shannon divergence between the inferred policy and the expert policy, which can be interpreted as the optimal log loss of a classifier that distinguishes between the two policiees. This is very GAN-like, where we have a discriminator trying to distinguish between two policies, and a generator (the learned policy) that's trying to fool the discriminator. Their proposed algorithm, GAIL, follows this quite precisely, alternating between an Adam gradient step to update the discriminator, and a TRPO step to update the policy. (Both the discriminator and the policy are neural nets.)

In the experiments, they imitate classic control and MuJoCo tasks that have been trained with TRPO. GAIL consistently achieves the expert performance, even with not much data, though this comes at a cost of lots of environment interaction (which eg. behavioral cloning does not require).