It's quite remarkable how some supervision (poses in this case) can lead to such large improvements in the task. Of course, the promise of deep RL is to accomplish tasks with very little supervision (just a reward function), so this isn't a huge breakthrough, but it's still better than I expected. Intuitively, this works so well because the "reward" during the imitation phase is extremely dense -- the reference motion provides feedback after each action, so you don't have to solve the credit assignment problem.

Learning Acrobatics by Watching YouTube

Xue Bin (Jason) Peng and Angjoo Kanazawa

To imitate human behavior in videos, it is sufficient to estimate the human pose for each frame, to smooth the poses across frames to eliminate any jittery artifacts or mistakes made by the pose estimator, and then to train the robot to match the motion exactly. This results in really good performance that looks significantly better than corresponding deep RL approaches, but of course it relies on having labeled poses to train the pose estimator in addition to the simulator.


CEM-RL: Combining evolutionary and gradient-based methods for policy search


Generalization and Regularization in DQN

Jesse Farebrother, Marlos C. Machado, Michael Bowling


Thirty-Third AAAI Conference On Artificial Intelligence, 2019

Combined Reinforcement Learning via Abstract Representations

Vincent François-Lavet, Yoshua Bengio, Doina Precup, Joelle Pineau


Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning


ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay


SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning

Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, Sergey Levine


Introducing a New Framework for Flexible and Reproducible Reinforcement Learning Research

Pablo Samuel Castro and Marc G. Bellemare

Researchers at Google have released Dopamine, a small framework for RL research on Atari games, with four built-in agents -- DQN, C51, a simplified version of Rainbow, and the recent Implicit Quantile Network. There's a particular emphasis on reproducibility, by providing logs from training runs, training data, etc.


Dexterous Manipulation with Reinforcement Learning: Efficient, General, and Low-Cost

Henry Zhu, Abhishek Gupta, Vikash Kumar, Aravind Rajeswaran, and Sergey Levine


There are still some things that the bots do that seem like bad decisions. You can interpret this a few ways. Five could have learned a large number of heuristics that make it good enough to beat almost all humans, but that break down in edge cases. In this story, Five is not good at learning logical or abstract reasoning, but can compensate for that in the average case with the sheer number of heuristics it can learn. Another interpretation is that Five learns a good representation of Dota which lets it come up with new, novel insights into the game, which we can't see or understand because the representation is alien to us. However, the representation makes it harder to come up with other insights about Dota that we have using our representations of Dota, and as a result Five makes some mistakes that humans can easily recognize as mistakes. I lean towards the first interpretation, but not very strongly.

The International 2018: Results

Two human teams beat OpenAI Five at The International. The games seemed much more like regular Dota, probably because there was now only one vulnerable courier for items instead of five invulnerable ones. This meant that OpenAI Five's strategy of a relentless team attack on the enemy was no longer as powerful, because they couldn't get the health regeneration items they needed to constantly stay alive to continue the attack. It's also possible (but less likely to me) that the matches were more normal because the teams were more even, or because the human teams knew about Five's strategy this time and were countering it in ways that I don't understand.


Latent Space Policies for Hierarchical Reinforcement Learning

Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, Sergey Levine


Unsupervised learning seems like the way forward to learn rich models of the world, because of the sheer volume of data that you can use.

Learning Actionable Representations from Visual Observations

Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, Pierre Sermanet

Prior work on Time Contrastive Networks (TCN)s showed that you can use time as an unsupervised learning signal, in order to learn good embeddings of states that you can then use in other tasks. This paper extends TCNs to work with multiple frames, so that it can understand motion as well. Consider any two short videos of a task demonstration. If they were taken at different times, then they should be mapped to different embedding vectors (since they correspond to different "parts" of the task). On the other hand, if they were taken at the same time (even if from different viewpoints), they should be mapped to the same embedding vector. The loss function based on this encourages the network to learn an embedding for these short videos that is invariant to changes in perspective (which are very large changes in pixel-space), but _is_ different for changes in time (which may be very small changes in pixel-space). They evaluate with a bunch of different experiments.


ICML 2018 Notes


We're seeing more and more work combining classical symbolic approaches with the current wave of statistical machine learning from big data, that gives them the best of both worlds. While the results we see are not general intelligence, it's becoming less and less true that you can point to a broad swath of capabilities that AI cannot do yet. I wouldn't be surprised if a combination of symbolic and stastical AI techniques led to large capability gains in the next few years.

Learning Plannable Representations with Causal InfoGAN

Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart Russell, Pieter Abbeel

Hierarchical reinforcement learning aims to learn a hierarchy of actions that an agent can take, each implemented in terms of actions lower in the hierarchy, in order to get more efficient planning. Another way we can achieve this is to use a classical planning algorithm to find a sequence of _waypoints_, or states that the agent should reach that will allow it to reach its goal. These waypoints can be thought of as a high-level plan. You can then use standard RL algorithms to figure out how to go from one waypoint to the next. However, typical planning algorithms that can produce a sequence of waypoints require very structured state representations, that were designed by humans in the past. How can we learn them directly from data? This paper proposes Causal InfoGAN. They use a GAN where the generator creates adjacent waypoints in the sequence, while the discriminator tries to distinguish between waypoints from the generator and pairs of points sampled from the true environment. This incentivizes the generator to generate waypoints that are close to each other, so that we can use an RL algorithm to learn to go from one waypoint to the next. However, this only lets us generate adjacent waypoints. In order to use this to make a sequence of waypoints that gets from a start state to a goal state, we need to use some classical planning algorithm. In order to do that, we need to have a structured state representation. GANs do not do this by default. InfoGAN tries to make the latent representation in a GAN more meaningful by providing the generator with a "code" (a state in our case) and maximizing the mutual information of the code and the output of the generator. In this setting, we want to learn representations that are good for planning, so we want to encode information about _transitions_ between states. This leads to the Causal InfoGAN objective, where we provide the generator with a pair of abstract states (s, s'), have it generate a pair of observations (o, o') and maximize the mutual information between (s, s') and (o, o'), so that s and s' become good low-dimensional representations of o and o'. They show that Causal InfoGAN can create sequences of waypoints in a rope manipulation task, that previously had to be done manually.


For over a year now people keep asking me whether something like this is doable, since it seems like an obvious win combining PL and ML, and why no one has done it yet. I've mentioned the issue about neural nets being too slow, but it still seemed doable, and I was really tempted to do it myself. So I'm really excited that it's finally been done!

Oh right, AI alignment. Yeah, I do actually think this is somewhat relevant -- this sort of work could lead to much better theorem provers and formal reasoning, which could make it possible to create AI systems with formal guarantees. I'm not very optimistic about this approach myself, but I know others are.

Learning Heuristics for Automated Reasoning through Deep Reinforcement Learning

Gil Lederman, Markus N. Rabe, Edward A. Lee, Sanjit A. Seshia

The formal methods community uses SAT solvers all the time in order to solve complex search-based problems. These solvers use handtuned heuristics in order to drastically improve performance. The heuristics only affect the choices in the search process, not the correctness of the algorithm overall. Obviously, we should consider using neural nets to learn these heuristics instead. However, neural nets take a long time to run, and SAT solvers have to make these decisions very frequently, so it's unlikely to actually be helpful -- the neural net would have to be orders of magnitude better than existing heuristics. So, they instead do this for QBF (quantified boolean formulas) -- these are PSPACE complete, and the infrastructure needed to support the theory takes more time, so it's more likely that neural nets can actually help. They implement this using a graph neural network and engineer some simple features for variables and clauses. (Feature engineering is needed because there can hundreds of thousands of variables, so you can only have ~10 numbers to describe the variable.) It works well, doing better than the handcoded heuristics.


This is a cool example of a relatively simple yet powerful idea -- instead of having a goal space over all states, learn a good latent representation and use that as your goal space. This enables unsupervised learning in order to figure out how to use a robot to generally affect the world, probably similarly to how babies explore and learn.

Visual Reinforcement Learning with Imagined Goals

Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, Sergey Levine

[Hindsight Experience Replay](https://blog.openai.com/ingredients-for-robotics-research/) (HER) introduced the idea of accelerating learning with sparse rewards, by taking trajectories where you fail to achieve the goal (and so get no reward, and thus no learning signal) and replacing the actual goal with an "imagined" goal chosen in hindsight such that you actually achieved that goal, which means you get reward and can learn. This requires that you have a space of goals such that for any trajectory, you can come up with a goal such that the trajectory achieves that goal. In practice, this means that you are limited to tasks where the goals are of the form "reach this goal state". However, if your goal state is an image, it is very hard to learn how to act in order to reach any possible image goal state (even if you restrict to realistic ones), since the space is so large and unstructured. The authors propose to first learn a structured latent representation of the space of images using a variational autoencoder (VAE), and then use that structured latent space as the space of goals which can be achieved. They also use Q-learning instead of DDPG (which is what HER used), so that they can imagine any goal with a minibatch (s, a, s') and learn from it (whereas HER/DDPG is limited to states on the trajectory).


I wonder if they are planning to play a game where both sides draft heroes, or where both sides get a randomly chosen team of 5 heroes. Previously I would have expected that they were choosing randomly, since it seems very difficult to learn solely from experience whether your team choice works well, given that the number of possible drafts is combinatorially large, and the way that the draft affects outcome is very complicated and long term and so hard to capture in a gradient. Now, I'm pretty uncertain -- if deep RL was enough to get this far, it could be good enough to deal with that as well. And it's possible that you can actually do well at drafting with some relatively simple heuristics -- I don't know Dota well enough to say.

OpenAI Five Benchmark

The benchmark match for OpenAI Five will be a best-of-three match on August 5 at 2pm. They have already removed many of the restrictions on gameplay, including the two most important ones (wards and Roshan), as well as widening the pool of heroes to choose from 5 to 18.


The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach

Iulian Vlad Serban, Chinnadhurai Sankar, Michael Pieper, Joelle Pineau, Yoshua Bengio


It's worth highlighting that this can be seen as an extension to [Upside-Down RL](http://arxiv.org/abs/1912.02877) ([AN #83](https://mailchi.mp/ff565f097630/an-83-sample-efficient-deep-learning-with-remixmatch)). In that report, the goal is also to produce actions consistent with the desired reward. This paper extends that line of work by using transformers to expand context beyond the immediate state which aids in long-term credit assignment. The authors claim this helps via self-attention, but it seems much more likely that this effect comes from using the return-to-go as in Upside-Down RL.

Decision Transformer: Reinforcement Learning via Sequence Modeling

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch

In this paper, the authors abstract reinforcement learning (RL) as a sequence modeling problem. The authors are inspired by the rise of powerful sequence models (i.e transformers) in natural language processing. Specifically, they hypothesize that when models are trained to predict the expected reward-to-go alongside state and action sequences, the transformer architecture can be used to do RL.

As an example, consider finding the shortest path between two vertices on a graph. We could start by recording random walks with their expected returns. Once we have enough data, we could condition on paths such that the expected return-to-go (length remaining) is low. This would effectively return shortest paths without the explicit need for optimization.

This framework works well in practice and is competitive with state-of-the-art model-free offline RL baselines on Atari and OpenAI gym. The authors also carry out ablation studies to determine if the sequence modeler is just doing imitation learning on a subset of the data with high returns. This turns out not to be the case, indicating that the approach effectively uses the entire dataset.


It's fun to note that this paper came out of the same lab as Decision Transformer (the paper summarized above) only a day after. In contrast to Decision Transformer, this paper focuses more on the finer technical details of transformers by utilizing beam-search and studying the attentional patterns of the resulting models. The figures in this paper are informative. I feel I did learn something about *why* transformers seem to work in this domain. While the idea of goal-conditioned RL itself isn't novel, showing that 'off-the-shelf' transformers can do well in this domain is impressive.

Reinforcement Learning as One Big Sequence Modeling Problem

Michael Janner, Qiyang Li, Sergey Levine

Typically, RL is concerned with estimating policies that utilize immediate state information to produce high returns. However, we can also view RL as concerned with predicting sequences of actions that lead to high returns. From this perspective, it's natural to wonder if sequence modelers that work well in other domains, such as transformers in NLP, would work well for RL. This paper tests this hypothesis and demonstrates the utility of transformers in RL for a variety of problem settings.

As with the last paper, the authors train the model to predict the reward-to-go. In place of trajectory optimizers, the authors make use of beam search as a planning algorithm. To do RL, rather than maximize the log-probability of potential sequences, the authors replace the log-probability search heuristic with the reward-to-go. In experiments, transformers that maintain the log-probability can imitate expert policies to high fidelity. Visually, the resulting policies are indistinguishable from that of the expert.

The authors also show that their method is competitive on the standard OpenAI gym benchmarks. Finally, the authors look at the attention patterns of the trained models. They identify two patterns: the first links variables in a strictly Markovian fashion and the other links dimensions of the state-action variables across time. Interestingly, action variables are more strongly coupled to past actions than past state variables. This suggests a connection to action-smoothing proposed previously for deep-dynamics models.


The Pursuit of (Robotic) Happiness: How TRPO and PPO Stabilize Policy Gradient Methods

I barely looked at this -- I think it's an introduction to policy gradient methods for reinforcement learning. It assumes very little background (less than I assume in these summaries).


Thirty-Second AAAI Conference on Artificial Intelligence

Counterfactual Multi-Agent Policy Gradients

Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson


The Uncertainty Bellman Equation and Exploration

Brendan O'Donoghue, Ian Osband, Remi Munos, Volodymyr Mnih


Procedural Level Generation Improves Generality of Deep Reinforcement Learning

Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, Sebastian Risi


Ranked Reward: Enabling Self-Play Reinforcement Learning for Combinatorial Optimization

Alexandre Laterre, Yunguan Fu, Mohamed Khalil Jabri, Alain-Sam Cohen, David Kas, Karl Hajjar, Torbjorn S. Dahl, Amine Kerkeni, Karim Beguir


Retro Contest: Results

John Schulman, Vicki Pfau, Alex Nichol, Christopher Hesse, Oleg Klimov and Larissa Schiavo

OpenAI has announced the results of the [Retro Contest](https://blog.openai.com/retro-contest/). The winning submissions were modified versions of existing algorithms like joint PPO and Rainbow, without any Sonic-specific parts.


Annual Review of Control, Robotics, and Autonomous Systems 2019

A Tour of Reinforcement Learning: The View from Continuous Control


Evolving simple programs for playing Atari games

Dennis G Wilson, Sylvain Cussat-Blanc, Hervé Luga, Julian F Miller


DeepSpeed: Extreme-scale model training for everyone

DeepSpeed Team, Rangan Majumder, Junhua Wang

In this post, Microsoft announces updates to DeepSpeed, its open-source deep learning training optimization library. The new updates include:

 - '3D parallelism', a scheme for carefully optimizing how training runs are split across machines. Training runs that use 3D parallelism demonstrate linear scaling of GPU memory and compute efficiency, enabling the theoretical training of extremely large models of over a trillion parameters on as few as 800 NVIDIA V100 GPUs.
 - 'ZeRO-Offload', which allows CPU memory to be used during training runs, enabling running models of up to 13 billion parameters on a single NVIDIA V100 GPU.
 - 'DeepSpeed Sparse Attention', an instrumental technology that reduces the compute and memory requirements of attention computations used in models like Transformers. Compared to models that use densely computed attention, this enables models that pay attention to sequences that are 10x longer and can be trained up to 6.3x faster.
 - '1-bit Adam', a scheme for compressing the communication requirements between machines doing training runs that use the Adam gradient descent optimizer. 1-bit Adam enables up to 5x less communication and up to 3.5x faster training runs.


I really like the idea of successor features. Similar to model-based systems, they allow us to evaluate policies for many different rewards, which can be useful for anticipating problematic behaviour before deploying a system. However, note that we still need to execute the policy we obtained by generalized policy improvement to evaluate it for different rewards: The only guarantees we have is that it is better than the previous policies for the reward for which the improvement step was carried out (and potentially some weaker bounds based on the similarity of different rewards).

Fast reinforcement learning with generalized policy updates

Fast reinforcement learning through the composition of behaviours

André Barreto, Shaobo Hou, Diana Borsa, David Silver, Doina Precup

While model-based RL agents can easily adapt their policy to changed rewards on the same environment, planning is expensive and learning good models can be challenging for many tasks. On the other hand, it is challenging to get model-free agents to adapt their policy to a new reward without extensive retraining. An intermediate solution is to use so-called successor features: Instead of a value function **V(π,s)** representing the expected discounted reward for a policy **π** starting in state **s**, successor features are a vector-valued value function **ψ(π,s)** representing an expected discounted feature vector **ϕ**. If our reward equals **r = w ⋅ ϕ** for some weight vector **w**, we can easily obtain the original value function by taking the scalar product of the successor features and the weight vector: **V(π,s) = w ⋅ ψ(π,s)**. Successor features thus allow us to evaluate a fixed policy **π** for all rewards that are linear in **ϕ**, which is called *generalized policy evaluation*.

Now that we can evaluate policies for different preferences, we would like to efficiently find a good policy for a given novel preference. Inspired by human learning that often combines previously learned skills, we employ *generalized policy improvement*. In vanilla policy improvement, we improve upon a policy **π** we can evaluate by choosing the action that maximizes the immediate reward plus the discounted value **V(π,s')** of following **π** starting in the next state **s'**. In generalized policy improvement, we have multiple policies and choose the action that maximizes the reward plus the discounted value of following the best of these policies starting in the next state **s'**. To obtain a policy for the new preference, we "stitch together" all policies we learnt for previous preferences and the resulting policy performs at least as good as all of the old policies with respect to the new preference. As generalized policy improvement does not require any additional environment samples, it enables zero-shot transfer to new preferences. Empirically, even if the weight vector **w** has to be learnt from reward signals, generalized policy improvement is very sample efficient. Additional samples can then be used to further improve the policy using standard RL.


I am a bit surprised that this works so well, as both bootstrapping and learning generative models for distributions can be unstable and the method combines both. On the other hand, there is a long tradition of continuous interpolations between different RL algorithms and their performance at the sweet spot is often significantly stronger than at the extremes.

γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction

Michael Janner, Igor Mordatch, Sergey Levine

Long planning horizons are often necessary for competitive performance of model-based agents, but single-step models get less and less accurate with longer planning horizons as errors accumulate. Model-free algorithms don't have this problem but are usually reward- and policy-specific, such that transfer to other tasks can be hard. The paper proposes policy-specific γ-models as an intermediate solution: instead of learning the distribution of the next state given a state-action pair **(s,a)**, or the final state of an n-step rollout given **(s,a)** and a policy **π**, it learns the distribution of a rollout with a stochastic, geometrically distributed length. Unlike for n-step models with n>1, the distribution follows a Bellman-style decomposition into the single-step distribution and the discounted distribution for the next state **s'**, which allows for off-policy training of the model by bootstrapping the target distribution.

Now, if rewards are consequentialist in the sense that they only depend on the state, the expected reward under this distribution is equal to **1-γ** times the Q-value for  **π** of **(s,a)** such that we can use the model for policy evaluation given arbitrary consequentialist rewards. Similar to how single-step models (0-models) can be rolled out to obtain (less accurate) multi-step models, sequential rollouts of a γ-model can be reweighed to obtain a γ-model with larger **γ**. While this introduces some error, it reduces the bootstrap error during training, which grows with **γ**. Being able to interpolate between rollouts of single-step models that accumulate error during testing and models with large **γ** that accumulate error during training allows us to find a sweet spot between the two extremes.

In practice, single-step models are often used for model-based value expansion (MVE), where only **N** steps are rolled out and a value function is used for evaluating longer-term consequences. The authors' algorithm, γ-MVE instead uses **N** rollouts of the γ-model and adjusts the weighing of the value function accordingly. γ-MVE performs strongly both in terms of sample efficiency and final performance on a set of low-dimensional continuous control tasks.


Putting less weight on updating values with fluctuating targets seems like a good idea. As the approach does not require much additional compute if weights are shared for the **Q**-network and the network estimating the bound, and as it seems quite orthogonal to previous improvements to methods based on **Q**-functions, I would not be surprised if it became somewhat widely used.

Paper: DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction

Does On-Policy Data Collection Fix Errors in Off-Policy Reinforcement Learning?

Aviral Kumar, Abhishek Gupta, Sergey Levine

Q-learning finds the optimal **Q**-function **Q*** by updating our estimate **Q(s,a)** for a state-action pair **(s,a)** to get closer to the immediate reward plus the discounted **Q**-value for the best action **a'** in the next state **s'**. To generate samples, we usually pick actions corresponding to high **Q**-values. In bandit problems where **s'** is always terminal and thus has all **Q**-values at zero, this leads to **corrective feedback**: If we overestimated an actions value, we will pick this action again soon and are quickly able to correct our misconception. In general MDPs, corrective feedback can be a lot weaker as our update of **Q(s,a)** also depends on the **Q**-values for the next state: To get corrective feedback, we need somewhat correct **Q**-values for the next state, but to get these we likely needed good values for the second to next state, etc. This is particularly problematic with function approximation as updating the current state's **Q**-value might lead to a worse estimate for values down the chain. Consequently, we might see convergence to suboptimal **Q**-functions, instable learning, or problems with sparse or noisy rewards.

To deal with this, we would like to first prioritize correct estimates for states near the end of the chain. But in many branching problems, we actually observe these states with the least frequency such that their values are influenced disproportionally by other states' values when function approximation is used. The authors' approach, dubbed DisCor, reweighs the data distribution to account for this: We would like to preferentially sample states for which we expect **Q** to be close to **Q*** after the update and thus give more weight to state-action pairs when we expect the error **|Q*-Q|** to already be small. As we don't know **Q***, we rely on a bound for the error at a state-action pair **(s,a)** equal to the sum of the magnitudes of previous updates down the chain plus the initial error, discounted by the usual discount rate **γ** as we move back in time. Thus, the error in the next state one step ago is discounted by **γ**, the error in the second to next state two steps ago is discounted by **γ** squared and the initial error is discounted by **γ** to the **k**. This bound can be approximated by a neural network using a SARSA-like update rule, for which the influence of the unknown initial error fades for large **k** due to the discounting.

DisCor is evaluated on MetaWorld tasks in both the single and multi-task setting and SAC augmented with DisCor clearly outperforms SAC in many settings. Similar improvements can be observed for DQN on Atari.


RUDDER: Return Decomposition for Delayed Rewards

Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, Sepp Hochreiter


This is a paper worth a skim to glimpse at the complexity of today's RL research while noting how little we understand and can predict about the behaviour of our algorithms. A fun game to play here is to go through the graphs in Appendices D through K and arrive at one's own interpretations _before_ comparing them to the authors' in the main text. What was unsatisfying was that often there were contradictory results between environments, meaning there was no insight to be gleaned about what was happening: for instance, value function normalization always helps except in Walker2d where it significantly hurts performance. Such work raises more questions than it answers; perhaps it will motivate future research that fundamentally rethinks our environments and algorithms.

A more mundane, but alignment-relevant observation is that seeing how difficult it is to tune an agent for a task in simulation, and how much hyperparameters may vary across tasks, is weak evidence against powerful sim-to-real transfer performance arising out of the current paradigm of simulators/tasks and algorithms: agents will need to be trained in the real world, spawning associated risks which we may want to avoid.

What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, Olivier Bachem

In what is likely the largest study of on-policy reinforcement learning agents, this work unifies various algorithms into a collection of over 50 design choices, which the authors implement as tunable hyperparameters and systematically investigate how those parameters impact learning across five standard continuous control environments. Specifically, they choose subsets of these hyperparameters in eight experiment themes -- policy losses, network architectures, normalization and clipping, advantage estimation, training setup, timestep handling, optimizers, and regularization. They train thousands of agents for various choices within each theme, for a total of over 250,000 agents.

They present nearly a hundred graphs summarizing their experiments for the reader to make their own conclusions. Their own recommendations include: using the PPO loss, using separate value and policy networks, initializing the last policy layer with x100 smaller weights, using tanh as activation functions, using observation normalization, using generalized advantage estimation with \lambda = 0.9, tuning the number of transitions gathered in each training loop if possible, tuning the discount factor, using the Adam optimizer with a linearly decaying learning rate, among several others.


Getting a good learning signal can be a key challenge with RL. I’m somewhat surprised it took this long for HER to be generalized to arbitrary reward spaces -- it seems like a clear win that shouldn’t have taken too long to discover (though I didn’t think of it when I first read HER).

Generalized Hindsight for Reinforcement Learning

Alexander C. Li, Lerrel Pinto, Pieter Abbeel

[Hindsight Experience Replay](https://arxiv.org/abs/1707.01495) (HER) introduced the idea of _relabeling_ trajectories in order to provide more learning signal for the algorithm. Intuitively, if you stumble upon the kitchen while searching for the bedroom, you can’t learn much about the task of going to the bedroom, but you can learn a lot about the task of going to the kitchen. So even if the original task was to go to the bedroom, we can simply pretend that the trajectory got rewards as if the task was to go to the kitchen, and then update our kitchen-traversal policy using an off-policy algorithm.

HER was limited to goal-reaching tasks, in which a trajectory would be relabeled as attempting to reach the state at the end of the trajectory. What if we want to handle other kinds of goals? The key insight of this paper is that trajectory relabeling is effectively an inverse RL problem: we want to find the task or goal for which the given trajectory is (near-)optimal. This allows us to generalize hindsight to arbitrary spaces of reward functions.

This leads to a simple algorithm: given a set of N possible tasks, when we get a new trajectory, rank how well that trajectory does relative to past experience for each of the N possible tasks, and then relabel that trajectory with the task for which it is closest to optimal (relative to past experience). Experiments show that this is quite effective and can lead to significant gains in sample efficiency. They also experiment with other heuristics for relabeling trajectories, which are less accurate but more computationally efficient.


Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement

Benjamin Eysenbach*, Xinyang Geng*, Sergey Levine, Ruslan Salakhutdinov

This paper was published at about the same time as the previous one, and has the same key insight. There are three main differences with the previous paper:

1. It shows theoretically that MaxEnt IRL is the “optimal” (sort of) way to relabel data if you want to optimize the multitask MaxEnt RL objective.
2. In addition to using the relabeled data with an off-policy RL algorithm, it also uses the relabeled data with behavior cloning.
3. It focuses on fewer environments and only uses a single relabeling strategy (MaxEnt IRL relabeling).


What I learned from this paper is that estimating the mean Q-value is not always enough for robustness. By leveraging distributional information, via ensembles or quantiles, these methods can become quite effective at offline DQN. The release of the dataset is also impressive. I think the dataset will have broad applicability to researchers interested in offline RL as well as imitation learning.

An Optimistic Perspective on Offline Reinforcement Learning

Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi

Off-policy reinforcement learning (RL) that can be done using offline-logged interactions is an important aspect of real-world applications. However, most RL algorithms assume that an agent interacts with an online environment or simulator and learns from its own collected experience. Moreover, the authors show that DQN trained offline on its _own_ experience replay buffer has markedly decreased performance on most of the Atari suite. The authors attempt to address this discrepancy by introducing a robust Q-learning algorithm that randomly mixes estimates for particular Q-values. Specifically, by creating convex combinations from an underlying basis of Q-value estimates the authors are able to create a much larger ensemble. This is similar in spirit to dropout in deep learning where connections in the network are randomly turned off. The authors then go on to show that offline DQN is feasible by training this algorithm and other related algorithms on the DQN Replay Dataset and show it has comparable performance to, and occasionally even surpasses, the original RL baselines. The DQN Replay Dataset is released at [https://offline-rl.github.io/](https://offline-rl.github.io/).


The parallelism afforded by evolutionary methods and genetic algorithms might be valuable in an environment where weak compute is plentiful, so it's exciting to see evidence of such methods besting GPU-hungry deep neural networks. However, I wonder how this would do on sparse reward tasks, where the fitness function is almost always uninformative. Finally, while it generalises to settings where there are task-irrelevant distractions, its deliberately sharp self-attention likely leaves it vulnerable to even simple adversarial attacks.

Paper: Neuroevolution of Self-Interpretable Agents

Using Selective Attention in Reinforcement Learning Agents

Recently winning a best paper award at GECCO 2020, this work marks a leap forward in the performance capabilities learned by small agents via evolutionary methods. Specifically, it shows that by jointly learning which small fraction of input to attend to, agents with only thousands of free parameters can be trained by an evolutionary strategy to achieve state-of-the-art performance in vision-based control tasks.

The key pieces include self-attention over input patches, non-differentiable top-K patch selection that affect 'inattentional blindness', and training via CMA-ES. By design, the agent is interpretable as the top-K patches that are selected can be examined. Empirically, the agent has 1000x fewer weights than a competing neural architecture, and the method shows robustness to changes in task-irrelevant inputs, as the agent learns to focus only on task-relevant patches.


Methods like this that can benefit from seeing a lot of action-independent environment observations might be quite important for applying RL to the real world, as this type of data is a lot cheaper to generate. For example, we can easily generate a ton of observations from a factory by equipping workers with cameras, but state-action-next-state triples from a robot interacting with the factory are very costly to obtain.

Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, Rob Fergus

Sample efficiency in RL can be improved by using off-policy methods that can reuse the same sample multiple times and by using self-supervised auxiliary losses that help with representation learning, especially when rewards are sparse. This work combines both approaches by proposing to learn a latent state representation using an autoencoder while jontly training an agent on that latent representation using <@SAC@>(@Soft Actor-Critic: Deep Reinforcement Learning for Robotics@). Previous work in the on-policy case shows a positive effect from propagating Actor-Critic gradients through the encoder to improve the usefulness of the encoding for policy learning. However, this destabilizes training in the off-policy case, as changing the encoding to facilitate the actor also changes the Q-function estimate, which in turn changes the actor's goal and can introduce nonstationarity. This problem is circumvented by only propagating the Q-network's gradients through the encoder while blocking the actor's gradients.

The method strongly outperforms SAC trained on pixels. It also matches the previous state-of-the-art set by model-based approaches on an image-based continuous control task and outperforms them for noisy observations (as these make dynamics models hard to learn). The authors also find that the learnt encodings generalize between tasks to some extent and that reconstructing the true environment state is easier using their latent representation than using a representation obtained by training SAC on pixels directly.


This paper represented real progress in automated Diplomacy, but is still far from human-level. I’ll be pretty interested to see whether we can reach human-level by creating improved self-play algorithms, like the one presented in this paper, and the ones used for Poker and Go, or if we will have to wait for novel, more general reasoning algorithms applied to Diplomacy. Unlike Poker, Diplomacy against multiple human players involves collusion and implicit signalling, even with No Press. It seems possible to me that it is very difficult to become good at modeling those dynamics through self-play alone. If we did get to human-level through self-play, it would make me more optimistic about the extent to which training is likely to be a bottleneck in other domains which require sophisticated models of human behavior.

Learning to Play No-Press Diplomacy with Best Response Policy Iteration

Thomas Anthony*, Tom Eccles*, Andrea Tacchetti, János Kramár, Ian Gemp, Thomas C. Hudson, Nicolas Porcel, Marc Lanctot, Julien Pérolat, Richard Everett, Satinder Singh, Thore Graepel, Yoram Bachrach

Diplomacy is a game with simple rules where 7 players simultaneously move units every turn to capture territory. Units are evenly matched by default, so winning relies on getting support from some players against others. 'No-Press' Diplomacy limits communication between players to only orders submitted to units, removing the complex verbal negotiations that characterize traditional gameplay.

Previous state-of-the-art No-Press Diplomacy methods were trained to imitate human actions after collecting a dataset of 150,000 human Diplomacy games. This paper presents a new algorithmic method for playing No-Press Diplomacy using a policy iteration approach initialized with human imitation. To find better policies, their methods use "best response" calculations, where the best response policy for some player is the policy that maximizes the expected return for that player against opponent policies. Diplomacy is far too large for exact best response calculation, so the paper introduces an approximation, "Sampled Best Response", which
- Uses Monte-Carlo sampling to estimate opponents' actions each turn
- Only considers a small set of actions sampled from each candidate best response policy
- Only tries to make a single-turn improvement to its policy (rather than trying to optimize for the whole rest of the game)
Similar to other policy iteration methods, the paper creates a dataset of games every iteration using its Sampled Best Response method, then trains neural networks to create policy and value functions that predict the actions chosen by Sampled Best Response. To remedy issues where Sampled Best Response continually cycles through the best strategy for the last iteration,  the paper tries several variants of a technique called "Fictitious Play". In the best-performing variant, the policy network is trained to predict the latest Sampled Best Response given explicitly averaged _historical_ opponent and player policies, rather than just the latest policies.

The paper's methods outperform existing algorithmic methods for No-Press Diplomacy on a variety of metrics, but are still fairly few-shot _exploitable_-- at the end of training, the strongest (non-human) exploiter of the final policy wins 48% of the time. They also find that the strongest exploit doesn't change much through training, though few-shot exploitability does decrease from the beginning of training to the end.


Towards Learning Multi-agent Negotiations via Self-Play

While the previous paper introduces other-play to become robust to unknown partners, this paper takes the other approach of simply training an agent that is robust to a wide, diverse population of possible agents. In particular, it studies a self-driving car "zipper merge" environment, and trains an agent to be robust to a variety of rule-based agents, as well as past versions of itself, and finds that this leads to a much more successful merging policy. However, this is evaluated against the population it is trained with, and not against any previously unseen agents.


If you want to keep track of progress in deep RL, probably -- this seems quite likely to become the new set of benchmarks that researchers work on. There's also another example of specification gaming in the post.

Retro Contest

Christopher Hesse, John Schulman, Vicki Pfau, Alex Nichol, Oleg Klimov, and Larissa Schiavo

OpenAI has released Gym Retro, providing an interface to work with video games from SEGA Genesis, which are more complex than the ones from Atari. They want to use these environments to test transfer learning in particular, where the agent may be pretrained on initial levels for as long as desired, and then must learn how to complete a new test level with only 1 million timesteps (~18 hours) of gameplay. (Humans do well with 2 hours of pretraining and 1 hour of play on the test level.)


It is worth noting that the authors use the <@Dopamine@>(@Introducing a New Framework for Flexible and Reproducible Reinforcement Learning Research@) framework for evaluating the model-free baselines, meaning that a slightly stunted version of Rainbow is used on an evaluation protocol different from the original publication without retuning hyperparameters. That said, DreamerV2 definitely performs at a level similar to Rainbow, which is significant progress in model-based RL. In particular, the fact that the reward can be inferred from the world model even without gradients flowing back from the reward suggests transferability of the world models to different tasks with the same underlying dynamics.

Paper: Mastering Atari with Discrete World Models

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, Jimmy Ba

Model-based reinforcement learning can have better sample efficiency, allows for smarter exploration strategies, and facilitates generalization between different tasks. Still, previous attempts at model-based RL on the Atari Benchmark like <@Dreamer@>(@Dream to Control: Learning Behaviors by Latent Imagination@) and <@SimPLe@>(@Simulated Policy Learning in Video Models@) were unable to compete with model-free algorithms in terms of final performance. This paper presents DreamerV2, a model-based algorithm that outperforms DQN and its variants -- including Rainbow -- in terms of both median human- or gamer-normalized performance and on mean world-record normalized performance on Atari after 200M environment steps, achieving roughly 35% on the latter (25% if algorithm performance is clipped to max out at 100% for each game).

DreamerV2 learns a recurrent state-space model that stochastically encodes frames and a hidden state into a latent variable and uses the hidden state to predict the next value of the latent variable. Frames and reward are then reconstructed using both the hidden state and the latent variable. A policy is obtained by actor-critic training on the latent state space, leveraging parallelization to train on 468B imagined samples. As DreamerV2 does not use MCTS, it requires 8x less wall clock time to train than the more complicated but better performing <@MuZero Reanalyze@>(@Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model@).  Unlike earlier approaches, DreamerV2 uses a vector of categorical latent variables rather than gaussians to enable better model predictions for dynamics with multiple distinct modes, as well as KL-balancing (scaling up the importance of the transition loss compared to the entropy regularizer on the latent variable). Ablations confirm that the image reconstruction loss is crucial for DreamerV2's performance and that both the use of discrete latent variables and KL-balancing lead to significant improvements. Interestingly, preventing the gradients for reward prediction from affecting the world model does not affect performance at all.


A common intuition that many researchers have is that specifying _what_ to do (the reward function) should be easier than specifying _how_ to do it (the policy). In practice, this _doesn't_ seem to be the case for deep learning, where imitation via inverse reinforcement learning (inferring a reward function and optimizing it) seems to be similar to imitation learning via behavior cloning ("copying" the policy). Similarly, this method seems broadly similar to meta learning algorithms like MAML and RL^2, though it does outperform them on one (probably carefully designed) transfer learning task.

What Can Learned Intrinsic Rewards Capture?

Zeyu Zheng*, Junhyuk Oh*, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado van Hasselt, David Silver, Satinder Singh

This paper studies whether a learned reward function can serve as a locus of knowledge about the environment, that can be used to accelerate training of new agents. In particular, such a learned intrinsic reward can help with test-time adaptation: in a novel environment, the intrinsic reward can quickly "tell" the agent e.g. where it should explore -- even if in the new environment the agent has a different action space, or uses a different learning algorithm (situations that meta learning would typically not be able to handle).

The authors create an algorithm that learns an intrinsic reward function, that when used to train a new agent over a “lifetime” (which consists of multiple episodes), leads to the best cumulative reward over the lifetime, using a meta-gradient approach. Experiments on gridworlds demonstrate that these learned intrinsic rewards: 1. switch between early exploration and later exploitation, 2. explore only for information that is relevant for optimal behavior, 3. capture invariant causal relationships, and 4. can anticipate and adapt to changes in the extrinsic reward within a lifetime.


At first glance, it would seem as though the idea of using an off-policy method to combine imitation and reinforcement learning is obvious. However, the implementation is complicated by the fact that we want the value functions being estimated by our agent to satisfy the optimality condition for the Bellman equation. Prior work, such as [Hester et al. 2018](https://aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16976/16682) uses n-step returns to help pre-training and make use of on-policy methods when performing RL. What I like about this paper is that they perform an ablation study and show that simple sequencing of imitation learning and RL algorithms isn't enough to get good performance. This means that combining the imitation and reinforcement objectives into a single loss function is providing a significant improvement over other methods.

Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Sparse Reward Environments

Vinicius G. Goecks, Gregory M. Gremillion, Vernon J. Lawhern, John Valasek, Nicholas R. Waytowich

This paper contributes to the effort of combining imitation and reinforcement learning to train agents more efficiently. The current difficulty in this area is that imitation and reinforcement learning proceed under rather different objectives which presents a significant challenge to updating a policy learned from a pure demonstration. A major portion of this difficulty stems from the use of so-called "on-policy" methods for training which require a significant number of environment interactions to be effective. In this paper, the authors propose a framework dubbed "Cycle-of-Learning" (CoL) that allows for the off-policy combination of imitation and reinforcement learning. This allows the two approaches to be combined much more directly which grounds the agent's policy in the expert demonstrations while simultaneously allowing for RL to fine-tune the policy. The authors show that CoL is an improvement over the current state of the art by testing their algorithm in several environments and performing an ablation study.


On net, I am quite happy to see work which argues in favour of reducing time spent hand-tuning and hand-crafting parts of a complex pipeline, and demonstrates the alternatives that currently exist to do so.

However, I feel the work did not fully compare the trade-off between tuning hyperparameters, and increasing the complexity of the pipeline by adding the adaptive components. I agree, though, that the latter is a one-time effort (per inductive bias), and is thus far more scalable than the former which needs to be repeated for each bias for every new task.

It would also be interesting to see how adaptive agents fare on problems where we care more about failures than successes, or if they are better/worse suited for safe exploration than baseline agents. My intuition is that adaptive internals of the agent cause it behave more noisily/unpredictably, and it may not fare as well as our current efforts for such problems.

**Rohin's opinion:** While it's certainly true that fewer inductive biases imply more general agents, it also usually means more compute and data requirements. For action repetition and learned discount factors, only one new parameter has to be learned, so it doesn't make much of a difference either way (and in fact performance on Atari doesn't change much). Clipped rewards do in fact learn faster than PopArt. I don't know why a recurrent network improves upon standard observation preprocessing for Atari -- perhaps initially RNNs were hard to train, and it became a de facto standard to use observation preprocessing, and no one checked about using recurrent networks later when RNNs became easier to train?

On Inductive Biases in Deep Reinforcement Learning

Matteo Hessel*, Hado van Hasselt*, Joseph Modayil, David Silver

The fewer inductive biases we use, the more general our algorithms will be. But how much does it really help to have fewer inductive biases? This paper replaces several hand-engineered components of an A2C agent with generic or adaptive variants.

Specifically, they compared: 1) reward clipping vs. reward normalization via <@PopArt@>(@Preserving Outputs Precisely while Adaptively Rescaling Targets@), 2) handpicked discount factor vs. online adaptive discounting via meta-learning, 3) fixed action repeats vs. learned action-commitment, and 4) standard Atari observation preprocessing vs. passing raw observations to a recurrent network. Over 57 Atari tasks, they found that the tuned algorithm outperformed the adaptive method only in (1). Performance was similar for (2) and (3), and the proposed method outperformed the baseline for (4). When the fully adaptive agent was compared to the vanilla agent (with heuristics designed for Atari) over 28 unseen continuous control tasks, the adaptive agent performed better in 14 of them, worse in one, and about the same in the rest, providing evidence that fewer inductive biases do lead to more general agents.


Investigating Generalisation in Continuous Deep Reinforcement Learning

Chenyang Zhao, Olivier Sigaud, Freek Stulp, Timothy M. Hospedales


Quantifying Generalization in Reinforcement Learning


Modular Architecture for StarCraft II with Deep Reinforcement Learning

Dennis Lee, Haoran Tang, Jeffrey O Zhang, Huazhe Xu, Trevor Darrell, Pieter Abbeel


While the work in this paper is solid, the conclusions don't seem particularly surprising: everyone knows that deep RL is incredibly sample intensive (which straightforwardly implies inaccurate gradient estimates) and relies on many implementation tricks. I'm not familiar enough with PPO to know how surprising their last result is.

Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms?

Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry

This paper argues that policy gradient algorithms are very dependent on additional optimisations (such as value function clipping, reward scaling, etc), and that they operate with poor estimates of the gradient. It also demonstrates that the PPO objective is unable to enforce a trust region, and that the algorithm's empirical success at doing so is due to the additional optimisations.


Learning Shared Dynamics with Meta-World Models


I agree that having standard benchmarks is often useful for spurring progress in deep learning, and that this one will be useful. I'm somewhat concerned that the tasks the authors have selected (CartPole, HalfCheetah, etc) are too simple, and that the property they're measuring is more like robustness to peturbations than the sort of combinatorial generalisation discussed in [this paper] (http://arxiv.org/abs/1806.01261) from [last week's newsletter](https://mailchi.mp/c1f376f3a12e/alignment-newsletter-30). The paper would benefit from more clarity about what they mean by "generalisation".

Assessing Generalization in Deep Reinforcement Learning

Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, Dawn Song

This paper aims to create a benchmark for measuring generalisation in reinforcement learning. They evaluate a range of standard model-free algorithms on OpenAI Gym and Roboschool environments; the extent of generalisation is measured by varying environmental parameters at test time (note that these tasks are intended for algorithms which do not update at test time, unlike many transfer and multi-task learners). They distinguish between two forms of generalisation: interpolation (between values seen during training) and extrapolation (beyond them). The latter, which is typically much harder for neural networks, is measured by setting environmental parameters to more extreme values in testing than in training.


While I don’t think this paper is particularly novel, it usefully combines several ideas and provides easily-interpretable results.

CURIOUS: Intrinsically Motivated Multi-Task, Multi-Goal Reinforcement Learning

Cédric Colas, Pierre Fournier, Olivier Sigaud, Mohamed Chetouani, Pierre-Yves Oudeyer

This paper presents an intrinsically-motivated algorithm (an extension of Universal Value Function Approximators) which learns to complete multiple tasks, each parameterised by multiple “goals” (e.g. the locations of targets). It prioritises replays of tasks which are neither too easy nor too hard, but instead allow maximal learning progress; this also help prevent catastrophic forgetting by refocusing on tasks which it begins to forget.


Reinforcement Learning for Improving Agent Design

This paper explores what happens when you allow an RL agent to modify aspects of the environment; in this case, the agent's body. This allows you to learn asymmetric body designs that are better suited for the task at hand. There's another fun example of specification gaming -- the agent makes its legs so long that it simply falls forward to reach the goal.


Learning and Planning with a Semantic Model

Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, Yuandong Tian


Introducing Holodeck


I find Starcraft more interesting as a test-bed for deep RL than as a goal in itself. While the results of this paper are cool,  I doubt that its methods will scale well - in general, approaches which rely on a lot of human knowledge being hard-coded in don't tend to.

Note the similarity between the macros in this paper and the way that OpenAI Five could choose between different hand-coded combinations of items. However, the latter is only a small part of the game, whereas the former is much more extensive.

TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game

Peng Sun, Xinghai Sun, Lei Han, Jiechao Xiong, Qing Wang, Bo Li, Yang Zheng, Ji Liu, Yongsheng Liu, Han Liu, Tong Zhang

This paper showcases an RL agent which is able to defeat the built-in Starcraft AI (roughly at the level of the 30th-50th percentile of players). It do so by choosing between 165 hand-coded macro actions, which each correspond to an elementary task like producing a certain building. This avoids the necessity of learning unimportant details like exactly where the building should be placed, as well as difficult rules like the prerequisites for each building. The authors create a second agent which uses an expert system to choose actions in a hierarchical fashion, which performs at a similar level to the first. See also [Import AI](https://jack-clark.net/2018/09/25/import-ai-113-why-satellitesai-gives-us-a-global-eye-industry-pays-academia-to-say-sorry-for-strip-mining-it-and-kindred-researchers-seek-robot-standardization/).


Hierarchical Deep Multiagent Reinforcement Learning

Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Changjie Fan, Li Wang


On Reinforcement Learning for Full-length Game of StarCraft

Zhen-Jia Pang, Ruo-Ze Liu, Zhou-Yu Meng, Yi Zhang, Yang Yu, Tong Lu


Zero-shot Sim-to-Real Transfer with Modular Priors

Robert Lee, Serena Mou, Vibhavari Dasagi, Jake Bruce, Jürgen Leitner, Niko Sünderhauf


The two properties that this paper highlights do seem important, and the fact that they can be varied in Space Fortress makes it a good benchmark for them.

I'm not convinced that the experimental work is particularly useful, though. It seems to reinforce the well-known point that shaped rewards can work well when they're shaped in sensible ways, and much less well otherwise.

Challenges of Context and Time in Reinforcement Learning: Introducing Space Fortress as a Benchmark

The authors note that most existing RL benchmarks (like Atari games) lack sharp context-dependence and temporal sensitivity. The former requires an agent to sometimes change strategies abruptly; the latter requires an agent's strategy to vary over time. Space Fortress is an arcade-style game which does have these properties, and which cannot be solved by standard RL algorithms, even when rewards are made dense in a naive way. However, when the authors shape the rewards to highlight the context changes, their agent achieves superhuman performance.


Automata Guided Reinforcement Learning With Demonstrations


Sim-to-Real Transfer Learning using Robustified Controllers in Robotic Tasks involving Complex Dynamics

Jeroen van Baar, Alan Sullivan, Radu Cordorel, Devesh Jha, Diego Romeres, Daniel Nikovski


Keep it stupid simple

Erik J Peterson, Necati Alp Müyesser, Timothy Verstynen, Kyle Dunovan


Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning

Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J. Mankowitz, Shie Mannor


World Models


Active Reinforcement Learning with Monte-Carlo Tree Search


Hierarchical Reinforcement Learning: Approximating Optimal Discounted TSP Using Local Policies


Learning Memory Access Patterns


Feedback-Based Tree Search for Reinforcement Learning

Daniel R. Jiang, Emmanuel Ekwedike, Han Liu


Episodic Memory Deep Q-Networks

Zichuan Lin, Tianqi Zhao, Guangwen Yang, Lintao Zhang


Learning Real-World Robot Policies by Dreaming

AJ Piergiovanni, Alan Wu, Michael S. Ryoo


On Learning Intrinsic Rewards for Policy Gradient Methods


USC CLASS Research Papers Series No. CLASS18-10

Improving On-policy Learning with Statistical Reward Accumulation

Yubin Deng, Ke Yu, Dahua Lin, Xiaoou Tang, Chen Change Loy


ViZDoom Competitions: Playing Doom from Pixels

Marek Wydmuch, Michał Kempka, Wojciech Jaśkowski


ExpIt-OOS: Towards Learning from Planning in Imperfect Information Games


Remember and Forget for Experience Replay


Accuracy-based Curriculum Learning in Deep Reinforcement Learning

Pierre Fournier, Olivier Sigaud, Mohamed Chetouani, Pierre-Yves Oudeyer


Adversarial Exploration Strategy for Self-Supervised Imitation Learning

Zhang-Wei Hong, Tsu-Jui Fu, Tzu-Yun Shann, Yi-Hsiang Chang, Chun-Yi Lee


Self-Imitation Learning

Junhyuk Oh, Yijie Guo, Satinder Singh, Honglak Lee


Learning to navigate in cities without a map

Piotr Mirowski, Raia Hadsell and Andrew Zisserman


This paper feels incomplete; it has very little discussion of the reasons why meta-learning and adversarial training failed, and the data augmentation result is rather perfunctory.

Learning Invariances for Policy Generalization

Remi Tachet des Combes, Philip Bachman, Harm van Seijen

This paper compares three ways to induce generalisation in deep RL agents: data augmentation, meta-learning and adversarial training. They find that data augmentation boosts performance on a simple task from 2.8% to 99.8%, whereas the other two don't improve on the baseline.


The use of embeddings in OpenAI Five

This blog post discusses some information which has been released by OpenAI about the structure of the OpenAI Five networks. Starting from Dota API information, each unit, ability, item and modifier is converted to an embedding. During processing, there's some surprising use of max-pooling over those embeddings even when it doesn't seem appropriate. Actions are chosen depending on the dot product similarity between their embedding and the final LSTM output. In a previous version, the target for the chosen action was picked without conditioning on what the action actually was; this has changed in more recent versions.


If you find my summary thought-provoking, then yes.

[Beyond algorithmic equivalence: algorithmic noise](https://www.lesserwrong.com/posts/meG3Pai2YeRYcwPwS/beyond-algorithmic-equivalence-rewards) and [Beyond algorithmic equivalence: self-modelling](https://www.lesserwrong.com/posts/kmLP3bTnBhc22DnqY/beyond-algorithmic-equivalence-self-modelling)

Given human behavior, it is hard to figure out whether the behavior is generated because it is good for the human's values, or because the human is biased. One way to tell whether or not it is a bias is to look at the decision process going on inside the human's brain and see whether or not the human would remove the bias if possible.


The approach seems very empirically-minded and not theoretically justified, which makes me worry that we would need to run a lot of experiments to actually get this into a form that doesn't fail immediately. That said, we do need some way of dealing with human inconsistencies, and I think the ideas behind the framework (such as endorsements) are intriguing and useful for future work on this problem.

Resolving human values, completely and adequately

IRL and other value learning approaches make the assumption that the human has a consistent reward function, whereas in reality human values are "contradictory, underdefined, changeable, and manipulable". How can we use the output from algorithms like IRL given this? He proposes a framework where humans can have _values_ that can "endorse" rewards and other values, and defines a process that can take these endorsements and compute an overall reward function to optimize. The hope here is for the overall reward to be adequate, not perfect.


If you found my example interesting and worth thinking about, it is worth reading the post to see an attempt at formalizing it.

Why we want unbiased learning processes

Suppose we have a reward learning agent where we have designed the reward space so that "ask the human whether to do X" always has higher reward than "do X". The agent is now considering whether to ask the human to try heroin, or just give them heroin. If the agent gives them heroin, it will see their look of ecstasy and will update to have the reward function "5 for giving the human heroin, 7 for asking the human". If the agent asks the human, then the human will say "no", and the agent will update to have the reward function "-1 for giving the human heroin, 1 for asking the human". In both cases asking the human is the optimal action, yet the agent will end up giving the human heroin, since that gets reward 5, while asking the human gets reward 1. The post doesn't have this example, it has a formal model and an abstract example in that model that is isomorphic to the example I have.


A theory of human values

This post presents an outline of how to construct a theory of human values. First, we need to infer preferences and meta-preferences from humans who are in "reasonable" situations. Then we need to synthesize these into a utility function, by resolving contradictions between preferences, applying meta-preferences to preferences, and having a way of changing the procedures used to do the previous two things. We then need to argue that this leads to adequate outcomes -- he gives some simple arguments for this, that rely on particular facts about humans (such as the fact that they are scope insensitive).


Hierarchical system preferences and subagent preferences

Often, when looking at a hierarchical system, you can ascribe preferences to the system as a whole, as well as to individual parts or subagents of the system. Often, any divergence between the parts can be either interpreted as a difference between the goals of the parts, or a failure of rationality of one of the parts. The post gives a particular algorithm, and shows how based on the structure of the code of the subagent, we could either infer that the subagent is mistaken, or that it has different goals.

Now, we could infer meta-preferences by seeing how the system tends to self-modify -- perhaps we notice that it tends to amplify the "goals" of one particular subagent, in which case we can infer that it has a meta-preference for those goals. But without that, there's no correct answer to what the true goals are. In the post's own words: "In the absence of some sort of meta-preferences, there are multiple ways of establishing the preferences of a hierarchical system, and many of them are equally valid."


I like this theoretical analysis that shows what could go wrong with processes that learn preferences. I did find it a bit hard to connect the ideas in this post with concrete reward learning algorithms (such as inverse reinforcement learning) -- it seems plausible to me that if I properly understood what the formal definitions of unriggable and uninfluenceable meant in the IRL setting, I wouldn't view them as desirable.

[Reward function learning: the value function](https://www.lesswrong.com/posts/55hJDq5y7Dv3S4h49/reward-function-learning-the-value-function) and [Reward function learning: the learning process](https://www.lesswrong.com/posts/upLot6eG8cbXdKiFS/reward-function-learning-the-learning-process)

These posts introduce a theoretical framework for reward learning, where a reward learning algorithm is modeled as something that produces a probability distribution over reward functions given a history and current policy. With such a general notion of reward learning, it becomes hard to define the value function -- while we still want something like sum of expected rewards, it is no longer clear how to take an expectation over the reward function, given that the distribution over it can change over time. Most plausible ways of doing this lead to time-inconsistent decisions, but one works well. The second post turns to the learning process and analyzes properties that it would be nice to have. In the worst case, we can get quite pathological behavior, but of course we get to choose the learning algorithm so we can avoid worst-case behavior. In general, we would want our learning algorithm to be _unriggable_ and/or _uninfluenceable_, but this is not possible when learning from humans since different policies on the AI's part will lead to it learning different rewards.


The fact that humans are quite good at inferring preferences should give us optimism about value learning. In the [framework](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/h9DesGT3WT9u2k7Hr) of rationality with a mistake model, we are trying to infer the mistake model from the way that humans infer preferences about other humans. This sidesteps the impossibility result by focusing on the _structure_ of the algorithm that generates the policy. However, it still seems like we have to make some assumption about how the structure of the algorithm leads to a mistake model, or a model for what values are. Though perhaps we can get an answer that is principled enough or intuitive enough that we believe it.

Figuring out what Alice wants: non-human Alice

We know that if we have a potentially irrational agent, then inferring their preferences is [impossible](https://arxiv.org/abs/1712.05812) without further assumptions. However, in practice we can infer preferences of humans quite well. This is because we have very specific and narrow models of how humans work: we tend to agree on our judgments of whether someone is angry, and what anger implies about their preferences. This is exactly what the theorem is meant to prohibit, which means that humans are making some strong assumptions about other humans. As a result, we can hope to solve the value learning problem by figuring out what assumptions humans are already making and using those assumptions.


By default, avoid ambiguous distant situations


There's one possible view where you look at the impossibility result around inferring preferences, and think that value alignment is hopeless. I don't subscribe to this view, for basically the reasons given in this post -- while you can't infer preferences for arbitrary agents, it certainly seems possible for humans in particular.

That said, I would expect that we accomplish this by learning a model that implicitly knows how to think about human preferences, rather than by explicitly constructing particular normative assumptions that we think will lead to good behavior. Explicit assumptions will inevitably be [misspecified](https://www.alignmentforum.org/posts/gnvrixhDfG7S2TpNL/latent-variables-and-model-mis-specification) ([AN #32](https://mailchi.mp/8f5d302499be/alignment-newsletter-32)), which is fine if we can correct the misspecification in the future, but at least under the threat model of an AI system that prevents us from changing its utility function (which I believe is the threat model Stuart usually considers) this isn't an option available to us.

["Normative assumptions" need not be complex](https://www.alignmentforum.org/posts/QzsCrzGd4zkNwk9cd/normative-assumptions-need-not-be-complex), [Humans interpreting humans](https://www.alignmentforum.org/posts/eXiuk8Yjd7FsunhxM/humans-interpreting-humans) and [Anchoring vs Taste: a model](https://www.alignmentforum.org/posts/7gMkkEMQa78h8Bpmr/anchoring-vs-taste-a-model)

We have seen before that since humans are not perfectly rational, it is [impossible](https://www.alignmentforum.org/posts/ANupXf8XfZo2EJxGv/humans-can-be-assigned-any-values-whatsoever) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)) to deduce their preferences, even with a simplicity prior, without any additional assumptions. This post makes the point that those assumptions need not be complex -- for example, if we could look at the "source code" of an agent, and we can find one major part with the same type signature as a reward function, and another major part with the type signature of a planner, then we can output the first part as the reward function. This won't work on humans, but we can hope that a similarly simple assumption that bakes in a lot of knowledge about humans could allow us to infer human preferences.

Since we seem to be very capable of inferring preferences of other humans, we might want to replicate our normative assumptions. The key idea is that we model ourselves and others in very similar ways. So, we could assume that if H is a human and G another human, then G's models of H's preferences and rationality are informative of H's preferences and rationality.

Stuart then shows how we could apply this to distinguish between choices made as a result of anchoring bias vs. actual taste preferences. Suppose that in condition 1, our human H would pay $1 or $3 for the same bar of chocolate depending on whether they were anchored on $0.01 or $100, and in condition 2 they would pay $1 or $3 depending on whether the chocolate has nuts. Ideally, we'd call the first case a bias, and the second one a preference. But in both cases, H's choice was determined by access to some information, so how can we distinguish between them? If we have access to H's internal model, we might expect that in the nuts case the information about nuts passes through a world model that then passes it on to a reward evaluator, whereas in the anchoring case the world model throws the information away, but it still affects the reward evaluator through a side channel. So we could add the normative assumption that only information that goes through the world model can be part of preferences. Of course, we could imagine another agent where the anchoring information goes through the world model and the nuts goes through the side channel -- but this agent is not human-like.


I'm not quite sure what overall point Stuart is trying to make. However, if we're concerned that an agent might manipulate humans, I don't see why we should trust it to aggregate the data from many one-step hypotheticals, since "manipulation" could then occur using the many degrees of freedom involved in choosing the questions and interpreting the answers.

[One-step hypothetical preferences](https://www.alignmentforum.org/posts/i6hWWcKyxBPj7ELT6/one-step-hypothetical-preferences) and [A small example of one-step hypotheticals](https://www.lesswrong.com/posts/zo5K8QeDZiLicSCe6/a-small-example-of-one-step-hypotheticals)

We don't hold most of our preferences in mind at any given time - rather, they need to be elicited from us by prompting us to think about them. However, a detailed prompt could be used to manipulate the resulting judgement. In this post, Stuart discusses hypothetical interventions which are short enough to avoid this problem, while still causing a human to pass judgement on some aspect of their existing model of the world - for example, being asked a brief question, or seeing something on a TV show. He defines a one-step hypothetical, by contrast, as a prompt which causes the human to reflect on a new issue that they hadn't considered before. While this data will be fairly noisy, he claims that there will still be useful information to be gained from it.


If you previously were optimistic about inverse reinforcement learning and similar techniques because you thought they could infer the same notion of value that the brain computes, this seems like an important counterargument. However, it should be noted that the authors are _not_ arguing that the brain is not optimizing some specific notion of value: just that it is not _explicitly computing_ such a notion of value. (Similarly, policies learned by RL optimize the reward function, even though they need not explicitly calculate the reward of every state in order to choose an action.) So you could still hope that the brain is optimizing some notion of value that isn’t explicitly computed, and then use inverse RL to recover that notion of value.

The case against economic values in the brain

It has been common in the neuroeconomics literature to assume (based on past research) that the brain explicitly computes some notion of value in order to make choices. This paper argues that this is wrong: it is plausible that the brain does not in fact explicitly calculate values, and instead directly learns a policy that produces actions.


AI Alignment Problem: “Human Values” don’t Actually Exist


Rigging is a form of wireheading

Ideally in reward learning, we want our AI to be learning a fact about the world -- which reward it should be optimizing. However, for most proposals of reward learning, the AI's actions can also influence this "fact" about the world. In this case, the AI can wirehead by influencing the world so that it learns an easy-to-maximize reward. This is what Stuart calls "rigging" of the learning process.


Utility versus Reward function: partial equivalence


It does seem like we should be able to learn the things that humans mostly agree on, and that this can help us a lot with inferring human preferences. I don't know if the goal is to use these models to infer broad human values, or something a lot simpler. Broad human values seems very unlikely to work, since you are trying to get to superhuman ability at knowing what humans want by mimicking human models (which are tautologically not superhuman).

Figuring out what Alice wants, parts [I](https://www.lesswrong.com/posts/so78tQvTCaBgw4bfL/figuring-out-what-alice-wants-part-i) and [II](https://www.lesswrong.com/posts/rcXaY3FgoobMkH2jc/figuring-out-what-alice-wants-part-ii)

Since it's not possible to infer human preferences without making some normative assumption about the human, we should try to learn the models that humans use to reason about each other that allow us to infer preferences of other humans. While we can't get access to these models directly, we can access fragments of them -- for example, whenever a person expresses regret, that can be taken as an mismatch between the model expectation and actual outcome. Part II goes through two example scenarios and what the internal human models might look like, and the challenges that arise in trying to learn them.


Defining the ways human values are messy


Some in the safety community have speculated that robustness to data shift (sometimes called "transfer learning" in the safety community) cannot be resolved only by leveraging more GPUs and more data. Also, it is argued that the difficulty in attaining data shift robustness suggests longer timelines. Both this paper and [Robustness properties of Facebook's ResNeXt WSL models](https://arxiv.org/abs/1907.07640) analyze the robustness of models trained on over 100 million to 1 billion images, rather than only training on ImageNet-1K's ~1 million images. Both papers show that data shift robustness greatly improves with more data, so data shift robustness appears more tractable with deep learning. These papers evaluate robustness using benchmarks collaborators and I created; they use [ImageNet-A](https://arxiv.org/abs/1907.07174), [ImageNet-C](https://arxiv.org/abs/1903.12261), and [ImageNet-P](https://arxiv.org/abs/1903.12261) to show that performance tremendously improves by simply training on more data. See [Figure 2](https://arxiv.org/pdf/1911.04252.pdf#page=6&zoom=100,0,88) of the Noisy Student paper for a summary of these three benchmarks. Both the Noisy Student and Facebook ResNeXt papers have problems. For example, the Noisy Student paper trains with a few expressly forbidden data augmentations which overlap with the ImageNet-C test set, so performance is somewhat inflated. Meanwhile, the Facebook ResNeXt paper shows that more data does not help on ImageNet-A, but this is because they computed the numbers incorrectly; I personally verified Facebook's ResNeXts and more data brings the ImageNet-A accuracy up to 60%, though this is still far below the 95%+ ceiling. Since [adversarial robustness can transfer to other tasks](https://arxiv.org/pdf/1901.09960.pdf), I would be surprised if robustness from these models could not transfer. These results suggest data shift robustness can be attained within the current paradigm, and that attaining image classifier robustness will not require a long timeline.

Self-training with Noisy Student improves ImageNet classification

Qizhe Xie, Eduard Hovy, Minh-Thang Luong, Quoc V. Le

Instead of summarizing this paper, I'll provide an opinion describing the implications of this and other recent papers.


I am quite excited about the focus on preventing low likelihood catastrophic events, particularly from the standpoint of existential risk reduction. The key assumption in this paper, that earlier in training the agent will fail in related ways but more frequently, seems plausible to me and in line with most of my experience training neural networks, and the experiments demonstrate a very large increase in efficiency.

I’d be interested to see theoretical analysis of what situations would make this assumption more or less likely in the context of more powerful future agents. For example, one situation where the failure modes might be distinct later in training is if an agent learns how to turn on a car, which then makes states where the agent has access to a car have significantly higher likelihood of catastrophic failures than they did before.

Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures

Jonathan Uesato*, Ananya Kumar*, Csaba Szepesvari*, Tom Erez, Avraham Ruderman, Keith Anderson, Krishmamurthy (Dj) Dvijotham, Nicolas Heess, Pushmeet Kohli

An important problem in safety-critical domains is accurately estimating slim probabilities of catastrophic failures: one in a million is very different from one in a billion. A standard Monte Carlo approach requires millions or billions of trials to find a single failure, which is prohibitively expensive. This paper proposes using agents from earlier in the training process to provide signals for a learned failure probability predictor. For example, with a Humanoid robot, failure is defined as the robot falling down. A neural net is trained on earlier agents to predict the probability that the agent will fall down from a given state. To evaluate the final agent, states are importance-sampled based on how likely the neural network believes they are to cause failure. This relies on the assumption that the failure modes of the final agent are similar to some failure mode of earlier agents. Overall, the approach reduces the number of samples required to accurately estimate the failure probability by multiple orders of magnitude.


I really like the analogy between programming and AI; a lot of my thoughts have been shaped by thinking about this analogy myself. I agree that the analogy implies that we are trying to solve problems that we've attacked before in a different context, but I do think there are significant differences now. In particular, with long-term AI safety we are considering a setting in which mistakes can be extremely costly, _and_ we can't provide a formal specification of what we want. Contrast this to traffic lights, where mistakes can be extremely costly but I'm guessing we can provide a formal specification of the safety constraints that need to be obeyed. To be fair, Pushmeet acknowledges this and highlights specification learning as a key area of research, but to me it feels like a qualitative difference from previous problems we've faced, whereas I think Pushmeet would disagree with that (but I'm not sure why).

<@Towards Robust and Verified AI: Specification Testing, Robust Training, and Formal Verification@>

Designing robust & reliable AI systems and how to succeed in AI

(As is typical for large content, I'm only summarizing the most salient points, and ignoring entire sections of the podcast that didn't seem as relevant.)

In this podcast, Rob delves into the details of Pushmeet's work on making AI systems _robust_. Pushmeet doesn't view AI safety and AI capabilities as particularly distinct -- part of building a good AI system is ensuring that the system is safe, robust, reliable, and generalizes well. Otherwise, it won't do what we want, so why would we even bother using it. He aims to improve robustness by actively searching for behaviors that violate the specification, or by formally verifying particular properties of the neural net. That said, he also thinks that one of the major challenges here is in figuring out the specification of what to verify in the first place.

He sees the problems in AI as being similar to the ones that arise in programming and computer security. In programming, it is often the case that the program that one writes down does not accurately match the intended specification, leading to bugs. Often we simply accept that these bugs happen, but for security critical systems such as traffic lights we can use techniques like testing, fuzzing, symbolic execution, and formal verification that allow us to find these failures in programs. We now need to develop these techniques for machine learning systems.

The analogy can go much further. Static analysis involves understanding properties of a program separately from any inputs, while dynamic analysis involves understanding a program with a specific input. Similarly, we can have "static" interpretability, which understands the model as a whole (as in [Feature visualization](https://distill.pub/2017/feature-visualization/)), or "dynamic" interpretability, which explains the model's output for a particular input. Another example is that the technique of abstract interpretation of programs is analogous to a particular method for verifying properties of neural nets.

This analogy suggests that we have faced the problems of AI safety before, and have made substantial progress on them; the challenge is now in doing it again but with machine learning systems. That said, there are some problems that are unique to AGI-type systems; it's just not the specification problem. For example, it is extremely unclear how we should communicate with such a system, which may have its own concepts and models that are very different from those of humans. We could try to use natural language, but if we do we need to ground the natural language in the way that humans do, and it's not clear how we could do that, though perhaps we could test if the learned concepts generalize to new settings. We could also try to look at the weights of our machine learning model and analyze whether it has learned the concept -- but only if we already have a formal specification of the concept, which seems hard to get.


A naive alignment strategy and optimism about generalization

We want to build an AI system that answers questions honestly, to the best of its ability. One obvious approach is to have humans generate answers to questions, select the question-answer pairs where we are most confident in the answers, and train an AI system on those question-answer pairs.

(I’ve described this with a supervised learning setup, but we don’t have to do that: we could also [learn](https://deepmind.com/blog/learning-through-human-feedback/) from [comparisons](https://ai-alignment.com/optimizing-with-comparisons-c02b8c0d7877) between answers, and we only provide comparisons where we are confident in the comparison.)

What will the AI system do on questions where we _wouldn’t_ be confident in the answers? For example, questions that are complex, where we may be misled by bad observations, where an adversary is manipulating us, etc.

One possibility is that the AI system learned the **intended policy**, where it answers questions honestly to the best of its ability. However, there is an **instrumental policy** which also gets good performance: it uses a predictive model of the human to say whatever a human would say. (This is “instrumental” in that the model is taking the actions that are instrumental to getting a low loss, even in the test environment.) This will give incorrect answers on complex, misleading, or manipulative questions -- _even if_ the model “knows” that the answer is incorrect.

Intuitively, “answer as well as you can” feels like a much simpler way to give correct answers, and so we might expect to get the intended policy rather than the instrumental policy. This view (which seems common amongst ML researchers) is _optimism about generalization_: we are hoping that the policy generalizes to continue to answer these more complex, misleading, manipulative questions to the best of its ability.

Are there reasons to instead be pessimistic about generalization? There are at least three:

1. If the answers we train on _aren’t_ perfectly correct, the instrumental policy might get a _lower_ training loss than the intended policy (which corrects errors that humans make), and so be more likely to be found by gradient descent.
2. If the AI already needs to make predictions about humans, it may not take much “additional work” to implement the instrumental policy. Conversely, if the AI reasons at a different level of abstraction than humans, it may take a lot of “additional work” to turn correct answers in the AI’s ontology into correct answers in human ontologies.
3. From [a followup post](https://www.alignmentforum.org/posts/SRJ5J9Tnyq7bySxbt/answering-questions-honestly-given-world-model-mismatches), the AI system might answer questions by translating its concepts to human concepts or observations, and then deduce the answer from those concepts or observations. This will systematically ignore information that the AI system understands that isn’t represented in the human concepts or observations. (Consider the [example](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/) of the robot hand that only _looked_ like it was grasping the appropriate object.)

A possible fourth problem: if the AI system did the deduction in its own concepts and only as a final step translated it to human concepts, we might _still_ lose relevant information. This seems not too bad though -- it seems like we should at least be able to <@explain the bad effects of a catastrophic failure@>(@Can there be an indescribable hellworld?@) in human concepts, even if we can’t explain why that failure occurred.

A [followup post](https://www.alignmentforum.org/posts/roZvoF6tRH6xYtHMF/avoiding-the-instrumental-policy-by-hiding-information-about) considers whether we could avoid the instrumental policy by <@preventing it from learning information about humans@>(@Thoughts on Human Models@), but concludes that while it would solve the problems outlined in the post, it seems hard to implement in practice.