This is a cool proof of concept, but the learned perturbation policy can only take you so far -- no learned perturbation policy is going to allow you to e.g. pick up an object after it is dropped, as you would want if you're training a robot to <@manipulate a Rubik's cube@>(@Solving Rubik’s Cube with a Robot Hand@). It seems hard to overcome this sort of problem in a fully automated and learned way (though perhaps you could use more classical techniques to have a "hardcoded" but still automated reset policy).

Paper: The Ingredients of Real World Robotic Reinforcement Learning

Henry Zhu*, Justin Yu*, Abhishek Gupta*, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, Sergey Levine

Suppose we wanted to train a robot to perform a task in the real world, and we didn't want to deal with the headache of sim-to-real transfer. Typically, since all of our experience must be collected in the real world, we would need a human to reset the robot to its initial state. The key idea of this paper is that the point of resets is to ensure that the robot explores a diversity of states causing it to learn a robust policy; this can be achieved by learning a _perturbation policy_ whose objective is to take the robot to states it hasn't visited before. They then combine this with representation learning (so that they can learn from pixels) and use a classifier that distinguishes goal states from non-goal states as the reward function, to get a fully automated setup where once you start the robot's training, it trains itself without any human in the loop.


Given how compute intensive deep RL is, I think it is quite useful to enable cheaper and faster training before these algorithms can be broadly useful. Their claimed speedup is quite impressive, and I like how well they can separate the training and inference from the simulation. I expect that specialized hardware for both training and inference will soon become the norm and SEED RL seems like it will scale well as those accelerators become faster. One thing to note is that this architecture seems very specifically tuned to the problem of games where CPUs can efficiently simulate the environment and it does not improve the sample efficiency for situations where we can’t run lots of simulations.

**Rohin's opinion:** It was quite surprising to me that this worked as well as it did: this model requires communication across machines _at every timestep of the environment_, which intuitively means that latency should be a major bottleneck, while the standard model only requires communication once per batch of trajectories.

Paper: SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference

Massively Scaling Reinforcement Learning with SEED RL

Lasse Espeholt*, Raphaël Marinier*, Piotr Stanczyk*, Ke Wang, Marcin Michalski

Deep learning has <@historically@>(@AI and Compute@) seen many improvements as a result of scaling to larger models with larger amounts of computation, as with the months-long training of <@OpenAI Five@>(@Dota 2 with Large Scale Deep Reinforcement Learning@) and <@AlphaStar@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@). SEED RL redesigns the architecture of distributed RL to enable better machine utilization and communication and achieves an order of magnitude improvement in training speed.

Current distributed architectures typically separate machines into *actors* and *learners*. *Actors* are typically CPUs that simulate the environment, and run inference to predict agent actions. They then send *trajectories* to the *learners*. *Learners* are typically accelerators (GPUs or TPUs), which are responsible for training the model. They then send the updated model parameters to the *actors*.

SEED RL addresses 3 main issues in this setup:
1. Inference could benefit from specialized accelerators
2. Sending model parameters and states requires high bandwidth.
3. Environment simulation and inference are very different tasks and having them on the same machine makes it hard to utilize the resource efficiently.

The solution is to instead have actors **only** simulate the environment. After each step, they send the resulting observation to the *learner*, which is responsible for both training and inference, possibly split on separate hardware). It then sends back just the actions to the environment. This enables each piece of hardware to be used for its designed purpose. Since they now need to communicate at each step, they use gRPC to minimize latency.


This is a cool improvement to domain randomization: it seems like it should be distinctly easier to learn a policy that is dependent on the hidden parameters, and that seems to come at the relatively low cost of needing just a little real world data.

Paper: Learning Agile Robotic Locomotion Skills by Imitating Animals

Robots Learning to Move like Animals

Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, Sergey Levine

<@Previous work@>(@Learning Acrobatics by Watching YouTube@) has suggested that we can get good policies by estimating and imitating poses. This work takes this idea and tries to make it work with sim-to-real transfer. Domain randomization would result in a policy that must be robust to all the possible values of the hidden parameters (such as friction). To make the problem easier, they do domain randomization, but give the agent access to (a latent representation of) the hidden parameters, so that its policy can depend on the hidden parameters. Then, to transfer to the real world, they simply need to search over the latent representation of the hidden parameters in order to find one where the policy actually works in the real world. In practice, they can adapt to the real world with just 8 minutes of real world data.


The GVG-AI framework seems like a useful tool to explore learning videogames. Setting up curriculum learning by using PPCG is also a clever idea. However, the results are a bit mixed. On two of the games they tested, training on a single difficult level works better than training on a variety of levels for generalization. Having said this, the method can learn the game Frogs (57% win rate) while DQN/A2C make zero progress even after 40 million steps. It seems as though certain conditions make PPCG a good method to use. It'd be interesting to investigate what those conditions are in a future publication.

Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation

Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, Sebastian Risi

Deep reinforcement learning has been able to use high-dimensional input, such as images, to learn optimal policies. However, when neural networks are trained in a fixed environment, such as on a single level in a video game, they will usually over-fit and fail to generalize to new levels. This paper uses procedurally generated levels during training in an attempt to increase the generality of deep RL. They make use of the General Video Game AI framework (GVG-AI) which allows rapid design of video games through the specification of rewards, objects, etc. Moreover, they introduce Progressive PCG (PPCG) to smoothly control the difficulty of generated levels to build a curriculum for the agent. The authors show that for some games procedural level generation enables generalization to new levels within the same distribution.


This approach fits neatly with older work including ["Learning to Reach Goals”](https://pdfs.semanticscholar.org/6df4/3f70f383007a946448122b75918e3a9d6682.pdf?_ga=2.78217424.59277899.1578587972-37524897.1574936748) and more recent work such as [Hindsight experience replay](https://arxiv.org/pdf/1707.01495.pdf) and [Goal-Conditioned Policies](https://arxiv.org/pdf/1912.06088.pdf). In particular, all of these methods seem to be effective at addressing the difficulty that comes with working with sparse rewards. I also found myself justifying the utility of selecting the objective of 'learning to achieve general goals' to be related to the idea that <@seeking power is instrumentally convergent@>(@Seeking Power is Provably Instrumentally Convergent in MDPs@).

**Rohin's opinion:** Both this and the previous paper have explored the idea of conditioning on rewards and predicting actions, trained by supervised learning. While this doesn't hit state-of-the-art performance, it works reasonably well for a new approach.

[Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions](http://arxiv.org/abs/1912.02875) and [Training Agents using Upside-Down Reinforcement Learning](http://arxiv.org/abs/1912.02877)

It's a common understanding that using supervised learning to solve RL problems is challenging because supervised learning works directly with error signals while RL only has access to evaluation signals. The approach in these papers introduce 'upside-down' reinforcement learning (UDRL) as a way to bridge this gap. Instead of learning how to predict rewards, UDRL learns how to take actions when given a state and a desired reward. Then, to get good behavior, we simply ask the policy to take actions that lead to particularly high rewards. The main approach is to slowly increase the desired goal behavior as the agent learns in order to maximize agent performance. The authors evaluate UDRL on the Lunar Lander and the Take Cover environments. UDRL ultimately performs worse on Lunar Lander and better on Take Cover so it's unclear whether or not UDRL is an improvement over popular methods. However, when rewards are made to be sparse UDRL is able to significantly outperform other RL methods.


The presentation of the paper is clear. In particular, the idea of planning a sequence of maximally feasible subgoals seems particularly intuitive. In general, I think that LEAP relies on the clever idea of reusing trajectory data to augment the data-set for the goal-conditioned policy. As the authors noted, the question of exploration was mostly neglected. I wonder how well the idea of reusing trajectory data generalizes to the general exploration problem.

**Rohin's opinion:** The general goal of inferring hierarchy and using this to plan more efficiently seems very compelling but hard to do well; this is the goal in most hierarchical RL algorithms and <@Learning Latent Plans from Play@>.

Planning with Goal-Conditioned Policies

Soroush Nasiriany*, Vitchyr H. Pong*, Steven Lin, Sergey Levine

Reinforcement learning can learn complex skills by interacting with the environment. However, temporally extended or long-range decision-making problems require more than just well-honed reactions. **In this paper, the authors investigate whether or not they can obtain the benefits of action planning found in model-based RL without the need to model the environment at the lowest level.** The authors propose a model-free planning framework that learns low-level goal-conditioned policies that use their value functions as implicit models. Goal-conditioned policies are policies that can be trained to reach a goal state provided as an additional input. Given a goal-conditioned policy, the agent can then plan over intermediate subgoals (goal states) using a goal-conditioned value function to estimate reachability. Since the state space is large, the authors propose what they call latent embeddings for abstracted planning (LEAP), which is able to find useful subgoals by first searching a much smaller latent representation space and then planning a sequence of reachable subgoals that reaches the target state. In experiments, LEAP significantly outperforms prior algorithms on 2D navigation and push/reach tasks. Moreover, their method can get a quadruped ant to navigate around walls which is difficult because much of the planning happens in configuration space. This shows that LEAP is able to be extended to non-visual domains.


The extension from PlaNet (adding actor-critic rather than direct single-step reward prediction) is relatively straightforward, but I think latent models are an interesting area - especially if they eventually become at all possible to interpret - and so I'm happy to see more work in this area.

Dream to Control: Learning Behaviors by Latent Imagination

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi

In the past year or so, the idea of learning a transition model in a latent space has gained traction, motivated by the hope that such an approach could combine the best of the worlds of model-free and model-based learning. The central appeal of learning a latent transition model is that it allows you to imagine future trajectories in a potentially high-dimensional, structured observation space without actually having to generate those high-dimensional observations.

Dreamer builds on a prior model by the same authors, <@PlaNet@>(@Learning Latent Dynamics for Planning from Pixels@), which learned a latent representation of the observations, p(s|o), trained both through a VAE-style observation reconstruction loss, and also a transition model q(s-next|s, a), which is trained to predict the state at the next step given only the state at the prior one, with no next-step observation data. Together, these two models allow you to simulate action-conditioned trajectories through latent state space. If you then predict reward from state, you can use this to simulate the value of trajectories. Dreamer extends on this by also training an Actor Critic-style model on top of states to predict action and value, forcing the state representation to not only capture next-step transition information, but also information relevant to predicting future rewards. The authors claim this extension makes their model more able to solve long-horizon problems, because the predicted value function can capture far-future rewards without needing to simulate the entire way there. Empirically, there seems to be reasonable evidence that this claim plays out, at least within the fairly simple environments the model is tested in.


Gym Retro, again

OpenAI is releasing the full version of Gym Retro, with over a thousand games, and a tool for integrating new games into the framework. And of course we see new games in which RL agents find infinite loops that give them lots of reward -- Cheese Cat-Astrophe and Blades of Vengeance.


I agree that episodic training and fixed world dynamics seem like unlikely conditions for most situations we would expect agents to encounter in the real world. Accounting for them seems particularly important to ensure safe exploration and robustness to distributional shift, and I think that these environments could serve as useful benchmarks for these safety problems as well.

Adaptive Online Planning for Continual Lifelong Learning

Lifelong learning is distinct from standard RL benchmarks because
 1. The environment is *sequential* rather than *episodic*; it is never reset to a new start state.
 2. The current *transition* and *reward* function are given, but they change over time.

Given this setup, there are two basic approaches: first, run model-free learning on simulated future trajectories and rerun it every time the dynamics change, and second, run model-based planning on the current model. If you ignore computational constraints, these should be equivalent; however, in practice, the second option tends to be more computationally efficient. The contribution of this work is to make this more efficient, rather than improving final performance, by starting with the second option and then using model-free learning to "distill" the knowledge produced by the model-based planner allowing for more efficient planning in the future.

Specifically, Adaptive Online Planning (AOP) balances between the model-based planner MPPI (a variant of MPC) and the model-free algorithm TD3. MPPI uses the given model to generate a trajectory up to a horizon and then uses an ensemble of value functions to estimate the cumulative reward. This knowledge is then distilled into TD3 for later use as a prior for MPPI. During future rollouts, the variance and Bellman error of the value function ensemble are used to determine how long the horizon should be, and therefore how much computation is used.


Paper: When to Trust Your Model: Model-Based Policy Optimization

Model-Based Reinforcement Learning: Theory and Practice

Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine

This post provides a broad overview of model-based reinforcement learning, and argues that a learned (explicit) model allows you to generate sample trajectories from the current policy at arbitrary states, correcting for off-policy error, at the cost of introducing model bias. Since model errors compound as you sample longer and longer trajectories, the authors propose an algorithm in which the model is used to sample short trajectories from states in the replay buffer, rather than sampling trajectories from the initial state (which are as long as the task's horizon).


This is one of those 'obvious' ideas that turns out to be very difficult to put into practice. I'm glad to see a paper like this simply because the authors do a good job at explaining why a naive execution of the transformer idea is bound to fail. Overall, the architecture seems to be a solid improvement over the TrXL variant. I'd be curious whether or not the architecture is also better in an NLP setting.

Stabilizing Transformers for Reinforcement Learning

Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, Raia Hadsell

Transformers have been incredibly successful in domains with sequential data. Naturally, one might expect transformers to be useful in partially observable RL problems. However, transformers have complex implementations making them difficult to use in an already challenging domain for learning. In this paper, the authors explore a novel transformer architecture they call Gated Transformer-XL (GTrXL) that can be used in the RL setting. The authors succeed in stabilizing training with a reordering of the layer normalization coupled with the addition of a new gating mechanism located at key points in the submodules of the transformer. The new architecture is tested on DMlab-30, a suite of RL tasks including memory, and shows improvement over baseline transformer architectures and the neural computer architecture MERLIN. Furthermore, GTrXL learns faster and is more robust than a baseline transformer architecture.


It's safe to say that work towards good evaluation metrics for RL agents is a good thing. I think this paper captures a lot of the notions of what makes an agent 'good' in a way that seems readily generalizable. The evaluation time on the suite is reasonable, no more than 30 minutes per experiment. Additionally, the ability to produce automated summary reports in standard formats is a nice feature. One thing that seems to be missing from the core set of experiments is a good notion of transfer learning capability beyond simple generalization. However, the authors readily note that the suite is a work in progress so I wouldn't doubt something covering that would be introduced in time.

**Rohin's opinion:** The most interesting thing about work like this is what "core issues" they choose to evaluate -- it's not clear to me whether e.g. "memory" in a simple environment is something that future research should optimize for.

Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepezvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, Hado Van Hasselt

Collecting clear, informative and scalable problems that capture important aspects about how to design general and efficient learning algorithms is difficult. Many current environments used to evaluate RL algorithms introduce confounding variables that make new algorithms difficult to evaluate. In this project, the authors assist this effort by introducing Behaviour Suite for Reinforcement Learning (bsuite), a library that facilitates reproducible and accessible research on core issues in RL. The idea of these experiments is to capture core issues, such as 'exploration' or 'memory', in a way that can be easily tested or evaluated. The main contribution of this project is an open-source project called bsuite, which instantiates all experiments in code and automates the evaluation and analysis of any RL agent on bsuite. The suite is designed to be flexible and includes code to run experiments in parallel on Google cloud, with Jupyter notebook, and integrations with OpenAI Gym.


I am a bit torn here: On one hand, the principle makes it plausible for us to find the globally optimal solution by solving our task on a finite set of training environments. This way the generalization problem feels more well-defined and amenable to theoretical analysis, which seems useful for advancing our understanding of reinforcement learning.

On the other hand, I don't expect the principle to hold for most real-world problems. For example, in interactions with other adapting agents performance will depend on these agents' policies, which can be hard to infer and change dynamically. This means that the principle of unchanged optimality won't hold without precise information about the other agent's policies, while this information can be very difficult to obtain.

More generally, with this and some of the criticism of the [AI safety gridworlds](https://deepmind.com/blog/specifying-ai-safety-problems/) that framed them as an ill-defined benchmark, I am a bit worried that too much focus on very "clean" benchmarks might divert from issues associated with the messiness of the real world. I would have liked to see a more conditional conclusion for the paper, instead of a general principle.

The Principle of Unchanged Optimality in Reinforcement Learning Generalization

In image recognition tasks, there is usually only one label per image, such that there exists an optimal solution that maps every image to the correct label. Good generalization of a model can therefore straightforwardly be defined as a good approximation of the image-to-label mapping for previously unseen data.

In reinforcement learning, our models usually don't map environments to the optimal policy, but states in a given environment to the corresponding optimal action. The optimal action in a state can depend on the environment. This means that there is a tradeoff regarding the performance of a model in different environments.

The authors suggest the principle of unchanged optimality: in a benchmark for generalization in reinforcement learning, there should be at least one policy that is optimal for all environments in the train and test sets. With this in place, generalization does not conflict with good performance in individual environments. If the principle does not initially hold for a given set of environments, we can change that by giving the agent more information. For example, the agent could receive a parameter that indicates which environment it is currently interacting with.


This paper seems like one that will make RL researchers usefully uncomfortable,  by pointing out that the complexity of our implementations means that just having a theoretical story of your algorithm's performance and empirical validation of that heightened performance isn't actually enough to confirm that the theory is actually the thing driving the performance. I do think the authors were a bit overly critical at points: I don't think anyone working in RL would have expected that the learned value function was perfect, or that gradient updates were un-noisy. But, it's a good reminder that saying things like "value functions as a baseline decrease variance" should be grounded in an empirical examination of how good they are at it, rather than just a theoretical argument that they should.

Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms?

Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry

This paper investigates whether and to what extent the stated conceptual justifications for common Policy Gradient algorithms are actually the things driving their success. The paper has two primary strains of empirical investigation.

In the first, they examine a few of the more rigorously theorized aspects of policy gradient methods: learned value functions as baselines for advantage calculations, surrogate rewards, and enforcement of a "trust region" where the KL divergence between old and updated policy is bounded in some way. For value functions and surrogate rewards, the authors find that both of these approximations are weak and perform poorly relative to the true value function and reward landscape respectively.

Basically, it turns out that we lose a lot by approximating in this context. When it comes to enforcing a trust region, they show that TRPO is able to enforce a bound on mean KL, but that it's much looser than the (more theoretically justified) bound on max KL that would be ideal but is hard to calculate. PPO is even stranger: they find that it enforces a mean KL bound, but only when optimizations present in the canonical implementation, but not the core definition of the algorithm, are present. These optimizations include: a custom weight initialization scheme, learning rate annealing on Adam, and reward values that are normalized according to a rolling sum. All of these optimizations contribute to  non-trivial increases in performance over the base algorithm, in addition to apparently being central to how PPO maintains its trust region.


I enjoyed this; it's a well-written paper that uses a few core interesting ideas (posterior sampling over a task distribution, representation of a task distribution as a distribution of embedding vectors passed in to condition Q functions), and builds them up to make a method that achieves some impressive empirical results.

Efficient Off-Policy Meta-RL via Probabilistic Context Variables

Learning to Learn with Probabilistic Task Embeddings

Kate Rakelly*, Aurick Zhou*, Deirdre Quillen, Chelsea Finn, Sergey Levine

This paper proposes a solution to off-policy meta reinforcement learning, an appealing problem because on-policy RL is so sample-intensive, and meta-RL is even worse because it needs to solve a distribution over RL problems. The authors' approach divides the problem into two subproblems: infer an embedding, z, of the current task given context, and learning an optimal policy q function conditioned on that task embedding. At the beginning of each task, z is sampled from the (Gaussian) prior, and as the agent gains more samples of that particular task, it updates its posterior over z, which can be thought of as refining its guess as to which task it's been dropped into this time. The trick here is that this subdividing of the problem allows it to be done mostly off-policy, because you only need to use on-policy learning for the task inference component (predicting z given current task transitions), and can learn the Actor-Critic model conditioned on z with off-policy data. The method works by alternating between these two learning modes.


The specific method is an interesting modification of previous Contrastive Predictive Coding work, but what I found most impressive about this paper was the engineering work involved in pulling metadata supervision signals out of the game by reading  comments on disassembled  source code to see exactly how metadata was being stored in RAM. This seems to have the potential of being a useful benchmark for Atari representation learning going forward (though admittedly Atari games are fairly conceptually straightforward to begin with).

Unsupervised State Representation Learning in Atari

Ankesh Anand*, Evan Racah*, Sherjil Ozair*, Yoshua Bengio, Marc-Alexandre Côté, R Devon Hjelm

This paper has two main contributions: an actual technique for learning representations in an unsupervised way, and an Atari-specific interface for giving access to the underlying conceptual state of the game (e.g. the locations of agents, locations of small objects, current remaining lives, etc) by parsing out the RAM associated with each state. Since the notional goal of unsupervised representation learning is often to find representations that can capture conceptually important features of the state without having direct access to them, this supervision system allows for more meaningful evaluation of existing methods by asking how well conceptual features can be predicted by learned representation vectors. The object-level method of the paper centers around learning representations that capture information about temporal state dynamics, which they do by maximizing mutual information between representations at  adjacent timesteps. More specifically, they have both a local version of this, where a given 1/16th patch of the image has a representation that  is optimized to be predictive of that same patches next-timestep representation, and a local-global version, where the global representation is optimized to be predictive of representations of each patch. They argue this patch-level prediction makes their method better at learning concepts attached to small objects, and the empirical results do seem to support this interpretation.


I really like the idea of being able to easily test whether an agent has robustly learned a behavior or not. To some extent, all of the transfer learning environments are also doing this, such as <@CoinRun@>(@Quantifying Generalization in Reinforcement Learning@) and the <@Retro Contest@>: if the learned behavior is not robust, then the agent will not perform well in the transfer environment. But with Toybox it looks like researchers will be able to run much more granular experiments looking at specific behaviors.

Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning

Emma Tosch, Kaleigh Clary, John Foley, David Jensen

Toybox is a reimplementation of three Atari games (Breakout, Amidar and Space Invaders) that enables researchers to customize the games themselves in order to perform better experimental evaluations of RL agents. They demonstrate its utility using a case study for each game. For example, in Breakout we often hear that the agents learn to "tunnel" through the layer of bricks so that the ball bounces around the top of the screen destroying many bricks. To test whether the agent has learned a robust tunneling behavior, they train an agent normally, and then at test time they remove all but one brick of a column and see if the agent quickly destroys the last brick to create a tunnel. It turns out that the agent only does this for the center column, and sometimes for the one directly to its left.


This is a great experimental paper, there's a lot of data that can help understand DQL algorithms. I wouldn't take the results too literally, since insights on simple environments may not generalize to more complex environments. For example, they found overfitting to be an issue in their environments -- it's plausible to me that with more complex environments (think Dota/StarCraft, not Mujoco) this reverses and you end up underfitting the data you have. Nonetheless, I think data like this is particularly valuable for coming up with an intuitive theory of how deep RL works, if not a formal one.

Diagnosing Bottlenecks in Deep Q-learning Algorithms

Justin Fu, Aviral Kumar, Matthew Soh, Sergey Levine

While the PreQN paper used a theoretical approach to tackle Deep Q-Learning algorithms, this one takes an empirical approach. Their results:
    - Small neural nets cannot represent Q*, and so have undesired bias that results in worse performance. However, they also have convergence issues, where the Q-function they actually converge to is significantly worse than the best Q-function that they could express. Larger architectures mitigate both of these problems.
    - When there are more samples, we get a lower validation loss, showing that we are overfitting. Despite this, larger architectures are better, because the performance loss from overfitting is not as bad as the performance loss from having a bad bias. A good early stopping criterion could help with this.
    - To study how non-stationarity affects DQL algorithms, they study a variant where the Q-function is a moving average of the past Q-functions (instead of the full update), which means that the target values don't change as quickly (i.e. it is closer to a stationary target). They find that non-stationarity doesn't matter much for large architectures.
    - To study distribution shift, they look at the difference between the expected Bellman error before and after an update to the parameters. They find that distribution shift doesn't correlate much with performance and so is likely not important.
    - Algorithms differ strongly in the distribution over (s, a) pairs that the DQL update is computed over. They study this in the absence of sampling (i.e. when they simply weight all possible (s, a) pairs, rather than just the ones sampled from a policy) and find that distributions that are "close to uniform" perform best. They hypothesize that this is the reason that experience replay helps -- initially an on-policy algorithm would take samples from a single policy, while experience replay adds samples from previous versions of the policy, which should increase the coverage of (s, a) pairs.

To sum up, the important factors are using an expressive neural net architecture, and designing a good sampling distribution. Inspired by this, they design Adversarial Feature Matching (AFM), which like Prioritized Experience Replay (PER) puts more weight on samples that have high Bellman error. However, unlike PER, AFM does not try to reduce distribution shift via importance sampling, since their experiments found that this was not important.


This work follows the standard pattern where model-based RL is more sample efficient but reaches worse final performance compared to model-free RL. Let's try to explain this using the same story as in the rest of this newsletter.

The sample efficiency comes from the fact that they learn a world model that can predict the future, and then use that model to solve the control problem (which has zero sample cost, since you are no longer interacting with the environment). It turns out that predicting the future is "easier" than selecting the optimal action, and so the world model can be trained in fewer samples than it would take to solve the control problem directly. Why is the world model "easier" to learn? One possibility is that solving the control problem requires you to model the world anyway, and so must be a harder problem. If you don't know what your actions are going to do, you can't choose the best one. I don't find this very compelling, since there are lots of aspects of world modeling that are irrelevant to the control problem -- you don't need to know exactly how the background art will change in order to choose what action to take, but world modeling requires you to do this. I think the real reason is that world modeling benefits from much more supervision -- rather than getting a sparse reward signal over a trajectory, you get a full grid of pixels every timestep that you were supposed to predict. This gives you many orders of magnitude more "supervision information" per sample, and so it makes it easier to learn. (This is basically the same argument as in [Yann Lecun's cake analogy](https://medium.com/syncedreview/yann-lecun-cake-analogy-2-0-a361da560dae).)

Why does it lead to worse performance overall? The policy is now being trained using rollouts that are subtly wrong, and so instead of specializing to the true Atari dynamics it will be specialized to the world model dynamics, which is going to be somewhat different and should lead to a slight dip in performance. (Imagine a basketball player having to shoot a ball that was a bit heavier than usual -- she'll probably still be good, but not as good as with a regular basketball.) In addition, since the world model is supervised by pixels, any small objects are not very important to the world model (i.e. getting them wrong does not incur much loss), even if they are very important for control. In fact, they find that bullets tend to disappear in Atlantis and Battle Zone, which is not good if you want to learn to play those games.

I'm not sure if they shared weights between the world model and the control policy. If they did, then they would also have the problem that the features that are useful for predicting the future are not the same as the features that are useful for selecting actions, which would also cause a drop in performance. My guess is that they didn't share weights for precisely this reason, but I'm not sure.

Model-Based Reinforcement Learning for Atari

Simulated Policy Learning in Video Models

Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Miłos, Błażej Osiński, Roy H Campbell, Konrad Czechowski, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker and Henryk Michalewski

This blog post and the associated [paper](https://arxiv.org/abs/1903.00374) tackle model-based RL for Atari. The recent <@world models@>(@Recurrent World Models Facilitate Policy Evolution@) paper proposed first learning a model of the world by interacting with the environment using a random policy, and then using the model to simulate the environment and training a control policy using those simulations. (This wasn't it's main point, but it was one of the things it talked about.) The authors take this idea and put it in an iterative loop: they first train the world model using experience from a random policy, then train a policy using the world model, retrain the world model with experience collected using the newly trained policy, retrain the policy, and so on. This allows us to correct any mistakes in the world model and let it adapt to novel situations that the control policy discovers. This allows them to train agents that can play Atari with only 100K interactions with the environment (corresponding to about two hours of real-time gameplay), though the final performance is lower than the state-of-the-art achieved with model-free RL. See [Import AI](https://jack-clark.net/2019/04/01/import-ai-140-surveilling-a-city-via-the-cityflow-dataset-25000-images-of-chinese-shop-signs-and-the-seven-traps-of-ai-ethics/) for more details.


This is a cool example of using structured knowledge in order to get generalization while also using deep learning in order to get performance. I also recently came across [Residual Reinforcement Learning for Robot Control](https://arxiv.org/abs/1812.03201), which seems to have the same idea of combining deep RL with conventional control mechanisms. I haven't read either of the papers in depth, so I can't compare them, but a _very_ brief skim suggests that their techniques are significantly different.

Unifying Physics and Deep Learning with TossingBot

TossingBot is a system that learns how to pick up and toss objects into bins using deep RL. The most interesting thing about it is that instead of using neural nets to directly predict actions, they are instead used to predict _adjustments_ to actions that are computed by a physics-based controller. Since the physics-based controller generalizes well to new situations, TossingBot is also able to generalize to new tossing locations.


Assessing Generalization in Deep Reinforcement Learning (blog post)

This is a blog post summarizing <@Assessing Generalization in Deep Reinforcement Learning@>.


Firstly, the blog post is very readable and provides a great introduction (it's much more friendly than my summary).

This technique does require that we can reinterpret any state as a goal state, similar to the assumption in [Hindsight Experience Replay](https://arxiv.org/abs/1707.01495) (HER). They do compare to HER, and find that HER doesn't do very well, which I was quite surprised by. Clicking through to the paper, it turns out the authors were surprised as well, but then realized that this is because HER is designed to work with sparse reward problems, whereas they were evaluating on problems with relatively shaped rewards.

TDM: From Model-Free to Model-Based Deep Reinforcement Learning

In many tasks, we have hierarchical structure where we want to plan at a high level, but to execute the low-level actions we want to rely on learning through experience. For example, when biking from UC Berkeley to the Golden Gate Bridge, you definitely want to plan in advance the route you'll take (as opposed to learning through trial-and-error), but you want to learn how to bike through trial-and-error. Temporal Difference Models allow you to do model-based planning at the high level, and model-free learning at the low level. Specifically, you learn a function Q(s1, a, s2, T), which intuitively says "if I start from state s1, taking action a, and running for T steps, how close can I get to state s2". It turns out that this can be thought of as a Q function and so can be trained using standard model-free RL techniques. Note that the constraint Q(s1, a, s2, T) = 0 says that it is possible to get from s1 to s2 in T steps after first taking action a.

One standard way to solve model-based RL is to search for a sequence of states and actions (s0, a0, s1, a1, ...) that is feasible (agrees with the dynamics) and maximizes the reward, and then take the first action from that sequence. Using TDMs, we can now search for the sequence (s0, a0, sK, aK, s2K, a2k, ...) that is feasible and maximizes reward. The feasibility requirement is expressed by the constraint Q(s0, a0, sK, K) = 0.


My guess is that multiagent learning will become very popular fairly soon. In addition to this paper and the Neural MMO paper, it was also a key part of the AlphaStar training process. The implications of this research direction for safety are still unclear, and it seems valuable to explore them further. One which comes to mind: the sort of deceptive behaviour required for treacherous turns seems more likely to emerge from multiagent training than from single-agent training.

Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research

Joel Z. Leibo, Edward Hughes, Marc Lanctot, Thore Graepel

The authors argue that the best solution to the problem of task generation is creating multi-agent systems where each agent must adapt to the others. These agents do so first by learning how to implement a high-level strategy, and then by adapting it based on the strategies of others. (The authors use the term "adaptive unit" rather than "agent" to emphasise that change can occur at many different hierarchical levels, and either by evolution or learning). This adaptation may be exogenous (driven by the need to respond to a changing environment) or endogenous (driven by a unit's need to improve its own functionality). An example of the latter is a society implementing institutions which enforce cooperation between individuals. Since individuals will try to exploit these institutions, the process of gradually robustifying them can be considered an automatically-generated curriculum (aka autocurriuclum).


This is a great example of a pattern that seems quite common: once we automate tasks using end-to-end training that previously required more structured approaches, new more complex tasks will arise that will use the end-to-end trained systems as building blocks in a bigger structured approach. In this case, we can now train robots to navigate over short distances using end-to-end training, and this has been used in a structured approach involving graphs and waypoints to create robots that can traverse larger distances.

It's also an example of what you can do when you have a ton of compute: for the learned controller, they learned both the network architecture and the reward shaping. About the only thing that had to be explicity specified was the sparse true reward. (Although I'm sure in practice it took a lot of effort to get everything to actually work.)

Long-Range Robotic Navigation via Automated Reinforcement Learning

How can we get robots that successfully navigate in the real world? One approach is to use a high-level route planner that uses a learned control policy over very short distances (10-15 meters). The control policy is learned using deep reinforcement learning, where the network architecture and reward shaping is also learned via neural architecture search (or at least something very similar). The simulations have enough noise that the learned control policy transfers well to new environments. Given this policy as well as a floorplan of the environment we want the robot to navigate in, we can build a graph of points on the floorplan, where there is an edge between two points if the robot can safely navigate between the two points using the learned controller (which I _think_ is checked in simulation). At execution time, we can find a path to the goal in this graph, and move along the edges using the learned policy. They were able to build a graph for the four buildings at the Google main campus using 300 workers over 4 days. They find that the robots are very robust in the real world. See also [Import AI](https://jack-clark.net/2019/03/04/import-ai-136-what-machine-learning-power-infrastructure-means-for-humanity-new-gca-benchmarkdataset-challenges-image-captioning-systems-and-google-uses-frankenrl-to-create-more-mobile-robot/).


Neural MMO

Neural MMO is "a massively multiagent game environment for reinforcement learning agents". It was designed to be persistent (with concurrent learning and no environment resets), large-scale, efficient and expandable. Agents need to traverse an environment to obtain food and water in order to survive for longer (the metric for which they are rewarded), and are also able to engage in combat with other agents. Agents trained within a larger population explore more and consistently outperform those trained in smaller populations (when evaluated together). The authors note that multiagent training is a curriculum magnifier, not a curriculum in itself, and that the environment must facilitate adaptive pressures by allowing a sufficient range of interactions.


Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence

I like this sort of research, and its findings are interesting (even if the authors don't arrive at any clear explanation for them). One concern: I may be missing something, but it seems like the coupled samples method they use doesn't allow investigation into whether distributional methods benefit from generating better data (e.g. via more effective exploration).

A Comparative Analysis of Expected and Distributional Reinforcement Learning

Clare Lyle, Pablo Samuel Castro, Marc G. Bellemare

Distributional RL systems learn distributions over the value of actions rather than just their expected values. In this paper, the authors investigate the reasons why this technique improves results, by training distribution learner agents and expectation learner agents on the same data. They provide evidence against a number of hypotheses: that distributional RL reduces variance; that distributional RL helps with policy iteration; and that distributional RL is more stable with function approximation. In fact, distributional methods have similar performance to expectation methods when using tabular representations or linear function approximators, but do better when using non-linear function approximators such as neural networks (especially in the earlier layers of networks).


I endorse the goals of this paper, but my guess is that Hanabi is simple enough that agents can solve it using isolated heuristics rather than general reasoning about other agents' beliefs.

*Rohin's opinion:* I'm particularly excited to see more work on ad hoc teamwork, since it seems like very similar to the setting we are in, where we would like to deploy AI system among groups of humans and have things go well. See [Following human norms](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/eBd6WvzhuqduCkYv3) ([AN #42](https://mailchi.mp/f6488137d76c/alignment-newsletter-42)) for more details.

A cooperative benchmark: Announcing the Hanabi Learning Environment

The Hanabi Challenge: A New Frontier for AI Research

Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, Iain Dunning, Shibl Mourad, Hugo Larochelle, Marc G. Bellemare, Michael Bowling

The authors propose the cooperative, imperfect-information card game Hanabi as a target for AI research, due to the necessity of reasoning about the beliefs and intentions of other players in order to win. They identify two challenges: firstly, discovering a policy for a whole team that allows it to win (the self-play setting); and secondly, discovering an individual policy that allows an agent to play with an ad-hoc team without previous coordination. They note that successful self-play policies are often very brittle in the ad-hoc setting, which makes the latter the key problem. The authors provide an open-source framework, an evaluation benchmark and the results of existing RL techniques.


Recurrent Experience Replay in Distributed Reinforcement Learning

Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, Will Dabney


I think that the "problem problem" is going to become increasingly important in RL, and that this is a promising approach. Note that this paper's contribution seems to be mainly that it combines ideas from previous papers on minimal criteria coevolution and innovation engines.

Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Rui Wang, Joel Lehman, Jeff Clune, Kenneth O. Stanley

The POET algorithm uses evolutionary strategies to evolve a population of pairs of tasks and agents. During each iteration, it first generates a new environment by perturbing an existing environment, then optimises each agent for its paired environment, then attempts to transfer agents between existing environments to improve performance (in case one environment turns out to be a useful "stepping stone" towards another). New environments are kept if they are neither too hard nor too easy for the current population of agents. This algorithm was tested using the Bipedal Walker environment, where it significantly outperformed standard evolutionary search.


While I agree with some of the concerns laid out in this paper, I'm not sure that these benchmarks are the best way to address them. The third task in particular is mainly testing for ability to ignore the "natural data" used, which doesn't seem very useful. I think a better alternative would be to replace Atari with tasks in procedurally-generated environments with realistic physics engines. However, this paper's benchmarks do benefit from being much easier to produce and less computationally demanding.

Natural Environment Benchmarks for Reinforcement Learning

This paper notes that RL performance tends to be measured in simple artificial environments - unlike other areas of ML in which using real-world data such as images or text is common. The authors propose three new benchmarks to address this disparity. In the first two, an agent is assigned to a random location in an image, and can only observe parts of the image near it. At every time step, it is able to move in one of the cardinal directions, unmasking new sections of the image, until it can classify the image correctly (task 1) or locate a given object (task 2). The third type of benchmark is adding natural video as background to existing Mujoco or Atari tasks. In testing this third category of benchmark, they find that PPO and A2C fall into a local optimum where they ignore the observed state when deciding the next action.


This is an interesting approach which I can imagine becoming useful. It would be nice to see more experimental work in more stochastic environments, though.

Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean-Baptiste Lespiau, Nicolas Heess

This paper aims to alleviate the data inefficiency of RL by using a model to synthesise data. However, even when environment dynamics can be modeled accurately, it can be difficult to generate data which matches the true distribution. To solve this problem, the authors use a Structured Causal Model trained to predict the outcomes which would have occurred if different actions had been taken from previous states. Data is then synthesised by rolling out from previously-seen states. The authors test performance in a partially-observable version of SOKOBAN, in which their system outperforms other methods of generating data.


This paper seems like a step forward in addressing the instability of using learned models in RL. However, the extent to which it's introducing new contributions, as opposed to combining existing ideas, is a little unclear.

Learning Latent Dynamics for Planning from Pixels

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson

The authors introduce PlaNet, an agent that learns an environment's dynamics from pixels and then chooses actions by planning in latent space. At each step, it searches for the best action sequence under its Recurrent State Space dynamics model, then executes the first action and replans. The authors note that having a model with both deterministic and stochastic transitions is critical to learning a good policy. They also use a technique called variational overshooting to train the model on multi-step predictions, by generalising the standard variational bound for one-step predictions. PlaNet approaches the performance of top model-free algorithms even when trained on 50x fewer episodes.


Evolved Policy Gradients

In this meta-learning approach for reinforcement learning, the outer optimization loop proposes a new _loss function_ for the inner loop to optimize (in contrast to eg. MAML, where the outer optimization leads to better initializations for the policy parameters). The outer optimization is done using evolution strategies, while the inner optimization is stochastic gradient descent. The authors see good results on generalization to out-of-distribution tasks, which other algorithms such as RL2 don't achieve.


This continues the trend of being able to easily implement deep learning at higher and higher levels of abstraction. I'm looking forward to using it.

Open sourcing TRFL: a library of reinforcement learning building blocks

Matteo Hessel, Miljan Martic, Diego de Las Casas and Gabriel Barth-Maron

DeepMind is open-sourcing a Tensorflow library of "key algorithmic components" used in their RL agents. They hope that this will allow less buggy RL code.


It's quite remarkable how some supervision (poses in this case) can lead to such large improvements in the task. Of course, the promise of deep RL is to accomplish tasks with very little supervision (just a reward function), so this isn't a huge breakthrough, but it's still better than I expected. Intuitively, this works so well because the "reward" during the imitation phase is extremely dense -- the reference motion provides feedback after each action, so you don't have to solve the credit assignment problem.

Learning Acrobatics by Watching YouTube

Xue Bin (Jason) Peng and Angjoo Kanazawa

To imitate human behavior in videos, it is sufficient to estimate the human pose for each frame, to smooth the poses across frames to eliminate any jittery artifacts or mistakes made by the pose estimator, and then to train the robot to match the motion exactly. This results in really good performance that looks significantly better than corresponding deep RL approaches, but of course it relies on having labeled poses to train the pose estimator in addition to the simulator.


Introducing a New Framework for Flexible and Reproducible Reinforcement Learning Research

Pablo Samuel Castro and Marc G. Bellemare

Researchers at Google have released Dopamine, a small framework for RL research on Atari games, with four built-in agents -- DQN, C51, a simplified version of Rainbow, and the recent Implicit Quantile Network. There's a particular emphasis on reproducibility, by providing logs from training runs, training data, etc.


There are still some things that the bots do that seem like bad decisions. You can interpret this a few ways. Five could have learned a large number of heuristics that make it good enough to beat almost all humans, but that break down in edge cases. In this story, Five is not good at learning logical or abstract reasoning, but can compensate for that in the average case with the sheer number of heuristics it can learn. Another interpretation is that Five learns a good representation of Dota which lets it come up with new, novel insights into the game, which we can't see or understand because the representation is alien to us. However, the representation makes it harder to come up with other insights about Dota that we have using our representations of Dota, and as a result Five makes some mistakes that humans can easily recognize as mistakes. I lean towards the first interpretation, but not very strongly.

The International 2018: Results

Two human teams beat OpenAI Five at The International. The games seemed much more like regular Dota, probably because there was now only one vulnerable courier for items instead of five invulnerable ones. This meant that OpenAI Five's strategy of a relentless team attack on the enemy was no longer as powerful, because they couldn't get the health regeneration items they needed to constantly stay alive to continue the attack. It's also possible (but less likely to me) that the matches were more normal because the teams were more even, or because the human teams knew about Five's strategy this time and were countering it in ways that I don't understand.


Unsupervised learning seems like the way forward to learn rich models of the world, because of the sheer volume of data that you can use.

Learning Actionable Representations from Visual Observations

Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, Pierre Sermanet

Prior work on Time Contrastive Networks (TCN)s showed that you can use time as an unsupervised learning signal, in order to learn good embeddings of states that you can then use in other tasks. This paper extends TCNs to work with multiple frames, so that it can understand motion as well. Consider any two short videos of a task demonstration. If they were taken at different times, then they should be mapped to different embedding vectors (since they correspond to different "parts" of the task). On the other hand, if they were taken at the same time (even if from different viewpoints), they should be mapped to the same embedding vector. The loss function based on this encourages the network to learn an embedding for these short videos that is invariant to changes in perspective (which are very large changes in pixel-space), but _is_ different for changes in time (which may be very small changes in pixel-space). They evaluate with a bunch of different experiments.


We're seeing more and more work combining classical symbolic approaches with the current wave of statistical machine learning from big data, that gives them the best of both worlds. While the results we see are not general intelligence, it's becoming less and less true that you can point to a broad swath of capabilities that AI cannot do yet. I wouldn't be surprised if a combination of symbolic and stastical AI techniques led to large capability gains in the next few years.

Learning Plannable Representations with Causal InfoGAN

Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart Russell, Pieter Abbeel

Hierarchical reinforcement learning aims to learn a hierarchy of actions that an agent can take, each implemented in terms of actions lower in the hierarchy, in order to get more efficient planning. Another way we can achieve this is to use a classical planning algorithm to find a sequence of _waypoints_, or states that the agent should reach that will allow it to reach its goal. These waypoints can be thought of as a high-level plan. You can then use standard RL algorithms to figure out how to go from one waypoint to the next. However, typical planning algorithms that can produce a sequence of waypoints require very structured state representations, that were designed by humans in the past. How can we learn them directly from data? This paper proposes Causal InfoGAN. They use a GAN where the generator creates adjacent waypoints in the sequence, while the discriminator tries to distinguish between waypoints from the generator and pairs of points sampled from the true environment. This incentivizes the generator to generate waypoints that are close to each other, so that we can use an RL algorithm to learn to go from one waypoint to the next. However, this only lets us generate adjacent waypoints. In order to use this to make a sequence of waypoints that gets from a start state to a goal state, we need to use some classical planning algorithm. In order to do that, we need to have a structured state representation. GANs do not do this by default. InfoGAN tries to make the latent representation in a GAN more meaningful by providing the generator with a "code" (a state in our case) and maximizing the mutual information of the code and the output of the generator. In this setting, we want to learn representations that are good for planning, so we want to encode information about _transitions_ between states. This leads to the Causal InfoGAN objective, where we provide the generator with a pair of abstract states (s, s'), have it generate a pair of observations (o, o') and maximize the mutual information between (s, s') and (o, o'), so that s and s' become good low-dimensional representations of o and o'. They show that Causal InfoGAN can create sequences of waypoints in a rope manipulation task, that previously had to be done manually.


For over a year now people keep asking me whether something like this is doable, since it seems like an obvious win combining PL and ML, and why no one has done it yet. I've mentioned the issue about neural nets being too slow, but it still seemed doable, and I was really tempted to do it myself. So I'm really excited that it's finally been done!

Oh right, AI alignment. Yeah, I do actually think this is somewhat relevant -- this sort of work could lead to much better theorem provers and formal reasoning, which could make it possible to create AI systems with formal guarantees. I'm not very optimistic about this approach myself, but I know others are.

Learning Heuristics for Automated Reasoning through Deep Reinforcement Learning

Gil Lederman, Markus N. Rabe, Edward A. Lee, Sanjit A. Seshia

The formal methods community uses SAT solvers all the time in order to solve complex search-based problems. These solvers use handtuned heuristics in order to drastically improve performance. The heuristics only affect the choices in the search process, not the correctness of the algorithm overall. Obviously, we should consider using neural nets to learn these heuristics instead. However, neural nets take a long time to run, and SAT solvers have to make these decisions very frequently, so it's unlikely to actually be helpful -- the neural net would have to be orders of magnitude better than existing heuristics. So, they instead do this for QBF (quantified boolean formulas) -- these are PSPACE complete, and the infrastructure needed to support the theory takes more time, so it's more likely that neural nets can actually help. They implement this using a graph neural network and engineer some simple features for variables and clauses. (Feature engineering is needed because there can hundreds of thousands of variables, so you can only have ~10 numbers to describe the variable.) It works well, doing better than the handcoded heuristics.


This is a cool example of a relatively simple yet powerful idea -- instead of having a goal space over all states, learn a good latent representation and use that as your goal space. This enables unsupervised learning in order to figure out how to use a robot to generally affect the world, probably similarly to how babies explore and learn.

Visual Reinforcement Learning with Imagined Goals

Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, Sergey Levine

[Hindsight Experience Replay](https://blog.openai.com/ingredients-for-robotics-research/) (HER) introduced the idea of accelerating learning with sparse rewards, by taking trajectories where you fail to achieve the goal (and so get no reward, and thus no learning signal) and replacing the actual goal with an "imagined" goal chosen in hindsight such that you actually achieved that goal, which means you get reward and can learn. This requires that you have a space of goals such that for any trajectory, you can come up with a goal such that the trajectory achieves that goal. In practice, this means that you are limited to tasks where the goals are of the form "reach this goal state". However, if your goal state is an image, it is very hard to learn how to act in order to reach any possible image goal state (even if you restrict to realistic ones), since the space is so large and unstructured. The authors propose to first learn a structured latent representation of the space of images using a variational autoencoder (VAE), and then use that structured latent space as the space of goals which can be achieved. They also use Q-learning instead of DDPG (which is what HER used), so that they can imagine any goal with a minibatch (s, a, s') and learn from it (whereas HER/DDPG is limited to states on the trajectory).


I wonder if they are planning to play a game where both sides draft heroes, or where both sides get a randomly chosen team of 5 heroes. Previously I would have expected that they were choosing randomly, since it seems very difficult to learn solely from experience whether your team choice works well, given that the number of possible drafts is combinatorially large, and the way that the draft affects outcome is very complicated and long term and so hard to capture in a gradient. Now, I'm pretty uncertain -- if deep RL was enough to get this far, it could be good enough to deal with that as well. And it's possible that you can actually do well at drafting with some relatively simple heuristics -- I don't know Dota well enough to say.

OpenAI Five Benchmark

The benchmark match for OpenAI Five will be a best-of-three match on August 5 at 2pm. They have already removed many of the restrictions on gameplay, including the two most important ones (wards and Roshan), as well as widening the pool of heroes to choose from 5 to 18.


It's worth highlighting that this can be seen as an extension to [Upside-Down RL](http://arxiv.org/abs/1912.02877) ([AN #83](https://mailchi.mp/ff565f097630/an-83-sample-efficient-deep-learning-with-remixmatch)). In that report, the goal is also to produce actions consistent with the desired reward. This paper extends that line of work by using transformers to expand context beyond the immediate state which aids in long-term credit assignment. The authors claim this helps via self-attention, but it seems much more likely that this effect comes from using the return-to-go as in Upside-Down RL.

Decision Transformer: Reinforcement Learning via Sequence Modeling

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch

In this paper, the authors abstract reinforcement learning (RL) as a sequence modeling problem. The authors are inspired by the rise of powerful sequence models (i.e transformers) in natural language processing. Specifically, they hypothesize that when models are trained to predict the expected reward-to-go alongside state and action sequences, the transformer architecture can be used to do RL.

As an example, consider finding the shortest path between two vertices on a graph. We could start by recording random walks with their expected returns. Once we have enough data, we could condition on paths such that the expected return-to-go (length remaining) is low. This would effectively return shortest paths without the explicit need for optimization.

This framework works well in practice and is competitive with state-of-the-art model-free offline RL baselines on Atari and OpenAI gym. The authors also carry out ablation studies to determine if the sequence modeler is just doing imitation learning on a subset of the data with high returns. This turns out not to be the case, indicating that the approach effectively uses the entire dataset.


It's fun to note that this paper came out of the same lab as Decision Transformer (the paper summarized above) only a day after. In contrast to Decision Transformer, this paper focuses more on the finer technical details of transformers by utilizing beam-search and studying the attentional patterns of the resulting models. The figures in this paper are informative. I feel I did learn something about *why* transformers seem to work in this domain. While the idea of goal-conditioned RL itself isn't novel, showing that 'off-the-shelf' transformers can do well in this domain is impressive.

Reinforcement Learning as One Big Sequence Modeling Problem

Michael Janner, Qiyang Li, Sergey Levine

Typically, RL is concerned with estimating policies that utilize immediate state information to produce high returns. However, we can also view RL as concerned with predicting sequences of actions that lead to high returns. From this perspective, it's natural to wonder if sequence modelers that work well in other domains, such as transformers in NLP, would work well for RL. This paper tests this hypothesis and demonstrates the utility of transformers in RL for a variety of problem settings.

As with the last paper, the authors train the model to predict the reward-to-go. In place of trajectory optimizers, the authors make use of beam search as a planning algorithm. To do RL, rather than maximize the log-probability of potential sequences, the authors replace the log-probability search heuristic with the reward-to-go. In experiments, transformers that maintain the log-probability can imitate expert policies to high fidelity. Visually, the resulting policies are indistinguishable from that of the expert.

The authors also show that their method is competitive on the standard OpenAI gym benchmarks. Finally, the authors look at the attention patterns of the trained models. They identify two patterns: the first links variables in a strictly Markovian fashion and the other links dimensions of the state-action variables across time. Interestingly, action variables are more strongly coupled to past actions than past state variables. This suggests a connection to action-smoothing proposed previously for deep-dynamics models.


The Pursuit of (Robotic) Happiness: How TRPO and PPO Stabilize Policy Gradient Methods

I barely looked at this -- I think it's an introduction to policy gradient methods for reinforcement learning. It assumes very little background (less than I assume in these summaries).


Retro Contest: Results

John Schulman, Vicki Pfau, Alex Nichol, Christopher Hesse, Oleg Klimov and Larissa Schiavo

OpenAI has announced the results of the [Retro Contest](https://blog.openai.com/retro-contest/). The winning submissions were modified versions of existing algorithms like joint PPO and Rainbow, without any Sonic-specific parts.


DeepSpeed: Extreme-scale model training for everyone

DeepSpeed Team, Rangan Majumder, Junhua Wang

In this post, Microsoft announces updates to DeepSpeed, its open-source deep learning training optimization library. The new updates include:

 - '3D parallelism', a scheme for carefully optimizing how training runs are split across machines. Training runs that use 3D parallelism demonstrate linear scaling of GPU memory and compute efficiency, enabling the theoretical training of extremely large models of over a trillion parameters on as few as 800 NVIDIA V100 GPUs.
 - 'ZeRO-Offload', which allows CPU memory to be used during training runs, enabling running models of up to 13 billion parameters on a single NVIDIA V100 GPU.
 - 'DeepSpeed Sparse Attention', an instrumental technology that reduces the compute and memory requirements of attention computations used in models like Transformers. Compared to models that use densely computed attention, this enables models that pay attention to sequences that are 10x longer and can be trained up to 6.3x faster.
 - '1-bit Adam', a scheme for compressing the communication requirements between machines doing training runs that use the Adam gradient descent optimizer. 1-bit Adam enables up to 5x less communication and up to 3.5x faster training runs.


I really like the idea of successor features. Similar to model-based systems, they allow us to evaluate policies for many different rewards, which can be useful for anticipating problematic behaviour before deploying a system. However, note that we still need to execute the policy we obtained by generalized policy improvement to evaluate it for different rewards: The only guarantees we have is that it is better than the previous policies for the reward for which the improvement step was carried out (and potentially some weaker bounds based on the similarity of different rewards).

Fast reinforcement learning with generalized policy updates

Fast reinforcement learning through the composition of behaviours

André Barreto, Shaobo Hou, Diana Borsa, David Silver, Doina Precup

While model-based RL agents can easily adapt their policy to changed rewards on the same environment, planning is expensive and learning good models can be challenging for many tasks. On the other hand, it is challenging to get model-free agents to adapt their policy to a new reward without extensive retraining. An intermediate solution is to use so-called successor features: Instead of a value function **V(π,s)** representing the expected discounted reward for a policy **π** starting in state **s**, successor features are a vector-valued value function **ψ(π,s)** representing an expected discounted feature vector **ϕ**. If our reward equals **r = w ⋅ ϕ** for some weight vector **w**, we can easily obtain the original value function by taking the scalar product of the successor features and the weight vector: **V(π,s) = w ⋅ ψ(π,s)**. Successor features thus allow us to evaluate a fixed policy **π** for all rewards that are linear in **ϕ**, which is called *generalized policy evaluation*.

Now that we can evaluate policies for different preferences, we would like to efficiently find a good policy for a given novel preference. Inspired by human learning that often combines previously learned skills, we employ *generalized policy improvement*. In vanilla policy improvement, we improve upon a policy **π** we can evaluate by choosing the action that maximizes the immediate reward plus the discounted value **V(π,s')** of following **π** starting in the next state **s'**. In generalized policy improvement, we have multiple policies and choose the action that maximizes the reward plus the discounted value of following the best of these policies starting in the next state **s'**. To obtain a policy for the new preference, we "stitch together" all policies we learnt for previous preferences and the resulting policy performs at least as good as all of the old policies with respect to the new preference. As generalized policy improvement does not require any additional environment samples, it enables zero-shot transfer to new preferences. Empirically, even if the weight vector **w** has to be learnt from reward signals, generalized policy improvement is very sample efficient. Additional samples can then be used to further improve the policy using standard RL.


I am a bit surprised that this works so well, as both bootstrapping and learning generative models for distributions can be unstable and the method combines both. On the other hand, there is a long tradition of continuous interpolations between different RL algorithms and their performance at the sweet spot is often significantly stronger than at the extremes.

γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction

Michael Janner, Igor Mordatch, Sergey Levine

Long planning horizons are often necessary for competitive performance of model-based agents, but single-step models get less and less accurate with longer planning horizons as errors accumulate. Model-free algorithms don't have this problem but are usually reward- and policy-specific, such that transfer to other tasks can be hard. The paper proposes policy-specific γ-models as an intermediate solution: instead of learning the distribution of the next state given a state-action pair **(s,a)**, or the final state of an n-step rollout given **(s,a)** and a policy **π**, it learns the distribution of a rollout with a stochastic, geometrically distributed length. Unlike for n-step models with n>1, the distribution follows a Bellman-style decomposition into the single-step distribution and the discounted distribution for the next state **s'**, which allows for off-policy training of the model by bootstrapping the target distribution.

Now, if rewards are consequentialist in the sense that they only depend on the state, the expected reward under this distribution is equal to **1-γ** times the Q-value for  **π** of **(s,a)** such that we can use the model for policy evaluation given arbitrary consequentialist rewards. Similar to how single-step models (0-models) can be rolled out to obtain (less accurate) multi-step models, sequential rollouts of a γ-model can be reweighed to obtain a γ-model with larger **γ**. While this introduces some error, it reduces the bootstrap error during training, which grows with **γ**. Being able to interpolate between rollouts of single-step models that accumulate error during testing and models with large **γ** that accumulate error during training allows us to find a sweet spot between the two extremes.

In practice, single-step models are often used for model-based value expansion (MVE), where only **N** steps are rolled out and a value function is used for evaluating longer-term consequences. The authors' algorithm, γ-MVE instead uses **N** rollouts of the γ-model and adjusts the weighing of the value function accordingly. γ-MVE performs strongly both in terms of sample efficiency and final performance on a set of low-dimensional continuous control tasks.


Putting less weight on updating values with fluctuating targets seems like a good idea. As the approach does not require much additional compute if weights are shared for the **Q**-network and the network estimating the bound, and as it seems quite orthogonal to previous improvements to methods based on **Q**-functions, I would not be surprised if it became somewhat widely used.

Paper: DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction

Does On-Policy Data Collection Fix Errors in Off-Policy Reinforcement Learning?

Aviral Kumar, Abhishek Gupta, Sergey Levine

Q-learning finds the optimal **Q**-function **Q*** by updating our estimate **Q(s,a)** for a state-action pair **(s,a)** to get closer to the immediate reward plus the discounted **Q**-value for the best action **a'** in the next state **s'**. To generate samples, we usually pick actions corresponding to high **Q**-values. In bandit problems where **s'** is always terminal and thus has all **Q**-values at zero, this leads to **corrective feedback**: If we overestimated an actions value, we will pick this action again soon and are quickly able to correct our misconception. In general MDPs, corrective feedback can be a lot weaker as our update of **Q(s,a)** also depends on the **Q**-values for the next state: To get corrective feedback, we need somewhat correct **Q**-values for the next state, but to get these we likely needed good values for the second to next state, etc. This is particularly problematic with function approximation as updating the current state's **Q**-value might lead to a worse estimate for values down the chain. Consequently, we might see convergence to suboptimal **Q**-functions, instable learning, or problems with sparse or noisy rewards.

To deal with this, we would like to first prioritize correct estimates for states near the end of the chain. But in many branching problems, we actually observe these states with the least frequency such that their values are influenced disproportionally by other states' values when function approximation is used. The authors' approach, dubbed DisCor, reweighs the data distribution to account for this: We would like to preferentially sample states for which we expect **Q** to be close to **Q*** after the update and thus give more weight to state-action pairs when we expect the error **|Q*-Q|** to already be small. As we don't know **Q***, we rely on a bound for the error at a state-action pair **(s,a)** equal to the sum of the magnitudes of previous updates down the chain plus the initial error, discounted by the usual discount rate **γ** as we move back in time. Thus, the error in the next state one step ago is discounted by **γ**, the error in the second to next state two steps ago is discounted by **γ** squared and the initial error is discounted by **γ** to the **k**. This bound can be approximated by a neural network using a SARSA-like update rule, for which the influence of the unknown initial error fades for large **k** due to the discounting.

DisCor is evaluated on MetaWorld tasks in both the single and multi-task setting and SAC augmented with DisCor clearly outperforms SAC in many settings. Similar improvements can be observed for DQN on Atari.


This is a paper worth a skim to glimpse at the complexity of today's RL research while noting how little we understand and can predict about the behaviour of our algorithms. A fun game to play here is to go through the graphs in Appendices D through K and arrive at one's own interpretations _before_ comparing them to the authors' in the main text. What was unsatisfying was that often there were contradictory results between environments, meaning there was no insight to be gleaned about what was happening: for instance, value function normalization always helps except in Walker2d where it significantly hurts performance. Such work raises more questions than it answers; perhaps it will motivate future research that fundamentally rethinks our environments and algorithms.

A more mundane, but alignment-relevant observation is that seeing how difficult it is to tune an agent for a task in simulation, and how much hyperparameters may vary across tasks, is weak evidence against powerful sim-to-real transfer performance arising out of the current paradigm of simulators/tasks and algorithms: agents will need to be trained in the real world, spawning associated risks which we may want to avoid.

What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, Olivier Bachem

In what is likely the largest study of on-policy reinforcement learning agents, this work unifies various algorithms into a collection of over 50 design choices, which the authors implement as tunable hyperparameters and systematically investigate how those parameters impact learning across five standard continuous control environments. Specifically, they choose subsets of these hyperparameters in eight experiment themes -- policy losses, network architectures, normalization and clipping, advantage estimation, training setup, timestep handling, optimizers, and regularization. They train thousands of agents for various choices within each theme, for a total of over 250,000 agents.

They present nearly a hundred graphs summarizing their experiments for the reader to make their own conclusions. Their own recommendations include: using the PPO loss, using separate value and policy networks, initializing the last policy layer with x100 smaller weights, using tanh as activation functions, using observation normalization, using generalized advantage estimation with \lambda = 0.9, tuning the number of transitions gathered in each training loop if possible, tuning the discount factor, using the Adam optimizer with a linearly decaying learning rate, among several others.


Getting a good learning signal can be a key challenge with RL. I’m somewhat surprised it took this long for HER to be generalized to arbitrary reward spaces -- it seems like a clear win that shouldn’t have taken too long to discover (though I didn’t think of it when I first read HER).

Generalized Hindsight for Reinforcement Learning

Alexander C. Li, Lerrel Pinto, Pieter Abbeel

[Hindsight Experience Replay](https://arxiv.org/abs/1707.01495) (HER) introduced the idea of _relabeling_ trajectories in order to provide more learning signal for the algorithm. Intuitively, if you stumble upon the kitchen while searching for the bedroom, you can't learn much about the task of going to the bedroom, but you can learn a lot about the task of going to the kitchen. So even if the original task was to go to the bedroom, we can simply pretend that the trajectory got rewards as if the task was to go to the kitchen, and then update our kitchen-traversal policy using an off-policy algorithm.

HER was limited to goal-reaching tasks, in which a trajectory would be relabeled as attempting to reach the state at the end of the trajectory. What if we want to handle other kinds of goals? The key insight of this paper is that trajectory relabeling is effectively an inverse RL problem: we want to find the task or goal for which the given trajectory is (near-)optimal. This allows us to generalize hindsight to arbitrary spaces of reward functions.

This leads to a simple algorithm: given a set of N possible tasks, when we get a new trajectory, rank how well that trajectory does relative to past experience for each of the N possible tasks, and then relabel that trajectory with the task for which it is closest to optimal (relative to past experience). Experiments show that this is quite effective and can lead to significant gains in sample efficiency. They also experiment with other heuristics for relabeling trajectories, which are less accurate but more computationally efficient.


Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement

Benjamin Eysenbach*, Xinyang Geng*, Sergey Levine, Ruslan Salakhutdinov

This paper was published at about the same time as the previous one, and has the same key insight. There are three main differences with the previous paper:

1. It shows theoretically that MaxEnt IRL is the "optimal" (sort of) way to relabel data if you want to optimize the multitask MaxEnt RL objective.
2. In addition to using the relabeled data with an off-policy RL algorithm, it also uses the relabeled data with behavior cloning.
3. It focuses on fewer environments and only uses a single relabeling strategy (MaxEnt IRL relabeling).


What I learned from this paper is that estimating the mean Q-value is not always enough for robustness. By leveraging distributional information, via ensembles or quantiles, these methods can become quite effective at offline DQN. The release of the dataset is also impressive. I think the dataset will have broad applicability to researchers interested in offline RL as well as imitation learning.

An Optimistic Perspective on Offline Reinforcement Learning

Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi

Off-policy reinforcement learning (RL) that can be done using offline-logged interactions is an important aspect of real-world applications. However, most RL algorithms assume that an agent interacts with an online environment or simulator and learns from its own collected experience. Moreover, the authors show that DQN trained offline on its _own_ experience replay buffer has markedly decreased performance on most of the Atari suite. The authors attempt to address this discrepancy by introducing a robust Q-learning algorithm that randomly mixes estimates for particular Q-values. Specifically, by creating convex combinations from an underlying basis of Q-value estimates the authors are able to create a much larger ensemble. This is similar in spirit to dropout in deep learning where connections in the network are randomly turned off. The authors then go on to show that offline DQN is feasible by training this algorithm and other related algorithms on the DQN Replay Dataset and show it has comparable performance to, and occasionally even surpasses, the original RL baselines. The DQN Replay Dataset is released at [https://offline-rl.github.io/](https://offline-rl.github.io/).


The parallelism afforded by evolutionary methods and genetic algorithms might be valuable in an environment where weak compute is plentiful, so it's exciting to see evidence of such methods besting GPU-hungry deep neural networks. However, I wonder how this would do on sparse reward tasks, where the fitness function is almost always uninformative. Finally, while it generalises to settings where there are task-irrelevant distractions, its deliberately sharp self-attention likely leaves it vulnerable to even simple adversarial attacks.

Paper: Neuroevolution of Self-Interpretable Agents

Using Selective Attention in Reinforcement Learning Agents

Recently winning a best paper award at GECCO 2020, this work marks a leap forward in the performance capabilities learned by small agents via evolutionary methods. Specifically, it shows that by jointly learning which small fraction of input to attend to, agents with only thousands of free parameters can be trained by an evolutionary strategy to achieve state-of-the-art performance in vision-based control tasks.

The key pieces include self-attention over input patches, non-differentiable top-K patch selection that affect 'inattentional blindness', and training via CMA-ES. By design, the agent is interpretable as the top-K patches that are selected can be examined. Empirically, the agent has 1000x fewer weights than a competing neural architecture, and the method shows robustness to changes in task-irrelevant inputs, as the agent learns to focus only on task-relevant patches.


Methods like this that can benefit from seeing a lot of action-independent environment observations might be quite important for applying RL to the real world, as this type of data is a lot cheaper to generate. For example, we can easily generate a ton of observations from a factory by equipping workers with cameras, but state-action-next-state triples from a robot interacting with the factory are very costly to obtain.

Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, Rob Fergus

Sample efficiency in RL can be improved by using off-policy methods that can reuse the same sample multiple times and by using self-supervised auxiliary losses that help with representation learning, especially when rewards are sparse. This work combines both approaches by proposing to learn a latent state representation using an autoencoder while jontly training an agent on that latent representation using <@SAC@>(@Soft Actor-Critic: Deep Reinforcement Learning for Robotics@). Previous work in the on-policy case shows a positive effect from propagating Actor-Critic gradients through the encoder to improve the usefulness of the encoding for policy learning. However, this destabilizes training in the off-policy case, as changing the encoding to facilitate the actor also changes the Q-function estimate, which in turn changes the actor's goal and can introduce nonstationarity. This problem is circumvented by only propagating the Q-network's gradients through the encoder while blocking the actor's gradients.

The method strongly outperforms SAC trained on pixels. It also matches the previous state-of-the-art set by model-based approaches on an image-based continuous control task and outperforms them for noisy observations (as these make dynamics models hard to learn). The authors also find that the learnt encodings generalize between tasks to some extent and that reconstructing the true environment state is easier using their latent representation than using a representation obtained by training SAC on pixels directly.


This paper represented real progress in automated Diplomacy, but is still far from human-level. I’ll be pretty interested to see whether we can reach human-level by creating improved self-play algorithms, like the one presented in this paper, and the ones used for Poker and Go, or if we will have to wait for novel, more general reasoning algorithms applied to Diplomacy. Unlike Poker, Diplomacy against multiple human players involves collusion and implicit signalling, even with No Press. It seems possible to me that it is very difficult to become good at modeling those dynamics through self-play alone. If we did get to human-level through self-play, it would make me more optimistic about the extent to which training is likely to be a bottleneck in other domains which require sophisticated models of human behavior.

Learning to Play No-Press Diplomacy with Best Response Policy Iteration

Thomas Anthony*, Tom Eccles*, Andrea Tacchetti, János Kramár, Ian Gemp, Thomas C. Hudson, Nicolas Porcel, Marc Lanctot, Julien Pérolat, Richard Everett, Satinder Singh, Thore Graepel, Yoram Bachrach

Diplomacy is a game with simple rules where 7 players simultaneously move units every turn to capture territory. Units are evenly matched by default, so winning relies on getting support from some players against others. 'No-Press' Diplomacy limits communication between players to only orders submitted to units, removing the complex verbal negotiations that characterize traditional gameplay.

Previous state-of-the-art No-Press Diplomacy methods were trained to imitate human actions after collecting a dataset of 150,000 human Diplomacy games. This paper presents a new algorithmic method for playing No-Press Diplomacy using a policy iteration approach initialized with human imitation. To find better policies, their methods use "best response" calculations, where the best response policy for some player is the policy that maximizes the expected return for that player against opponent policies. Diplomacy is far too large for exact best response calculation, so the paper introduces an approximation, "Sampled Best Response", which
- Uses Monte-Carlo sampling to estimate opponents' actions each turn
- Only considers a small set of actions sampled from each candidate best response policy
- Only tries to make a single-turn improvement to its policy (rather than trying to optimize for the whole rest of the game)
Similar to other policy iteration methods, the paper creates a dataset of games every iteration using its Sampled Best Response method, then trains neural networks to create policy and value functions that predict the actions chosen by Sampled Best Response. To remedy issues where Sampled Best Response continually cycles through the best strategy for the last iteration,  the paper tries several variants of a technique called "Fictitious Play". In the best-performing variant, the policy network is trained to predict the latest Sampled Best Response given explicitly averaged _historical_ opponent and player policies, rather than just the latest policies.

The paper's methods outperform existing algorithmic methods for No-Press Diplomacy on a variety of metrics, but are still fairly few-shot _exploitable_-- at the end of training, the strongest (non-human) exploiter of the final policy wins 48% of the time. They also find that the strongest exploit doesn't change much through training, though few-shot exploitability does decrease from the beginning of training to the end.


Towards Learning Multi-agent Negotiations via Self-Play

While the previous paper introduces other-play to become robust to unknown partners, this paper takes the other approach of simply training an agent that is robust to a wide, diverse population of possible agents. In particular, it studies a self-driving car "zipper merge" environment, and trains an agent to be robust to a variety of rule-based agents, as well as past versions of itself, and finds that this leads to a much more successful merging policy. However, this is evaluated against the population it is trained with, and not against any previously unseen agents.


If you want to keep track of progress in deep RL, probably -- this seems quite likely to become the new set of benchmarks that researchers work on. There's also another example of specification gaming in the post.

Retro Contest

Christopher Hesse, John Schulman, Vicki Pfau, Alex Nichol, Oleg Klimov, and Larissa Schiavo

OpenAI has released Gym Retro, providing an interface to work with video games from SEGA Genesis, which are more complex than the ones from Atari. They want to use these environments to test transfer learning in particular, where the agent may be pretrained on initial levels for as long as desired, and then must learn how to complete a new test level with only 1 million timesteps (~18 hours) of gameplay. (Humans do well with 2 hours of pretraining and 1 hour of play on the test level.)