I really like the potential of Minecraft as a deep RL research environment, and I'm glad that there's finally a benchmark / competition that takes advantage of Minecraft being very open world and hierarchical. The tasks that they define are very challenging; there are ways in which it is harder than Dota (no self-play curriculum, learning from pixels instead of states, more explicit hierarchy) and ways in which it is easier (slightly shorter episodes, smaller action space, don't have to be adaptive based on opponents). Of course, the hope is that with demonstrations of human gameplay, it will not be necessary to use as much compute as was necessary to <@solve Dota@>(@How to Train Your OpenAI Five@).

I also like the emphasis on how to leverage human priors within general learning algorithms: I share the authors' intuition that human priors can lead to significant gains in sample efficiency. I suspect that, at least for the near future, many of the most important applications of AI will either involve hardcoded structure imposed by humans, or will involve general algorithms that leverage human priors, rather than being learned "from scratch" via e.g. RL.

NeurIPS 2019 Competition: The MineRL Competition on Sample Efficient Reinforcement Learning using Human Priors

William H. Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, Manuela Veloso, Phillip Wang

In this challenge which is slated to start on June 1, competitors will try to build agents that obtain a diamond in Minecraft, without using too much environment interaction. This is an incredibly difficult task: in order to make this feasible, the competition also provides a large amount of human demonstrations. They also have a list of simpler tasks that will likely be prerequisites to obtaining a diamond, such as navigating, chopping trees, obtaining an iron pickaxe, and obtaining cooked meat, for which they also collect demonstrations of human gameplay. As the name suggests, the authors hope that the competition will spur researchers into **embedding human priors into general algorithms in order to get sample efficient learning**.


In this era of scaling up compute via parallelism, it was quite surprising to see OpenAI scaling up compute simply by training for almost a year. That feels like one of the last resorts to scale up compute, so maybe we're seeing the limits of the trend identified in <@AI and Compute@>?

Back when OpenAI Five beat a strong team in their <@Benchmark@>(@OpenAI Five Benchmark: Results@), I and a few others predicted that the team would be able to beat Five after playing a few games against it. I think this prediction has been somewhat validated, given that four teams figured out how to beat a much stronger version of the bot. Of course, humans played over 7000 games against Five, not just a few, so this could be that enough random search finds a weakness. Still, I'd expect pros to be able to do this in tens, maybe hundreds of games, and probably this would have been much easier at the time of the Benchmark.

The underlying model here is that Dota has an extremely large space of strategies, and neither Five nor humans have explored it all. However, pros have a better (lower-dimensional) representation of strategy space (concepts like "split-push") that allow them to update quickly when seeing a better opponent. I don't know what it would take to have AI systems learn these sorts of low-dimensional representations, but it seems key to having AI systems that can adapt quickly like humans can.

Vox: AI triumphs against the world’s top pro team in strategy game Dota 2

How to Train Your OpenAI Five

<@OpenAI Five@> has now beaten the Dota world champions 2-0, after training for 8x longer, for a total of 800 petaflop/s-days or 45000 years of Dota self-play experience. During this insanely long training run, OpenAI grew the LSTM to 4096 units, added buybacks to the game, and switched versions twice. Interestingly, they found it hard to add in new heroes: they could bring a few new heroes up to 95th percentile of humans, but it didn't look like they would train fast enough to reach pro level. This could be because the other heroes were already so capable that it was too hard to learn, since the new heroes would constantly be beaten. The resulting team was also able to play cooperatively with humans, even though they had never been trained with humans.

As usual, I like [Alex Irpan's thoughts](https://www.alexirpan.com/2019/04/14/openai-finals.html). On the Dota side, he found Five's reaction times more believable, but was disappointed by the limited hero pool. He also predicted that with [OpenAI Five Arena](https://arena.openai.com/), which allowed anyone to play either alongside Five, or against Five, at least one of the _many_ teams would figure out a strategy that could reliably beat Five. He was right: while Five had a 99.4% win rate, one team was able to beat it [10 times in a row](https://arena.openai.com/#/results), another beat it thrice in a row, and two teams beat it twice in a row.


This is a really interesting mystery with deep RL, and after reading this post I have a story for it. Note I am far from an expert in this field and it's quite plausible that if I read the papers cited in this post I could tell this story is false, but here's the story anyway. As we saw with PreQN earlier in this issue, one of the most important aspects of deep RL is how information about one (s, a) pair is used to generalize to other (s, a) pairs. I'd guess that the benefit from distributional RL is primarily that you get "good representations" that let you do this generalization well. With a tabular representation you don't do any generalization, and with a linear feature space the representation is hand-designed by humans to do this generalization well, so distributional RL doesn't help in those cases.

But why does distributional RL learn good representations? I claim that it provides stronger supervision given the same amount of experience. With normal expected RL, the final layer of the neural net need only be useful for predicting the expected reward, but with distributional RL they must be useful for predicting all of the quantiles of the reward distribution. There may be "shortcuts" or "heuristics" that allow you to predict expected reward well because of spurious correlations in your environment, but it's less likely that those heuristics work well for all of the quantiles of the reward distribution. As a result, having to predict more things enforces a stronger constraint on what representations your neural net must have, and thus you are more likely to find good representations. This perspective also explains why predicting value functions for multiple discount rates helps with Atari, and why adding auxiliary tasks is often helpful (as long as the auxiliary task is relevant to the main task).

The important aspect here is that all of the quantiles are forcing the same neural net to learn good representations. If you instead have different neural nets predicting each quantile, each neural net has roughly the same amount of supervision as in expected RL, so I'd expect that to work about as well as expected RL, maybe a little worse since quantiles are probably harder to predict than means. If anyone actually runs this experiment, please do let me know the result!

Eighteen Months of RL Research at Google Brain in Montreal

One approach to reinforcement learning is to predict the entire distribution of rewards from taking an action, instead of predicting just the expected reward. Empirically, this works better, even though in both cases we choose the action with highest expected reward. This blog post provides an overview of work at Google Brain Montreal that attempts to understand this phenomenon. I'm only summarizing the part that most interested me.

First, they found that in theory, distributional RL performs on par with or worse than standard RL when using either a tabular representation or linear features. They then tested this empirically on Cartpole, and found similar results: distributional RL performed worse when using tabular or linear representations, but better when using a deep neural net. This suggests that distributional RL "learns better representations". So, they visualize representations for RL on the four-room environment, and find that distributional RL captures more structured representations. Similarly this [paper](https://arxiv.org/abs/1902.0686) showed that predicting value functions for multiple discount rates is an effective way to produce auxiliary tasks for Atari.


I really like this post, and agree with almost everything in it. On the imitation learning point, I also found it surprising how well imitation learning worked. Alex suggests that it could be that human data has enough variation that the agent can learn how to recover from incorrect decisions it could make. I think this is a partial explanation at best -- there is a huge combinatorial explosion, it's not clear why you don't need a much larger dataset to cover the entire space. Maybe there are "natural" representations in any realistic complex environment that you start to accurately learn at the level of compute that they're using, and once you have those then imitation learning with sufficient variation can work well.

On the last point about tossing techniques together, I think this might sometimes be worth doing but often may not be. It makes sense to do this with any real task, since that's a test of the technique against reality. (Here StarCraft counts as a "real" task while Atari does not; the criterion is something like "if the task is successfully automated we are impressed regardless of how it is solved".) I'm less keen on tossing techniques together for artificial benchmarks. I think typically these techniques improve the sample efficiency by a constant multiplicative factor by adding something akin to a good inductive bias; in that case throwing them together may let us solve the artificial benchmark sooner but it doesn't give us great evidence that the "inductive bias" will be good for realistic tasks. I think I don't actually disagree with Alex very much on the object-level recommendations, I would just frame them differently.

An Overdue Post on AlphaStar

The [first post](https://www.alexirpan.com/2019/02/22/alphastar.html) in this two-parter talks about the impact of [AlphaStar](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/) ([AN #43](https://mailchi.mp/768a8130013f/alignment-newsletter-43)) on the StarCraft community and broader public. I'm focusing on the second one, which talks about AlphaStar's technical details and implications. Some of this post overlaps with my summary of AlphaStar, but those parts are better fleshed out and have more details.

First, imitation learning is a surprisingly good base policy, getting to the level of a Gold player. It's surprising because you might expect the [DAgger](https://www.ri.cmu.edu/pub_files/2011/4/Ross-AISTATS11-NoRegret.pdf) problem to be extreme: since there are so many actions in a StarCraft game, your imitation learning policy will make some errors, and those errors will then compound over the very long remainder of the episode as they take the policy further away from normal human play into states that the policy wasn't trained on.

Second, population-based training is probably crucial and will be important in the future, because it allows for exploring the full strategy space.

Third, the major challenge is making RL achieve okay performance, and after that they very quickly become great. It took years of research to get Dota and StarCraft bots reach decent play, and then a few days of more training got them to be world class. Fun quote: "although OpenAI’s DotA 2 agent lost against a pro team, [they were able to beat their old agent 80% of the time with 10 days of training](https://twitter.com/openai/status/1037765547427954688)".

Fourth, there were a lot of research results that went into AlphaStar. This suggests that there are large gains to be had by throwing a lot of techniques together and seeing how well they work, which doesn't happen very much currently. There are good reasons for this: it's much easier to evaluate a technique if its built upon a simple, standard algorithm rather than having to consider all of its interactions with other techniques which you may or may not be able to properly compare against. Still, there are going to be some cool results that we could do now if we just threw the right things together, and this sort of work also lets us test techniques in new settings to see which ones actually work in general, as opposed to only in the original evaluation.


Introducing PlaNet: A Deep Planning Network for Reinforcement Learning

Danijar Hafner, Student Researcher, Google AI


Soft Actor-Critic: Deep Reinforcement Learning for Robotics

Tuomas Haarnoja, Student Researcher and Sergey Levine, Faculty Advisor, Robotics at Google


This is an interesting paper, with a good balance of intuitive motivations, theoretical proofs, and empirical results. While it's not directly safety-related, the broad direction of combining imitation learning and reinforcement learning seems like it might have promise. Relatedly, I wish the authors had discussed in more depth what assumptions can or should be made about the source of batch data. For example, BCQ would presumably perform worse than DQN when data is collected from an expert trying to minimise reward, and (from the paper’s experiments) performs worse than behavioural cloning when data is collected from an expert trying to maximise reward. Most human data an advanced AI might learn from is presumably somewhere in between those two extremes, and so understanding how well algorithms like BCQ would work on it may be valuable.

Off-Policy Deep Reinforcement Learning without Exploration

Scott Fujimoto, David Meger, Doina Precup

This paper discusses off-policy batch reinforcement learning, in which an agent is trying to learn a policy from data which is not based on its own policy, and without the opportunity to collect more data during training. The authors demonstrate that standard RL algorithms do badly in this setting because they give unseen state-action pairs unrealistically high values, and lack the opportunity to update them. They  proposes to address this problem by only selecting actions from previously seen state-action pairs; they prove various optimality results for this algorithm in the MDP setting. To adapt this approach to the continuous control case, the authors train a generative model to produce likely actions (conditional on the state and the data batch) and then only select from the top n actions. Their batch-conditional q-learning algorithm (BCQ) consists of that generative model, a perturbation model to slightly alter the top actions, and a value network and critic to perform the selection. When n = 0, BCQ resembles behavioural cloning, and when n -> ∞, it resembles Q-learning. BCQ with n=10 handily outperformed DQN and DDPG on some Mujoco experiments using batch data.


This is probably the easiest way to get a robot to do interesting things, since you just need it to collect experience autonomously with very little human involvement, you don't need to have good object detection, and in many cases goal specification can be done without too much effort. I'm surprised that using random actions is enough -- how does the robot get enough examples of picking up an object with random actions? Maybe the robot's random strategy is actually coded up in such a way that it is particularly likely to do interesting things like picking up an object.

It does seem like this approach will need something else in order to scale to more advanced capabilities, especially hierarchical tasks -- for example, you'll never have an example of picking up a napkin, getting it wet, and wiping down a table. But perhaps we can iterate the process, where after we learn how to grasp and push, we start collecting data again using grasping and pushing instead of random low-level actions. Safe exploration would become more of a concern here.

Visual Model-Based Reinforcement Learning as a Path towards Generalist Robots

Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine

How can we get general robots that can perform a diverse array of tasks? We could collect a lot of data from robots acting randomly, train a dynamics model on pixels, and then use model-predictive control to plan. The dynamics model is a neural net trained to predict the next image given the current image and action. It helps to use temporal skip connections, because this allows the robot to get some object permanence since it can now "remember" objects it saw in the past that are currently blocked by something else. Model predictive control then samples sequences of actions (called plans), predicts the final image achieved, chooses the plan that best achieves the goal, and takes the first action of that plan. This is then repeated to choose subsequent actions. (Their method is slightly more sophisticated but this is the basic idea.) We can specify the goal by choosing a particular pixel and asking that the object at that pixel be moved to some other pixel. Alternatively, [Few-Shot Goal Inference for Visuomotor Learning and Planning](https://arxiv.org/abs/1810.00482) ([AN #27](https://mailchi.mp/0212425e5544/alignment-newsletter-27)) trains a classifier that can take a few demonstrations and output a goal.


AlphaZero: Shedding new light on the grand games of chess, shogi and Go

David Silver, Thomas Hubert, Julian Schrittwieser, and Demis Hassabis

If you didn't already believe that AlphaZero is excellent at Go, Chess and Shogi, this post and the associated paper show it more clearly with a detailed evaluation. A few highlights:
- AlphaZero can beat Stockfish starting from common human openings, suggesting that it generalizes well
- The amount of computation given to AlphaZero to choose a move has a larger effect on the win probability than I was expecting
- I always wondered why they use MCTS and not alpha-beta search. They speculate that alpha-beta search with a neural net evaluation function succumbs to the [winner's curse](https://www.investopedia.com/terms/w/winnerscurse.asp) since alpha-beta involves a lot of maxes and mins, whereas MCTS averages over evaluations and so is more robust. In contrast, evaluation functions designed by humans are much more likely to generalize well, and alpha-beta outperforms MCTS.


International Conference on Artificial Intelligence in Education 2018

Learning Actionable Representations with Goal-Conditioned Policies


I like how this demonstrates the faults of our current QA systems -- for example, instead of understanding the semantic content of a question, they instead focus on terms that are repeated multiple times. In fact, this might be a great way to tell whether our systems are "actually understanding" the question (as opposed to, say, learning a heuristic of searching for sentences with similar words and taking the last noun phrase of that sentence and returning it as the answer). For a good QA system, one would hope that the optimal question reformulation is just to ask the same question again. However, this won't work exactly as stated, since the RL system could learn the answers itself, which could allow it to "reformulate" the question such that the answer is obvious, for example reformulating "In what year did India gain independence?" to "What is 1946 + 1?" Unless the QA system is perfectly optimal, there will be some questions where the RL system could memorize the answer this way to improve performance.

Open Sourcing Active Question Reformulation with Reinforcement Learning

Michelle Chen Huebscher and Rodrigo Nogueira

Given a question-answering (QA) system, we can get better performance by reformulating a question into a format that is better processed by that system. (A real-world example is [google-fu](https://en.wiktionary.org/wiki/Google-fu), especially several years ago when using the right search terms was more important.) This blog post and accompanying paper consider doing this using reinforcement learning -- try a question reformulation, see if gives a good answer, and if so increase the probability of generating that reformulation. For this to work at all, the neural net generating reformulations has to be pretrained to output sensible questions (otherwise it is an _extremely_ sparse reward problem). They do this by training an English-English machine translation system. The generated reformulations are quite interesting -- 99.8% start with "what is name", and many of them repeat words. Presumably the repetition of words is meant to tell the underlying QA system that the word is particularly important.


This is a fairly mathematical paper and I didn't entirely follow the proofs, so I'm not sure how dependent they are on the particular choice of objective function. However, the empirical results using that objective seem very impressive, and significantly outperform alternative methods of learning representations.

Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine

This paper discusses the use of learned representations in hierarchical RL. In the setting where a higher-level policy chooses goals which lower-level policies are rewarded for reaching, how bad is it when the goal representation isn't able to express all possible states? The authors define a metric for a representation's lossiness based on how close to optimal the policies which can be learned using that representation are, and prove that using a certain objective function, representations with bounded lossiness can be learned. They note a similarity between this objective function and those of mutual information estimators.

The authors test their learner on the MuJoCo Ant Maze environment, achieving compelling results.


This was a great overview of the state of the field. A recurring theme that the authors highlight is that offline RL requires counterfactual reasoning which may be fundamentally difficult to achieve because of distributional shift. Some results shown in the paper suggest that offline RL may just be fundamentally hard. However, I find myself sharing optimism with the authors on the subject of policy constraint techniques and the inevitable importance of better datasets.

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Sergey Levine, Aviral Kumar, George Tucker, Justin Fu

The authors in this paper give an overview of offline-reinforcement learning with the aim that readers gain enough familiarity to start thinking about how to make contributions in this area. The utility of a fully offline RL framework is significant: just as supervised learning methods have been able to utilize data for generalizable and powerful pattern recognition, offline RL methods could enable data to be funneled into decision-making machines for applications such as health-care, robotics, and recommender systems. The organization of the article is split into a section on formulation and another on benchmarks, followed by a section on applications and a general discussion.

In the formulation portion of the review, the authors give an overview of the offline learning problem and then discuss a number of approaches. Broadly speaking, the biggest challenge is the need for counterfactual reasoning because the agent must learn using data by another agent. Thus, the agent is forced to reason about what would happen if a different decision was used. Importance sampling, approximate dynamic programming, and offline model-based approaches are discussed as possible approaches to this counterfactual reasoning problem. In the benchmarks section, the authors review evaluation techniques for offline RL methods. While the authors find that there are many domain-specific evaluations, general benchmarking is less well established. A major issue in creating benchmarks is deciding whether or not to use diverse trajectories/replay buffer data, or only the final expert policy.

In the discussion, the authors argue that while importance sampling and dynamic programming work on low-dimensional and short-horizon tasks, they struggle to integrate well with function approximators. On the other hand, the authors see approaches that constrain the space of policies to be near the dataset as a promising direction to mitigate the effects of distributional shift. However, the authors acknowledge that it may ultimately take more systematic datasets to push the field forward.


NeurIPS 2019 Competition and Demonstration Track

I’m not sure that the path to general intelligence needs to go through replicating embodied animal intelligence. Nonetheless, I really like this benchmark, because its evaluation setup involves new, unseen tasks in order to prevent overfitting, and because of its focus on learning multiple different skills. These features seem important for RL benchmarks regardless of whether we are replicating animal intelligence or not.

Building Thinking Machines by Solving Animal Cognition Tasks

The Animal-AI Testbed and Competition

Matthew Crosby, Benjamin Beyret, Murray Shanahan, José Hernández-Orallo, Lucy Cheke, Marta Halina

The Animal-AI testbed tests agents on the ability to solve the sorts of tasks that are used to test animal cognition: for example, is the agent able to reach around a transparent obstacle in order to obtain the food inside. This has a few benefits over standard RL environments:

1. The Animal-AI testbed is designed to test for specific abilities, unlike environments based on existing games like Atari.
2. A single agent is evaluated on multiple hidden tasks, preventing overfitting. In contrast, in typical RL environments the test setting is identical to the train setting, and so overfitting would count as a valid solution.

The authors ran a competition at NeurIPS 2019 in which submissions were tested on a wide variety of hidden tasks. The winning submission used an iterative method to design the agent: after using PPO to train an agent with the current reward and environment suite, the designer would analyze the behavior of the resulting agent, and tweak the reward and environments and then continue training, in order to increase robustness. However, it still falls far short of the perfect 100% that the author can achieve on the tests (though the author is not seeing the tests for the first time, as the agents are).


Combining Deep Reinforcement Learning and Search for Imperfect-Information Games

Noam Brown*, Anton Bakhtin*, Adam Lerer, Qucheng Gong

<@AlphaZero@>(@AlphaZero: Shedding new light on the grand games of chess, shogi and Go@) and its predecessors have achieved impressive results in zero-sum two-player perfect-information games, by using a combination of search (MCTS) and RL. This paper provides the first combination of search and deep RL for _imperfect-information_ games like poker. (Prior work like <@Pluribus@>(@Superhuman AI for multiplayer poker@) did use search, but didn’t combine it with deep RL, instead relying on significant expert information about poker.)

The key idea that makes AlphaZero work is that we can estimate the value of a state independently of other states without any interaction effects. For any given state s, we can simulate possible future rollouts of the game, and propagate the values of the resulting new states back up to s. In contrast, for imperfect information games, this approach does not work since you cannot estimate the value of a state independently of the policy you used to get to that state. The solution is to instead estimate values for _public belief states_, which capture the public common knowledge that all players have. Once this is done, it is possible to once again use the strategy of backing up values from simulated future states to the current state, and to train a value network and policy network based on this.


Before I read this, I thought of curricula as 'hacks' used to improve training. However, the authors' presentation of connections with transfer learning and experience replay has significantly changed my opinion. In particular, the phrasing of curriculum learning as a kind of 'meta-MDP seems particularly interesting to me. Moreover, there seem to be interesting challenges in this field. One such challenge is that there does not seem to be a great amount of theory about *why* curricula work which could indicate a point of departure for people interested in safety research. Knowing more about theory could help answer safety questions. For example, how do we design curricula so that we can guarantee/check the agent is behaving correctly at each step?

Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, Peter Stone

For a variety of learning problems, the training process is organized so that new concepts and tasks leverage previously learned information. This can serve as a broad definition of curriculum learning. This paper gives an overview of curriculum learning and a framework to organize various approaches to the curriculum learning problem. One central difficulty is that there is a broad class of methods that can be considered curricula. At one extreme, we have curricula where new tasks are created to speed up learning. At another extreme, some curricula simply reorder experience samples. For example, the prioritized replay buffer is one such reordering method. Thus, to cover as much of the literature as possible the authors outline a framework for curriculum learning and then use that structure to classify various approaches. In general, the definition, learning, construction, and the evaluation of curricula are all covered in this work. This is done by breaking the curriculum learning problem into three steps: task generation, sequencing, and transfer learning. Using this problem decomposition the authors give an overview of work addressing each component.


I am a bit curious how they removed observations from the oracle agent, given that you usually have to keep the structure of the input to a neural net constant. Perhaps they simply zeroed out the observations they didn't want?

Suphx: Mastering Mahjong with Deep Reinforcement Learning

Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie-Yan Liu, Hsiao-Wuen Hon

Mahjong is a large imperfect information game with complex rules where turn order can be interrupted. This makes it challenging to solve with existing techniques like MCTS and counterfactual regret minimization. This paper details what was necessary to build _Suphx_, an AI system that is stronger than 99.99% of humans. Some highlights:

- Like the original AlphaGo, they first learned from human gameplay and then finetuned using reinforcement learning, with deep CNNs as their models. They learned both action models as well as value models. They added an entropy bonus to ensure that the policy remained stochastic enough to continue learning over the course of RL.
- They have _five_ learned action models, corresponding to five different decisions that need to be made in Mahjong, as well as a rule-based system for deciding whether or not to declare a winning hand.
- To handle imperfect information, they first train an _oracle agent_ that gets access to all information, and then slowly reduce the amount of information that it gets to observe.
- They could use search to improve the performance online, but did not do so in their evaluation (since Suphx was playing on a website with time constraints). Suphx with search would probably be significantly stronger.


Mastering Complex Control in MOBA Games with Deep Reinforcement Learning

Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin, Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, Lanxiao Huang

This paper presents an AI system that can play the Multi-player Online Battle Arena (MOBA) game _Honor of Kings_. They are inspired by <@OpenAI Five@> (and Honor of Kings sounds quite similar to Dota, though it is 1v1 instead of 5v5), and have a similar learning setup: reinforcement learning using PPO. Their architecture requires an off-policy algorithm (I’m not sure why, maybe they have stale parameters across their rollout servers), so they add an importance sampling correction to the PPO objective, as well as an additional type of gradient clipping. The input is a combination of the image and underlying game state info. The resulting agents are able to beat top human players, and in an event with the public, the AI system lost only 4 out of 2100 matches. Unlike OpenAI Five, this required only around 100 hours to train (though it’s unclear how much compute was used).


For comparison, I think I get around 17-22 when playing with new players, out of a max of 25, so 15.75 is quite a healthy score given that it doesn't use _any_ human data. That being said, it seems hard to use this method in other settings -- even in the relatively simple <@Overcooked environment@>(@Collaborating with Humans Requires Understanding Them@), there aren't any obvious symmetries to use for such training. Perhaps future work will allow us to find approximate symmetries in games somehow, that we can then train to be robust to?

"Other-Play" for Zero-Shot Coordination

Hengyuan Hu, Adam Lerer, Alex Peysakhovich, Jakob Foerster

How can we build AI systems that can _coordinate_ with humans? While <@past@>(@Collaborating with Humans Requires Understanding Them@) <@work@>(@Learning Existing Social Conventions via Observationally Augmented Self-Play@) has assumed access to some amount of human data, this paper aims to coordinate _without any human data at all_, which they call _zero-shot coordination_. In order to develop an algorithm, they assume that their partner is also "trained" for zero-shot coordination.

Their key idea is that in zero-shot coordination, since you can't break symmetries by agreeing upon a protocol in advance (i.e. you can't agree on things like "we'll drive on the left, not the right"), you need a policy that is _robust to relabelings that preserve these symmetries_. This is easy to train for: you just train in self-play, but randomly relabel the states, actions and observations separately for each side in a way that preserves the MDP structure (i.e. uses one of the symmetries). Thus, each side must play a policy that works well _without knowing how the other agent's observations and actions have been relabeled_. In practice, for an N-player game you only need to randomize N-1 of the relabelings, and so in the two player games they consider they only randomly relabel one side of the self-play.

They evaluate this in Hanabi (where the game is invariant to relabeling of the colors), and show that the resulting agents are better at playing with other agents trained on different seeds or with slightly different architectures, and also that they play better with humans, achieving an average score of 15.75 with non-expert human players, compared to 9.15 for agents trained via regular self-play.


This solution seems stunningly obvious in retrospect. While the authors informally report that their approach improves robustness to replacing other agents by humans, the example they give seems to indicate that this is because search prevents obvious mistakes in novel situations induced by human behaviour. Thus, I still expect (implicit) <@human models@>(@Thoughts on Human Models@) to be a vital component of human-machine cooperation.

Paper: Improving Policies via Search in Cooperative Partially Observable Games

Building AI that can master complex cooperative games with hidden information

Adam Lerer, Hengyuan Hu, Jakob Foerster, Noam Brown

This paper improves on the state of the art for AI agents playing <@Hanabi@>(@The Hanabi Challenge: A New Frontier for AI Research@), a cooperative multiplayer game that is challenging because of distributed hidden information and restricted communication.

The approach works by improving a baseline policy using search. In the simplest case, only one agent performs search while all other agents follow a fixed policy, such that the problem is reduced to search in a POMDP. This alone leads to relevant improvements, even when the search is very shallow. The fixed policies help because they allow the searching agent to correctly update its belief about hidden information when it sees other agents behaving (as it knows how other agents would behave given different states of the hidden information). This idea can be generalized to the case where all agents perform search by letting the agents simulate each other's search process. This can get expensive quickly as agent A's beliefs in the second round also depend on agent B's search process in counterfactual scenarios in the first round, such that agent B's search in round two also has to simulate these counterfactuals.  A computation budget is introduced to make this computationally feasible and all agents know that the other agents will only use search in a turn if the cost of this is below the budget.

As search can be performed on top of any policy and allows to leverage compute during inference, not just training, it nicely complements more direct approaches using deep RL, which is a theme that has also been observed in Go and Poker.


Learning to Explore with Meta-Policy Gradient


Simple random search provides a competitive approach to reinforcement learning


Thinking Fast and Slow with Deep Learning and Tree Search


Learning by Playing


Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning


CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Aravind Srinivas*, Michael Laskin*, Pieter Abbeel

This paper applies contrastive learning (discussed above) to reinforcement learning. In RL, rather than training in an initial unsupervised phase, the contrastive learning happens alongside the RL training, and so serves as an auxiliary objective to speed up learning. They use random crops for their data augmentation.


I'd be interested in seeing a variant on CURL where the weight for the contrastive loss decays over time: if the author's speculation is correct, this should mitigate the problem with CURL, and one would hope that it would then be better than RAD.

Reinforcement Learning with Augmented Data

Michael Laskin*, Kimin Lee*, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas

While CURL (summarized above) applies contrastive learning in order to ensure the network is invariant to specific data augmentations, we can try something even simpler: what if we just run a regular RL algorithm on augmented observations (e.g. observations that have been randomly cropped)? The authors term this approach RAD (RL with Augmented Data), and find that this actually _outperforms_ CURL, despite not using the contrastive learning objective. The authors speculate that CURL is handicapped by using the contrastive loss as an auxiliary objective, and so its representations are forced to be good both for the true task and for the contrastive prediction task, whereas RAD only trains on the true task.


Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

Ilya Kostrikov, Denis Yarats, Rob Fergus

This paper applies data augmentation to Q-learning algorithms, again without a contrastive loss. Specifically, they suggest that the Q-values of states should be invariant to data augmentations (e.g. random translations, which is what they use), and so any time we need to estimate a Q-value, we can reduce the variance of this estimate by sampling multiple data augmentations of the state, and averaging the predicted Q-values for each of them. They apply this to Soft Actor-Critic (SAC) and find that it significantly improves results.


This is a cool proof of concept, but the learned perturbation policy can only take you so far -- no learned perturbation policy is going to allow you to e.g. pick up an object after it is dropped, as you would want if you're training a robot to <@manipulate a Rubik's cube@>(@Solving Rubik’s Cube with a Robot Hand@). It seems hard to overcome this sort of problem in a fully automated and learned way (though perhaps you could use more classical techniques to have a "hardcoded" but still automated reset policy).

Paper: The Ingredients of Real World Robotic Reinforcement Learning

Henry Zhu*, Justin Yu*, Abhishek Gupta*, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, Sergey Levine

Suppose we wanted to train a robot to perform a task in the real world, and we didn't want to deal with the headache of sim-to-real transfer. Typically, since all of our experience must be collected in the real world, we would need a human to reset the robot to its initial state. The key idea of this paper is that the point of resets is to ensure that the robot explores a diversity of states causing it to learn a robust policy; this can be achieved by learning a _perturbation policy_ whose objective is to take the robot to states it hasn't visited before. They then combine this with representation learning (so that they can learn from pixels) and use a classifier that distinguishes goal states from non-goal states as the reward function, to get a fully automated setup where once you start the robot's training, it trains itself without any human in the loop.


Given how compute intensive deep RL is, I think it is quite useful to enable cheaper and faster training before these algorithms can be broadly useful. Their claimed speedup is quite impressive, and I like how well they can separate the training and inference from the simulation. I expect that specialized hardware for both training and inference will soon become the norm and SEED RL seems like it will scale well as those accelerators become faster. One thing to note is that this architecture seems very specifically tuned to the problem of games where CPUs can efficiently simulate the environment and it does not improve the sample efficiency for situations where we can’t run lots of simulations.

**Rohin's opinion:** It was quite surprising to me that this worked as well as it did: this model requires communication across machines _at every timestep of the environment_, which intuitively means that latency should be a major bottleneck, while the standard model only requires communication once per batch of trajectories.

Paper: SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference

Massively Scaling Reinforcement Learning with SEED RL

Lasse Espeholt*, Raphaël Marinier*, Piotr Stanczyk*, Ke Wang, Marcin Michalski

Deep learning has <@historically@>(@AI and Compute@) seen many improvements as a result of scaling to larger models with larger amounts of computation, as with the months-long training of <@OpenAI Five@>(@Dota 2 with Large Scale Deep Reinforcement Learning@) and <@AlphaStar@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@). SEED RL redesigns the architecture of distributed RL to enable better machine utilization and communication and achieves an order of magnitude improvement in training speed.

Current distributed architectures typically separate machines into *actors* and *learners*. *Actors* are typically CPUs that simulate the environment, and run inference to predict agent actions. They then send *trajectories* to the *learners*. *Learners* are typically accelerators (GPUs or TPUs), which are responsible for training the model. They then send the updated model parameters to the *actors*.

SEED RL addresses 3 main issues in this setup:
1. Inference could benefit from specialized accelerators
2. Sending model parameters and states requires high bandwidth.
3. Environment simulation and inference are very different tasks and having them on the same machine makes it hard to utilize the resource efficiently.

The solution is to instead have actors **only** simulate the environment. After each step, they send the resulting observation to the *learner*, which is responsible for both training and inference, possibly split on separate hardware). It then sends back just the actions to the environment. This enables each piece of hardware to be used for its designed purpose. Since they now need to communicate at each step, they use gRPC to minimize latency.


This is a cool improvement to domain randomization: it seems like it should be distinctly easier to learn a policy that is dependent on the hidden parameters, and that seems to come at the relatively low cost of needing just a little real world data.

Paper: Learning Agile Robotic Locomotion Skills by Imitating Animals

Robots Learning to Move like Animals

Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, Sergey Levine

<@Previous work@>(@Learning Acrobatics by Watching YouTube@) has suggested that we can get good policies by estimating and imitating poses. This work takes this idea and tries to make it work with sim-to-real transfer. Domain randomization would result in a policy that must be robust to all the possible values of the hidden parameters (such as friction). To make the problem easier, they do domain randomization, but give the agent access to (a latent representation of) the hidden parameters, so that its policy can depend on the hidden parameters. Then, to transfer to the real world, they simply need to search over the latent representation of the hidden parameters in order to find one where the policy actually works in the real world. In practice, they can adapt to the real world with just 8 minutes of real world data.


The GVG-AI framework seems like a useful tool to explore learning videogames. Setting up curriculum learning by using PPCG is also a clever idea. However, the results are a bit mixed. On two of the games they tested, training on a single difficult level works better than training on a variety of levels for generalization. Having said this, the method can learn the game Frogs (57% win rate) while DQN/A2C make zero progress even after 40 million steps. It seems as though certain conditions make PPCG a good method to use. It'd be interesting to investigate what those conditions are in a future publication.

Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation

Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, Sebastian Risi

Deep reinforcement learning has been able to use high-dimensional input, such as images, to learn optimal policies. However, when neural networks are trained in a fixed environment, such as on a single level in a video game, they will usually over-fit and fail to generalize to new levels. This paper uses procedurally generated levels during training in an attempt to increase the generality of deep RL. They make use of the General Video Game AI framework (GVG-AI) which allows rapid design of video games through the specification of rewards, objects, etc. Moreover, they introduce Progressive PCG (PPCG) to smoothly control the difficulty of generated levels to build a curriculum for the agent. The authors show that for some games procedural level generation enables generalization to new levels within the same distribution.


Meta-Gradient Reinforcement Learning

Zhongwen Xu, Hado van Hasselt, David Silver


This approach fits neatly with older work including ["Learning to Reach Goals”](https://pdfs.semanticscholar.org/6df4/3f70f383007a946448122b75918e3a9d6682.pdf?_ga=2.78217424.59277899.1578587972-37524897.1574936748) and more recent work such as [Hindsight experience replay](https://arxiv.org/pdf/1707.01495.pdf) and [Goal-Conditioned Policies](https://arxiv.org/pdf/1912.06088.pdf). In particular, all of these methods seem to be effective at addressing the difficulty that comes with working with sparse rewards. I also found myself justifying the utility of selecting the objective of 'learning to achieve general goals' to be related to the idea that <@seeking power is instrumentally convergent@>(@Seeking Power is Provably Instrumentally Convergent in MDPs@).

**Rohin's opinion:** Both this and the previous paper have explored the idea of conditioning on rewards and predicting actions, trained by supervised learning. While this doesn't hit state-of-the-art performance, it works reasonably well for a new approach.

[Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions](http://arxiv.org/abs/1912.02875) and [Training Agents using Upside-Down Reinforcement Learning](http://arxiv.org/abs/1912.02877)

It's a common understanding that using supervised learning to solve RL problems is challenging because supervised learning works directly with error signals while RL only has access to evaluation signals. The approach in these papers introduce 'upside-down' reinforcement learning (UDRL) as a way to bridge this gap. Instead of learning how to predict rewards, UDRL learns how to take actions when given a state and a desired reward. Then, to get good behavior, we simply ask the policy to take actions that lead to particularly high rewards. The main approach is to slowly increase the desired goal behavior as the agent learns in order to maximize agent performance. The authors evaluate UDRL on the Lunar Lander and the Take Cover environments. UDRL ultimately performs worse on Lunar Lander and better on Take Cover so it's unclear whether or not UDRL is an improvement over popular methods. However, when rewards are made to be sparse UDRL is able to significantly outperform other RL methods.


The presentation of the paper is clear. In particular, the idea of planning a sequence of maximally feasible subgoals seems particularly intuitive. In general, I think that LEAP relies on the clever idea of reusing trajectory data to augment the data-set for the goal-conditioned policy. As the authors noted, the question of exploration was mostly neglected. I wonder how well the idea of reusing trajectory data generalizes to the general exploration problem.

**Rohin's opinion:** The general goal of inferring hierarchy and using this to plan more efficiently seems very compelling but hard to do well; this is the goal in most hierarchical RL algorithms and <@Learning Latent Plans from Play@>.

Planning with Goal-Conditioned Policies

Soroush Nasiriany*, Vitchyr H. Pong*, Steven Lin, Sergey Levine

Reinforcement learning can learn complex skills by interacting with the environment. However, temporally extended or long-range decision-making problems require more than just well-honed reactions. **In this paper, the authors investigate whether or not they can obtain the benefits of action planning found in model-based RL without the need to model the environment at the lowest level.** The authors propose a model-free planning framework that learns low-level goal-conditioned policies that use their value functions as implicit models. Goal-conditioned policies are policies that can be trained to reach a goal state provided as an additional input. Given a goal-conditioned policy, the agent can then plan over intermediate subgoals (goal states) using a goal-conditioned value function to estimate reachability. Since the state space is large, the authors propose what they call latent embeddings for abstracted planning (LEAP), which is able to find useful subgoals by first searching a much smaller latent representation space and then planning a sequence of reachable subgoals that reaches the target state. In experiments, LEAP significantly outperforms prior algorithms on 2D navigation and push/reach tasks. Moreover, their method can get a quadruped ant to navigate around walls which is difficult because much of the planning happens in configuration space. This shows that LEAP is able to be extended to non-visual domains.


The extension from PlaNet (adding actor-critic rather than direct single-step reward prediction) is relatively straightforward, but I think latent models are an interesting area - especially if they eventually become at all possible to interpret - and so I'm happy to see more work in this area.

Dream to Control: Learning Behaviors by Latent Imagination

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi

In the past year or so, the idea of learning a transition model in a latent space has gained traction, motivated by the hope that such an approach could combine the best of the worlds of model-free and model-based learning. The central appeal of learning a latent transition model is that it allows you to imagine future trajectories in a potentially high-dimensional, structured observation space without actually having to generate those high-dimensional observations.

Dreamer builds on a prior model by the same authors, <@PlaNet@>(@Learning Latent Dynamics for Planning from Pixels@), which learned a latent representation of the observations, p(s|o), trained both through a VAE-style observation reconstruction loss, and also a transition model q(s-next|s, a), which is trained to predict the state at the next step given only the state at the prior one, with no next-step observation data. Together, these two models allow you to simulate action-conditioned trajectories through latent state space. If you then predict reward from state, you can use this to simulate the value of trajectories. Dreamer extends on this by also training an Actor Critic-style model on top of states to predict action and value, forcing the state representation to not only capture next-step transition information, but also information relevant to predicting future rewards. The authors claim this extension makes their model more able to solve long-horizon problems, because the predicted value function can capture far-future rewards without needing to simulate the entire way there. Empirically, there seems to be reasonable evidence that this claim plays out, at least within the fairly simple environments the model is tested in.


Gym Retro, again

OpenAI is releasing the full version of Gym Retro, with over a thousand games, and a tool for integrating new games into the framework. And of course we see new games in which RL agents find infinite loops that give them lots of reward -- Cheese Cat-Astrophe and Blades of Vengeance.


GECCO '18 Proceedings of the Genetic and Evolutionary Computation Conference

Evolutionary Reinforcement Learning


Learning Time-Sensitive Strategies in Space Fortress


I agree that episodic training and fixed world dynamics seem like unlikely conditions for most situations we would expect agents to encounter in the real world. Accounting for them seems particularly important to ensure safe exploration and robustness to distributional shift, and I think that these environments could serve as useful benchmarks for these safety problems as well.

Adaptive Online Planning for Continual Lifelong Learning

Lifelong learning is distinct from standard RL benchmarks because
 1. The environment is *sequential* rather than *episodic*; it is never reset to a new start state.
 2. The current *transition* and *reward* function are given, but they change over time.

Given this setup, there are two basic approaches: first, run model-free learning on simulated future trajectories and rerun it every time the dynamics change, and second, run model-based planning on the current model. If you ignore computational constraints, these should be equivalent; however, in practice, the second option tends to be more computationally efficient. The contribution of this work is to make this more efficient, rather than improving final performance, by starting with the second option and then using model-free learning to “distill” the knowledge produced by the model-based planner allowing for more efficient planning in the future.

Specifically, Adaptive Online Planning (AOP) balances between the model-based planner MPPI (a variant of MPC) and the model-free algorithm TD3. MPPI uses the given model to generate a trajectory up to a horizon and then uses an ensemble of value functions to estimate the cumulative reward. This knowledge is then distilled into TD3 for later use as a prior for MPPI. During future rollouts, the variance and Bellman error of the value function ensemble are used to determine how long the horizon should be, and therefore how much computation is used.


Paper: When to Trust Your Model: Model-Based Policy Optimization

Model-Based Reinforcement Learning: Theory and Practice

Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine

This post provides a broad overview of model-based reinforcement learning, and argues that a learned (explicit) model allows you to generate sample trajectories from the current policy at arbitrary states, correcting for off-policy error, at the cost of introducing model bias. Since model errors compound as you sample longer and longer trajectories, the authors propose an algorithm in which the model is used to sample short trajectories from states in the replay buffer, rather than sampling trajectories from the initial state (which are as long as the task's horizon).


This is one of those 'obvious' ideas that turns out to be very difficult to put into practice. I'm glad to see a paper like this simply because the authors do a good job at explaining why a naive execution of the transformer idea is bound to fail. Overall, the architecture seems to be a solid improvement over the TrXL variant. I'd be curious whether or not the architecture is also better in an NLP setting.

Stabilizing Transformers for Reinforcement Learning

Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, Raia Hadsell

Transformers have been incredibly successful in domains with sequential data. Naturally, one might expect transformers to be useful in partially observable RL problems. However, transformers have complex implementations making them difficult to use in an already challenging domain for learning. In this paper, the authors explore a novel transformer architecture they call Gated Transformer-XL (GTrXL) that can be used in the RL setting. The authors succeed in stabilizing training with a reordering of the layer normalization coupled with the addition of a new gating mechanism located at key points in the submodules of the transformer. The new architecture is tested on DMlab-30, a suite of RL tasks including memory, and shows improvement over baseline transformer architectures and the neural computer architecture MERLIN. Furthermore, GTrXL learns faster and is more robust than a baseline transformer architecture.


Reward Estimation for Variance Reduction in Deep Reinforcement Learning

Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavet, Joelle Pineau


It's safe to say that work towards good evaluation metrics for RL agents is a good thing. I think this paper captures a lot of the notions of what makes an agent 'good' in a way that seems readily generalizable. The evaluation time on the suite is reasonable, no more than 30 minutes per experiment. Additionally, the ability to produce automated summary reports in standard formats is a nice feature. One thing that seems to be missing from the core set of experiments is a good notion of transfer learning capability beyond simple generalization. However, the authors readily note that the suite is a work in progress so I wouldn't doubt something covering that would be introduced in time.

**Rohin's opinion:** The most interesting thing about work like this is what "core issues" they choose to evaluate -- it's not clear to me whether e.g. "memory" in a simple environment is something that future research should optimize for.

Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepezvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, Hado Van Hasselt

Collecting clear, informative and scalable problems that capture important aspects about how to design general and efficient learning algorithms is difficult. Many current environments used to evaluate RL algorithms introduce confounding variables that make new algorithms difficult to evaluate. In this project, the authors assist this effort by introducing Behaviour Suite for Reinforcement Learning (bsuite), a library that facilitates reproducible and accessible research on core issues in RL. The idea of these experiments is to capture core issues, such as 'exploration' or 'memory', in a way that can be easily tested or evaluated. The main contribution of this project is an open-source project called bsuite, which instantiates all experiments in code and automates the evaluation and analysis of any RL agent on bsuite. The suite is designed to be flexible and includes code to run experiments in parallel on Google cloud, with Jupyter notebook, and integrations with OpenAI Gym.


I am a bit torn here: On one hand, the principle makes it plausible for us to find the globally optimal solution by solving our task on a finite set of training environments. This way the generalization problem feels more well-defined and amenable to theoretical analysis, which seems useful for advancing our understanding of reinforcement learning.

On the other hand, I don't expect the principle to hold for most real-world problems. For example, in interactions with other adapting agents performance will depend on these agents' policies, which can be hard to infer and change dynamically. This means that the principle of unchanged optimality won't hold without precise information about the other agent's policies, while this information can be very difficult to obtain.

More generally, with this and some of the criticism of the [AI safety gridworlds](https://deepmind.com/blog/specifying-ai-safety-problems/) that framed them as an ill-defined benchmark, I am a bit worried that too much focus on very "clean" benchmarks might divert from issues associated with the messiness of the real world. I would have liked to see a more conditional conclusion for the paper, instead of a general principle.

The Principle of Unchanged Optimality in Reinforcement Learning Generalization

In image recognition tasks, there is usually only one label per image, such that there exists an optimal solution that maps every image to the correct label. Good generalization of a model can therefore straightforwardly be defined as a good approximation of the image-to-label mapping for previously unseen data.

In reinforcement learning, our models usually don't map environments to the optimal policy, but states in a given environment to the corresponding optimal action. The optimal action in a state can depend on the environment. This means that there is a tradeoff regarding the performance of a model in different environments.

The authors suggest the principle of unchanged optimality: in a benchmark for generalization in reinforcement learning, there should be at least one policy that is optimal for all environments in the train and test sets. With this in place, generalization does not conflict with good performance in individual environments. If the principle does not initially hold for a given set of environments, we can change that by giving the agent more information. For example, the agent could receive a parameter that indicates which environment it is currently interacting with.


This paper seems like one that will make RL researchers usefully uncomfortable,  by pointing out that the complexity of our implementations means that just having a theoretical story of your algorithm's performance and empirical validation of that heightened performance isn't actually enough to confirm that the theory is actually the thing driving the performance. I do think the authors were a bit overly critical at points: I don't think anyone working in RL would have expected that the learned value function was perfect, or that gradient updates were un-noisy. But, it's a good reminder that saying things like "value functions as a baseline decrease variance" should be grounded in an empirical examination of how good they are at it, rather than just a theoretical argument that they should.

Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms?

Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry

This paper investigates whether and to what extent the stated conceptual justifications for common Policy Gradient algorithms are actually the things driving their success. The paper has two primary strains of empirical investigation.

In the first, they examine a few of the more rigorously theorized aspects of policy gradient methods: learned value functions as baselines for advantage calculations, surrogate rewards, and enforcement of a "trust region" where the KL divergence between old and updated policy is bounded in some way. For value functions and surrogate rewards, the authors find that both of these approximations are weak and perform poorly relative to the true value function and reward landscape respectively.

Basically, it turns out that we lose a lot by approximating in this context. When it comes to enforcing a trust region, they show that TRPO is able to enforce a bound on mean KL, but that it's much looser than the (more theoretically justified) bound on max KL that would be ideal but is hard to calculate. PPO is even stranger: they find that it enforces a mean KL bound, but only when optimizations present in the canonical implementation, but not the core definition of the algorithm, are present. These optimizations include: a custom weight initialization scheme, learning rate annealing on Adam, and reward values that are normalized according to a rolling sum. All of these optimizations contribute to  non-trivial increases in performance over the base algorithm, in addition to apparently being central to how PPO maintains its trust region.


I enjoyed this; it's a well-written paper that uses a few core interesting ideas (posterior sampling over a task distribution, representation of a task distribution as a distribution of embedding vectors passed in to condition Q functions), and builds them up to make a method that achieves some impressive empirical results.

Efficient Off-Policy Meta-RL via Probabilistic Context Variables

Learning to Learn with Probabilistic Task Embeddings

Kate Rakelly*, Aurick Zhou*, Deirdre Quillen, Chelsea Finn, Sergey Levine

This paper proposes a solution to off-policy meta reinforcement learning, an appealing problem because on-policy RL is so sample-intensive, and meta-RL is even worse because it needs to solve a distribution over RL problems. The authors' approach divides the problem into two subproblems: infer an embedding, z, of the current task given context, and learning an optimal policy q function conditioned on that task embedding. At the beginning of each task, z is sampled from the (Gaussian) prior, and as the agent gains more samples of that particular task, it updates its posterior over z, which can be thought of as refining its guess as to which task it's been dropped into this time. The trick here is that this subdividing of the problem allows it to be done mostly off-policy, because you only need to use on-policy learning for the task inference component (predicting z given current task transitions), and can learn the Actor-Critic model conditioned on z with off-policy data. The method works by alternating between these two learning modes.


The specific method is an interesting modification of previous Contrastive Predictive Coding work, but what I found most impressive about this paper was the engineering work involved in pulling metadata supervision signals out of the game by reading  comments on disassembled  source code to see exactly how metadata was being stored in RAM. This seems to have the potential of being a useful benchmark for Atari representation learning going forward (though admittedly Atari games are fairly conceptually straightforward to begin with).

Unsupervised State Representation Learning in Atari

Ankesh Anand*, Evan Racah*, Sherjil Ozair*, Yoshua Bengio, Marc-Alexandre Côté, R Devon Hjelm

This paper has two main contributions: an actual technique for learning representations in an unsupervised way, and an Atari-specific interface for giving access to the underlying conceptual state of the game (e.g. the locations of agents, locations of small objects, current remaining lives, etc) by parsing out the RAM associated with each state. Since the notional goal of unsupervised representation learning is often to find representations that can capture conceptually important features of the state without having direct access to them, this supervision system allows for more meaningful evaluation of existing methods by asking how well conceptual features can be predicted by learned representation vectors. The object-level method of the paper centers around learning representations that capture information about temporal state dynamics, which they do by maximizing mutual information between representations at  adjacent timesteps. More specifically, they have both a local version of this, where a given 1/16th patch of the image has a representation that  is optimized to be predictive of that same patches next-timestep representation, and a local-global version, where the global representation is optimized to be predictive of representations of each patch. They argue this patch-level prediction makes their method better at learning concepts attached to small objects, and the empirical results do seem to support this interpretation.


I really like the idea of being able to easily test whether an agent has robustly learned a behavior or not. To some extent, all of the transfer learning environments are also doing this, such as <@CoinRun@>(@Quantifying Generalization in Reinforcement Learning@) and the <@Retro Contest@>: if the learned behavior is not robust, then the agent will not perform well in the transfer environment. But with Toybox it looks like researchers will be able to run much more granular experiments looking at specific behaviors.

Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning

Emma Tosch, Kaleigh Clary, John Foley, David Jensen

Toybox is a reimplementation of three Atari games (Breakout, Amidar and Space Invaders) that enables researchers to customize the games themselves in order to perform better experimental evaluations of RL agents. They demonstrate its utility using a case study for each game. For example, in Breakout we often hear that the agents learn to "tunnel" through the layer of bricks so that the ball bounces around the top of the screen destroying many bricks. To test whether the agent has learned a robust tunneling behavior, they train an agent normally, and then at test time they remove all but one brick of a column and see if the agent quickly destroys the last brick to create a tunnel. It turns out that the agent only does this for the center column, and sometimes for the one directly to its left.


Smoothing Policies and Safe Policy Gradients

Matteo Papini, Matteo Pirotta, Marcello Restelli


This is a great experimental paper, there's a lot of data that can help understand DQL algorithms. I wouldn't take the results too literally, since insights on simple environments may not generalize to more complex environments. For example, they found overfitting to be an issue in their environments -- it's plausible to me that with more complex environments (think Dota/StarCraft, not Mujoco) this reverses and you end up underfitting the data you have. Nonetheless, I think data like this is particularly valuable for coming up with an intuitive theory of how deep RL works, if not a formal one.

Diagnosing Bottlenecks in Deep Q-learning Algorithms

Justin Fu, Aviral Kumar, Matthew Soh, Sergey Levine

While the PreQN paper used a theoretical approach to tackle Deep Q-Learning algorithms, this one takes an empirical approach. Their results:
    - Small neural nets cannot represent Q*, and so have undesired bias that results in worse performance. However, they also have convergence issues, where the Q-function they actually converge to is significantly worse than the best Q-function that they could express. Larger architectures mitigate both of these problems.
    - When there are more samples, we get a lower validation loss, showing that we are overfitting. Despite this, larger architectures are better, because the performance loss from overfitting is not as bad as the performance loss from having a bad bias. A good early stopping criterion could help with this.
    - To study how non-stationarity affects DQL algorithms, they study a variant where the Q-function is a moving average of the past Q-functions (instead of the full update), which means that the target values don't change as quickly (i.e. it is closer to a stationary target). They find that non-stationarity doesn't matter much for large architectures.
    - To study distribution shift, they look at the difference between the expected Bellman error before and after an update to the parameters. They find that distribution shift doesn't correlate much with performance and so is likely not important.
    - Algorithms differ strongly in the distribution over (s, a) pairs that the DQL update is computed over. They study this in the absence of sampling (i.e. when they simply weight all possible (s, a) pairs, rather than just the ones sampled from a policy) and find that distributions that are "close to uniform" perform best. They hypothesize that this is the reason that experience replay helps -- initially an on-policy algorithm would take samples from a single policy, while experience replay adds samples from previous versions of the policy, which should increase the coverage of (s, a) pairs.

To sum up, the important factors are using an expressive neural net architecture, and designing a good sampling distribution. Inspired by this, they design Adversarial Feature Matching (AFM), which like Prioritized Experience Replay (PER) puts more weight on samples that have high Bellman error. However, unlike PER, AFM does not try to reduce distribution shift via importance sampling, since their experiments found that this was not important.


This work follows the standard pattern where model-based RL is more sample efficient but reaches worse final performance compared to model-free RL. Let's try to explain this using the same story as in the rest of this newsletter.

The sample efficiency comes from the fact that they learn a world model that can predict the future, and then use that model to solve the control problem (which has zero sample cost, since you are no longer interacting with the environment). It turns out that predicting the future is "easier" than selecting the optimal action, and so the world model can be trained in fewer samples than it would take to solve the control problem directly. Why is the world model "easier" to learn? One possibility is that solving the control problem requires you to model the world anyway, and so must be a harder problem. If you don't know what your actions are going to do, you can't choose the best one. I don't find this very compelling, since there are lots of aspects of world modeling that are irrelevant to the control problem -- you don't need to know exactly how the background art will change in order to choose what action to take, but world modeling requires you to do this. I think the real reason is that world modeling benefits from much more supervision -- rather than getting a sparse reward signal over a trajectory, you get a full grid of pixels every timestep that you were supposed to predict. This gives you many orders of magnitude more "supervision information" per sample, and so it makes it easier to learn. (This is basically the same argument as in [Yann Lecun's cake analogy](https://medium.com/syncedreview/yann-lecun-cake-analogy-2-0-a361da560dae).)

Why does it lead to worse performance overall? The policy is now being trained using rollouts that are subtly wrong, and so instead of specializing to the true Atari dynamics it will be specialized to the world model dynamics, which is going to be somewhat different and should lead to a slight dip in performance. (Imagine a basketball player having to shoot a ball that was a bit heavier than usual -- she'll probably still be good, but not as good as with a regular basketball.) In addition, since the world model is supervised by pixels, any small objects are not very important to the world model (i.e. getting them wrong does not incur much loss), even if they are very important for control. In fact, they find that bullets tend to disappear in Atlantis and Battle Zone, which is not good if you want to learn to play those games.

I'm not sure if they shared weights between the world model and the control policy. If they did, then they would also have the problem that the features that are useful for predicting the future are not the same as the features that are useful for selecting actions, which would also cause a drop in performance. My guess is that they didn't share weights for precisely this reason, but I'm not sure.

Model-Based Reinforcement Learning for Atari

Simulated Policy Learning in Video Models

Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Miłos, Błażej Osiński, Roy H Campbell, Konrad Czechowski, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker and Henryk Michalewski

This blog post and the associated [paper](https://arxiv.org/abs/1903.00374) tackle model-based RL for Atari. The recent <@world models@>(@Recurrent World Models Facilitate Policy Evolution@) paper proposed first learning a model of the world by interacting with the environment using a random policy, and then using the model to simulate the environment and training a control policy using those simulations. (This wasn't it's main point, but it was one of the things it talked about.) The authors take this idea and put it in an iterative loop: they first train the world model using experience from a random policy, then train a policy using the world model, retrain the world model with experience collected using the newly trained policy, retrain the policy, and so on. This allows us to correct any mistakes in the world model and let it adapt to novel situations that the control policy discovers. This allows them to train agents that can play Atari with only 100K interactions with the environment (corresponding to about two hours of real-time gameplay), though the final performance is lower than the state-of-the-art achieved with model-free RL. See [Import AI](https://jack-clark.net/2019/04/01/import-ai-140-surveilling-a-city-via-the-cityflow-dataset-25000-images-of-chinese-shop-signs-and-the-seven-traps-of-ai-ethics/) for more details.


This is a cool example of using structured knowledge in order to get generalization while also using deep learning in order to get performance. I also recently came across [Residual Reinforcement Learning for Robot Control](https://arxiv.org/abs/1812.03201), which seems to have the same idea of combining deep RL with conventional control mechanisms. I haven't read either of the papers in depth, so I can't compare them, but a _very_ brief skim suggests that their techniques are significantly different.

Unifying Physics and Deep Learning with TossingBot

TossingBot is a system that learns how to pick up and toss objects into bins using deep RL. The most interesting thing about it is that instead of using neural nets to directly predict actions, they are instead used to predict _adjustments_ to actions that are computed by a physics-based controller. Since the physics-based controller generalizes well to new situations, TossingBot is also able to generalize to new tossing locations.


Assessing Generalization in Deep Reinforcement Learning (blog post)

This is a blog post summarizing <@Assessing Generalization in Deep Reinforcement Learning@>.


Firstly, the blog post is very readable and provides a great introduction (it's much more friendly than my summary).

This technique does require that we can reinterpret any state as a goal state, similar to the assumption in [Hindsight Experience Replay](https://arxiv.org/abs/1707.01495) (HER). They do compare to HER, and find that HER doesn't do very well, which I was quite surprised by. Clicking through to the paper, it turns out the authors were surprised as well, but then realized that this is because HER is designed to work with sparse reward problems, whereas they were evaluating on problems with relatively shaped rewards.

TDM: From Model-Free to Model-Based Deep Reinforcement Learning

In many tasks, we have hierarchical structure where we want to plan at a high level, but to execute the low-level actions we want to rely on learning through experience. For example, when biking from UC Berkeley to the Golden Gate Bridge, you definitely want to plan in advance the route you'll take (as opposed to learning through trial-and-error), but you want to learn how to bike through trial-and-error. Temporal Difference Models allow you to do model-based planning at the high level, and model-free learning at the low level. Specifically, you learn a function Q(s1, a, s2, T), which intuitively says "if I start from state s1, taking action a, and running for T steps, how close can I get to state s2". It turns out that this can be thought of as a Q function and so can be trained using standard model-free RL techniques. Note that the constraint Q(s1, a, s2, T) = 0 says that it is possible to get from s1 to s2 in T steps after first taking action a.

One standard way to solve model-based RL is to search for a sequence of states and actions (s0, a0, s1, a1, ...) that is feasible (agrees with the dynamics) and maximizes the reward, and then take the first action from that sequence. Using TDMs, we can now search for the sequence (s0, a0, sK, aK, s2K, a2k, ...) that is feasible and maximizes reward. The feasibility requirement is expressed by the constraint Q(s0, a0, sK, K) = 0.


Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review


Towards Symbolic Reinforcement Learning with Common Sense

Artur d'Avila Garcez, Aimore Resende Riquetti Dutra, Eduardo Alonso


Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future

Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, Dhruv Batra


My guess is that multiagent learning will become very popular fairly soon. In addition to this paper and the Neural MMO paper, it was also a key part of the AlphaStar training process. The implications of this research direction for safety are still unclear, and it seems valuable to explore them further. One which comes to mind: the sort of deceptive behaviour required for treacherous turns seems more likely to emerge from multiagent training than from single-agent training.

Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research

Joel Z. Leibo, Edward Hughes, Marc Lanctot, Thore Graepel

The authors argue that the best solution to the problem of task generation is creating multi-agent systems where each agent must adapt to the others. These agents do so first by learning how to implement a high-level strategy, and then by adapting it based on the strategies of others. (The authors use the term "adaptive unit" rather than "agent" to emphasise that change can occur at many different hierarchical levels, and either by evolution or learning). This adaptation may be exogenous (driven by the need to respond to a changing environment) or endogenous (driven by a unit's need to improve its own functionality). An example of the latter is a society implementing institutions which enforce cooperation between individuals. Since individuals will try to exploit these institutions, the process of gradually robustifying them can be considered an automatically-generated curriculum (aka autocurriuclum).


This is a great example of a pattern that seems quite common: once we automate tasks using end-to-end training that previously required more structured approaches, new more complex tasks will arise that will use the end-to-end trained systems as building blocks in a bigger structured approach. In this case, we can now train robots to navigate over short distances using end-to-end training, and this has been used in a structured approach involving graphs and waypoints to create robots that can traverse larger distances.

It's also an example of what you can do when you have a ton of compute: for the learned controller, they learned both the network architecture and the reward shaping. About the only thing that had to be explicity specified was the sparse true reward. (Although I'm sure in practice it took a lot of effort to get everything to actually work.)

Long-Range Robotic Navigation via Automated Reinforcement Learning

How can we get robots that successfully navigate in the real world? One approach is to use a high-level route planner that uses a learned control policy over very short distances (10-15 meters). The control policy is learned using deep reinforcement learning, where the network architecture and reward shaping is also learned via neural architecture search (or at least something very similar). The simulations have enough noise that the learned control policy transfers well to new environments. Given this policy as well as a floorplan of the environment we want the robot to navigate in, we can build a graph of points on the floorplan, where there is an edge between two points if the robot can safely navigate between the two points using the learned controller (which I _think_ is checked in simulation). At execution time, we can find a path to the goal in this graph, and move along the edges using the learned policy. They were able to build a graph for the four buildings at the Google main campus using 300 workers over 4 days. They find that the robots are very robust in the real world. See also [Import AI](https://jack-clark.net/2019/03/04/import-ai-136-what-machine-learning-power-infrastructure-means-for-humanity-new-gca-benchmarkdataset-challenges-image-captioning-systems-and-google-uses-frankenrl-to-create-more-mobile-robot/).


Neural MMO

Neural MMO is "a massively multiagent game environment for reinforcement learning agents". It was designed to be persistent (with concurrent learning and no environment resets), large-scale, efficient and expandable. Agents need to traverse an environment to obtain food and water in order to survive for longer (the metric for which they are rewarded), and are also able to engage in combat with other agents. Agents trained within a larger population explore more and consistently outperform those trained in smaller populations (when evaluated together). The authors note that multiagent training is a curriculum magnifier, not a curriculum in itself, and that the environment must facilitate adaptive pressures by allowing a sufficient range of interactions.


Competitive Experience Replay

Hao Liu, Alexander Trott, Richard Socher, Caiming Xiong


Reward Shaping via Meta-Learning

Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, Jun Zhu


Learning to Generalize from Sparse and Underspecified Rewards

Rishabh Agarwal, Mohammad Norouzi, Chen Liang and Dale Schuurmans


Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence

I like this sort of research, and its findings are interesting (even if the authors don't arrive at any clear explanation for them). One concern: I may be missing something, but it seems like the coupled samples method they use doesn't allow investigation into whether distributional methods benefit from generating better data (e.g. via more effective exploration).

A Comparative Analysis of Expected and Distributional Reinforcement Learning

Clare Lyle, Pablo Samuel Castro, Marc G. Bellemare

Distributional RL systems learn distributions over the value of actions rather than just their expected values. In this paper, the authors investigate the reasons why this technique improves results, by training distribution learner agents and expectation learner agents on the same data. They provide evidence against a number of hypotheses: that distributional RL reduces variance; that distributional RL helps with policy iteration; and that distributional RL is more stable with function approximation. In fact, distributional methods have similar performance to expectation methods when using tabular representations or linear function approximators, but do better when using non-linear function approximators such as neural networks (especially in the earlier layers of networks).


I endorse the goals of this paper, but my guess is that Hanabi is simple enough that agents can solve it using isolated heuristics rather than general reasoning about other agents' beliefs.

*Rohin's opinion:* I'm particularly excited to see more work on ad hoc teamwork, since it seems like very similar to the setting we are in, where we would like to deploy AI system among groups of humans and have things go well. See [Following human norms](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/eBd6WvzhuqduCkYv3) ([AN #42](https://mailchi.mp/f6488137d76c/alignment-newsletter-42)) for more details.

A cooperative benchmark: Announcing the Hanabi Learning Environment

The Hanabi Challenge: A New Frontier for AI Research

Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, Iain Dunning, Shibl Mourad, Hugo Larochelle, Marc G. Bellemare, Michael Bowling

The authors propose the cooperative, imperfect-information card game Hanabi as a target for AI research, due to the necessity of reasoning about the beliefs and intentions of other players in order to win. They identify two challenges: firstly, discovering a policy for a whole team that allows it to win (the self-play setting); and secondly, discovering an individual policy that allows an agent to play with an ad-hoc team without previous coordination. They note that successful self-play policies are often very brittle in the ad-hoc setting, which makes the latter the key problem. The authors provide an open-source framework, an evaluation benchmark and the results of existing RL techniques.


Recurrent Experience Replay in Distributed Reinforcement Learning

Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, Will Dabney


A Geometric Perspective on Optimal Representations for Reinforcement Learning

Marc G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, Clare Lyle


The Value Function Polytope in Reinforcement Learning

Robert Dadashi, Adrien Ali Taïga, Nicolas Le Roux, Dale Schuurmans, Marc G. Bellemare


Visual Hindsight Experience Replay

Himanshu Sahni, Toby Buckley, Pieter Abbeel, Ilya Kuzovkin


Causal Reasoning from Meta-reinforcement Learning

Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, Zeb Kurth-Nelson


I think that the "problem problem" is going to become increasingly important in RL, and that this is a promising approach. Note that this paper's contribution seems to be mainly that it combines ideas from previous papers on minimal criteria coevolution and innovation engines.

Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Rui Wang, Joel Lehman, Jeff Clune, Kenneth O. Stanley

The POET algorithm uses evolutionary strategies to evolve a population of pairs of tasks and agents. During each iteration, it first generates a new environment by perturbing an existing environment, then optimises each agent for its paired environment, then attempts to transfer agents between existing environments to improve performance (in case one environment turns out to be a useful "stepping stone" towards another). New environments are kept if they are neither too hard nor too easy for the current population of agents. This algorithm was tested using the Bipedal Walker environment, where it significantly outperformed standard evolutionary search.


Self-supervised Learning of Image Embedding for Continuous Control

Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, Martin Riedmiller


A Study on Overfitting in Deep Reinforcement Learning

Chiyuan Zhang, Oriol Vinyals, Remi Munos, Samy Bengio


Soft Actor Critic—Deep Reinforcement Learning with Real-World Robots

Tuomas Haarnoja, Vitchyr Pong, Kristian Hartikainen, Aurick Zhou, Murtaza Dalal, and Sergey Levine


Foundations and TrendsÂ® in Machine Learning, Vol 11, Issue 3-4

An Introduction to Deep Reinforcement Learning

Vincent Francois-Lavet, Peter Henderson, Riashat Islam, Marc G. Bellemare, Joelle Pineau


Unsupervised Control Through Non-Parametric Discriminative Rewards


While I agree with some of the concerns laid out in this paper, I'm not sure that these benchmarks are the best way to address them. The third task in particular is mainly testing for ability to ignore the "natural data" used, which doesn't seem very useful. I think a better alternative would be to replace Atari with tasks in procedurally-generated environments with realistic physics engines. However, this paper's benchmarks do benefit from being much easier to produce and less computationally demanding.

Natural Environment Benchmarks for Reinforcement Learning

This paper notes that RL performance tends to be measured in simple artificial environments - unlike other areas of ML in which using real-world data such as images or text is common. The authors propose three new benchmarks to address this disparity. In the first two, an agent is assigned to a random location in an image, and can only observe parts of the image near it. At every time step, it is able to move in one of the cardinal directions, unmasking new sections of the image, until it can classify the image correctly (task 1) or locate a given object (task 2). The third type of benchmark is adding natural video as background to existing Mujoco or Atari tasks. In testing this third category of benchmark, they find that PPO and A2C fall into a local optimum where they ignore the observed state when deciding the next action.


This is an interesting approach which I can imagine becoming useful. It would be nice to see more experimental work in more stochastic environments, though.

Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean-Baptiste Lespiau, Nicolas Heess

This paper aims to alleviate the data inefficiency of RL by using a model to synthesise data. However, even when environment dynamics can be modeled accurately, it can be difficult to generate data which matches the true distribution. To solve this problem, the authors use a Structured Causal Model trained to predict the outcomes which would have occurred if different actions had been taken from previous states. Data is then synthesised by rolling out from previously-seen states. The authors test performance in a partially-observable version of SOKOBAN, in which their system outperforms other methods of generating data.


This paper seems like a step forward in addressing the instability of using learned models in RL. However, the extent to which it's introducing new contributions, as opposed to combining existing ideas, is a little unclear.

Learning Latent Dynamics for Planning from Pixels

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson

The authors introduce PlaNet, an agent that learns an environment's dynamics from pixels and then chooses actions by planning in latent space. At each step, it searches for the best action sequence under its Recurrent State Space dynamics model, then executes the first action and replans. The authors note that having a model with both deterministic and stochastic transitions is critical to learning a good policy. They also use a technique called variational overshooting to train the model on multi-step predictions, by generalising the standard variational bound for one-step predictions. PlaNet approaches the performance of top model-free algorithms even when trained on 50x fewer episodes.


Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, Igor Mordatch


VIREL: A Variational Inference Framework for Reinforcement Learning

Matthew Fellows, Anuj Mahajan, Tim G. J. Rudner, Shimon Whiteson


Efficient Eligibility Traces for Deep Reinforcement Learning


Evolved Policy Gradients

In this meta-learning approach for reinforcement learning, the outer optimization loop proposes a new _loss function_ for the inner loop to optimize (in contrast to eg. MAML, where the outer optimization leads to better initializations for the policy parameters). The outer optimization is done using evolution strategies, while the inner optimization is stochastic gradient descent. The authors see good results on generalization to out-of-distribution tasks, which other algorithms such as RL2 don't achieve.


This continues the trend of being able to easily implement deep learning at higher and higher levels of abstraction. I'm looking forward to using it.

Open sourcing TRFL: a library of reinforcement learning building blocks

Matteo Hessel, Miljan Martic, Diego de Las Casas and Gabriel Barth-Maron

DeepMind is open-sourcing a Tensorflow library of "key algorithmic components" used in their RL agents. They hope that this will allow less buggy RL code.