This article is interesting because it formalizes and experimentally demonstrates an intuitive concern many have regarding recommender systems.  I also found the formalization of instrumental goals to be of independent interest. The most surprising result was that the agents who exploit tampering are not particularly more effective than policies that avoid tampering. This suggests that the instrumental incentive is not really pointing at what is actually optimal, which I found to be an illuminating distinction.

User Tampering in Reinforcement Learning Recommender Systems

Large-scale recommender systems have emerged as a way to filter through large pools of content to identify and recommend content to users. However, these advances have led to social and ethical concerns over the use of recommender systems in applications. This paper focuses on the potential for social manipulability and polarization from the use of RL-based recommender systems. In particular, they present evidence that such recommender systems have an instrumental goal to engage in user tampering by polarizing users early on in an attempt to make later predictions easier.

To formalize the problem the authors introduce a causal model. Essentially, they note that predicting user preferences requires an exogenous, non-observable variable, that models click-through rates. They then introduce a notion of instrumental goal that models the general behavior of RL-based algorithms over a set of potential tasks. The authors argue that such algorithms will have an instrumental goal to influence the exogenous/preference variables whenever user opinions are malleable. This ultimately introduces a risk for preference manipulation.

The author's hypothesis is tested using a simple media recommendation problem. They model the exogenous variable as either leftist, centrist, or right-wing. User preferences are malleable in the sense that a user shown content from an opposing side will polarize their initial preferences. In experiments, the authors show that a standard Q-learning algorithm will learn to tamper with user preferences which increases polarization in both leftist and right-wing populations. Moreover, even though the agent makes use of tampering it fails to outperform a crude baseline policy that avoids tampering.


I’m glad to see a simple summary of the evidence we currently have on the effects of social media on polarization. I feel like for the past year or two I’ve constantly heard people speculating about massive harms and even existential risks based on a couple of anecdotes or armchair reasoning, without bothering to check what has actually happened; whenever I talk to someone who seems to have actually studied the topic in depth, it seems they think that there are problems with recommender systems, but they are different from what people usually imagine. (The post also notes a reason to expect our intuitions to be misguided: we are unusual in that we get most of our news online; apparently every age group, starting from 18-24, gets more news from television than online.)

Note that there have been a few pieces arguing for these harms; I haven't sent them out in the newsletter because I don't find them very convincing, but you can find links to some of them along with my thoughts [here](https://www.alignmentforum.org/posts/TmHRACaxXrLbXb5tS/rohinmshah-s-shortform?commentId=EAKEfPmP8mKbEbERv).

How Much Do Recommender Systems Drive Polarization?

There is a common worry that social media (and recommender systems in particular) are responsible for increased polarization in recent years. This post delves into the evidence for this claim. By “polarization”, we mostly mean _affective polarization_, which measures how positive your feelings towards the opposing party are (though we also sometimes mean _issue polarization_, which measures the correlation between your opinions on e.g. gun control, abortion, and taxes). The main relevant facts are:

1. Polarization in the US has increased steadily since 1980 (i.e. pre-internet), though arguably there was a slight increase from the trend around 2016.
2. Since 2000, polarization has only increased in some Western countries, even though Internet use has increased relatively uniformly across countries.
3. Polarization in the US has increased most in the 65+ age group (which has the least social media usage).

(2) could be partly explained by social media causing polarization only in two-party systems, and (3) could be explained by saying that social media changed the incentives of more traditional media (such as TV) which then increased polarization in the 65+ age group. Nevertheless, overall it seems like social media is probably not the main driver of increased polarization. Social media may have accelerated the process (for instance by changing the incentives of traditional media), but the data is too noisy to tell one way or the other.


Beyond Engagement: Aligning Algorithmic Recommendations With Prosocial Goals

To decide what item to show a user, a recommender system needs to have some metric by which to rank items. Since this metric must usually be automated, it often contains in large part some operationalization of “engagement”. Unfortunately, this metric may not be able to differentiate between clickbait or extremist content on the one hand, and actually valuable posts on the other. In a workshop on the topic, participants brainstormed four main approaches for improvement:

1. **Build better controls:** Offer users more and better ways to control their feed.
2. **Develop standardized survey-based metrics:** Surveys should be able to get a significantly higher quality signal to optimize than engagement.
3. **Pay users for better data,** such as survey data.
4. **Recommend feeds, not items:** If we rank items individually, it is quite likely that all the posts of the same type (e.g. controversial posts) will get high scores. By ranking entire feeds, we can also optimize for the diversity of items within the feed.
5. **Incentivize the creation of different feeds,** so that users can choose which ones they prefer all things considered.


Both this paper and the previous post seem like meaningful progress in ideas for making recommendation engines better (though as a caveat, I don’t follow this space and so don’t know to what extent this has been said before). I’m glad that we’re getting to the point of actually proposing technical solutions; I hope to see more papers implementing such solutions (we’ve seen one from Twitter <@recently@>(@From Optimizing Engagement to Measuring Value@)).

[Aligning AI Optimization to Community Well-Being](https://link.springer.com/article/10.1007/s42413-020-00086-3) (an expanded version of <@Aligning AI to Human Values means Picking the Right Metrics@>)

What are you optimizing for? Aligning Recommender Systems with Human Values

Jonathan Stray, Steven Adler, Dylan Hadfield-Menell

While the previous blog post focused on societal-level approaches to recommender systems, this paper looks at what can be done at a technical level. By analyzing existing case studies of improvements to recommender systems (some of which we’ve <@seen before@>(@Aligning AI to Human Values means Picking the Right Metrics@)), the authors identify a typical approach taken in industry today.

First, engineers identify a problem with an already-deployed recommendation engine, perhaps from user feedback, or through monitoring. Second, they develop a concrete procedure to identify instances of this problem in the recommendations -- a typical approach is to curate a dataset and train an ML classifier to identify these instances, though it is also possible to use manual review. Finally, the recommender system is adjusted to avoid the problem, for example by adding a term to the objective when training the recommender system, or by filtering its outputs based on the classifier’s output.

The authors then propose four high-level technical approaches to recommender alignment:
1. Develop better measures of what we want out of a recommendation engine, for example, an operationalization of “well-being” rather than “engagement”.
2. Allow users to collaboratively design the recommendation engine (called _participatory design_). Rather than have a company decide on how to trade off between different objectives, allow the users to settle upon the appropriate tradeoffs themselves.
3. Interactively learn about the user’s values. While this could look like building better controls as suggested in the previous post, it could also involve e.g. using <@Inverse Reward Design@> to maintain appropriate uncertainty over what the user cares about.
4. Design around “time well spent”, as evaluated by users on reflection or after consideration, rather than revealed preferences or immediate judgments. For example, we could show users a summary of their activity over the past month and ask how happy they are about it.


It seems to me that this is how we'll have to solve exploration in practice if we don't want to have a huge sample complexity, though I know other researchers are optimistic about solving exploration using curiosity or diversity. It's pretty exciting that they could use a source of data that was already present in the real world.

Playing hard exploration games by watching YouTube

Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, Nando de Freitas

There are many YouTube videos demonstrating how to play levels of eg. Montezuma's Revenge. Can we use these demonstrations to solve the hard exploration tasks in Atari? One challenge is that the videos have slightly different visual properties (like color and resolution). They propose to learn a shared feature space by using an auxiliary loss where the network must predict the number of timesteps between two frames of a video, or to predict the delay between a video and audio clip from the same trajectory. Using this shared feature space, they can define a reward function that encourages the agent to take trajectories whose features match those of the demonstrations. In experiments they exceed human performance on Atari games with hard exploration problems.


I feel similarly to my opinion on <@AlphaStar@>(@AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning@) here. The result is definitely impressive and a major step up in complexity from shorter, discrete games like chess or go. However, I don’t see how the approach of just running PPO at a large scale brings us closer to AGI because we can’t run massively parallel simulations of real world tasks. Even for tasks that can be simulated, this seems prohibitively expensive for most use cases (I couldn’t find the exact costs, but I’d estimate this model cost tens of millions of dollars). I’d be quite excited to see an example of deep RL being used for a complex real world task without training in simulation.

Dota 2 with Large Scale Deep Reinforcement Learning

OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, Susan Zhang

In April, <@OpenAI Five@>(@How to Train Your OpenAI Five@) defeated the world champion Dota 2 team, OG. This paper describes its training process. OpenAI et al. hand-engineered the reward function as well as some features, actions, and parts of the policy. The rest of the policy was trained using PPO with an LSTM architecture at a massive scale. They trained this in a distributed fashion as follows:

-   The *Controller* receives and distributes the updated parameters.
-   The *Rollout Worker CPUs* simulate the game, send observations to the *Forward Pass GPUs* and publish samples to the *Experience Buffer*.
-   The *Forward Pass GPUs* determine the actions to use and send them to the *Rollout Workers*.
-   The *Optimizer GPUs* sample experience from the *Experience Buffer*, calculate gradient updates, and then publish updated parameters to the *Controller*.

The model trained over 296 days. In that time, OpenAI needed to adapt it to changes in the code and game mechanics. This was done via model “surgery”, in which they would try to initialize a new model to maintain the same input-output mapping as the old one. When this was not possible, they gradually increased the proportion of games played with the new version over time.


This general idea has been proposed in robotics as well, in [Reverse Curriculum Generation for Reinforcement Learning](https://arxiv.org/abs/1707.05300), where there is a single goal state. However, in this setting we have the added benefit of perfect inverse dynamics, that is, for any action a that moves us from state s to s', we can find the inverse action a' that moves us from state s' to s. This allows the authors to start from the goal state, generate nearby states, and automatically know the value of those states (or at least a very good approximation to it). [Hindsight Experience Replay](https://arxiv.org/abs/1707.01495) also tackles similar issues -- I'd be interested to see if it could solve the Rubik's cube. Overall, the problem of sparse rewards is very difficult, and it seems like we now have another solution in the case where we have a single goal state and perfect (or perhaps just sufficiently good?) inverse dynamics.

Solving the Rubik's Cube Without Human Knowledge

Stephen McAleer, Forest Agostinelli, Alexander Shmakov, Pierre Baldi

This paper proposes _Autodidactic Iteration_ (ADI), which is a technique that can be combined with the techniques in AlphaGo and expert iteration to solve problems with only one goal state, such as the Rubik's cube. MCTS with value and policy networks will not suffice, because when starting from a randomly scrambled cube, MCTS will never find a path to the goal state, and so there will never be any reward signal. (Whereas with Go, even if you play randomly the game will end relatively quickly, giving you some reward signal.) To get around this, they start _from the goal state_ and generate states that are near the goal state. This gives them a training dataset of states for which they know (a good approximation to) the value and the best action, which they can use to train a value and policy network. They then use this with MCTS to solve the full problem, as in AlphaGo.


This is clearly a major step for model-based RL, becoming the state of the art on a very popular benchmark and enabling planning approaches to be used in domains with unknown rules or dynamics. I am typically optimistic about model-based approaches as progress towards safe AGI. They map well to how humans think about most complex tasks: we consider the likely outcomes of our actions and then plan accordingly. Additionally, model-based RL typically has the safety property that the programmers know what states the algorithm expects to pass through and end up in, which aids with interpretability and auditing. However, *MuZero* loses that property by using a learned model whose internal states are not constrained to have any semantic meaning. I would be quite excited to see follow up work that enables us to understand what the model components are learning and how to audit them for particularly bad inaccuracies.

**Rohin's opinion:** _Note: This is more speculative than usual._ This approach seems really obvious and useful in hindsight (something I last felt for [population-based training](https://deepmind.com/blog/article/population-based-training-neural-networks) of hyperparameters). The main performance benefit (that I see) of model-based planning is that it only needs to use the environment interactions to learn how the environment _works_, rather than how to _act optimally_ in the environment -- it can do the "act optimally" part using some MDP planning algorithm, or by simulating trajectories from the world model rather than requiring the actual environment. Intuitively, it should be significantly easier to learn how an environment works -- consider how easy it is for us to learn the rules of a game, as opposed to playing it well. However, most model-based approaches force the learned model to learn features that are useful for predicting the state, which may not be the ones that are useful for playing well, which can handicap their final performance. Model-free approaches on the other hand learn exactly the features that are needed for playing well -- but they have a much harder learning task, so it takes many more samples to learn, but can lead to better final performance. Ideally, we would like to get the benefits of using an MDP planning algorithm, while still only requiring the agent to learn features that are useful for acting optimally.

This is exactly what MuZero does, similarly to [this previous paper](https://arxiv.org/abs/1707.03497): its "model" only predicts actions, rewards, and value functions, all of which are much more clearly relevant to acting optimally. However, the tasks that are learned from environment interactions are in some sense "easier" -- the model only needs to predict, _given a sequence of actions_, what the immediate reward will be. It notably _doesn't_ need to do a great job of predicting how an action now will affect things ten turns from now, as long as it can predict how things ten turns from now will be _given_ the ten actions used to get there. Of course, the model does need to predict the policy and the value function (both hard and dependent on the future), but the learning signal for this comes from MCTS, whereas model-free RL relies on credit assignment for this purpose. Since MCTS can consider multiple possible future scenarios, while credit assignment only gets to see the trajectory that was actually rolled out, we should expect that MCTS leads to significantly better gradients and faster learning.

Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, David Silver

Up until now, model-free RL approaches have been state of the art at visually rich domains such as Atari, while model-based RL has excelled for games which require planning many steps ahead, such as Go, chess, and shogi. This paper attains state of the art performance on Atari using a model-based approach, *MuZero*, while matching <@AlphaZero@>(@AlphaZero: Shedding new light on the grand games of chess, shogi and Go@) at Go, chess, and shogi while using less compute. Importantly, it does this without requiring any advance knowledge of the rules of the game.

*MuZero*'s model has three components:

1. The *representation* function produces an initial internal state from all existing observations.
2. The *dynamics* function predicts the next internal state and immediate reward after taking an action in a given internal state.
3. The *prediction* function generates a policy and a value prediction from an internal state.

Although these are based on the structure of an MDP, **the internal states of the model do not necessarily have any human-interpretable meaning**. They are trained end-to-end only to accurately predict the policy, value function, and immediate reward. This model is then used to simulate trajectories for use in MCTS.


I really liked this paper: it was creative in combining conceptual components from variational methods and imitation learning, and it was pragmatic in trying to address the problem of how to get viable human-demonstration data in a way that avoids having to get distinct datasets for a huge set of different discrete tasks.

Learning Latent Plans from Play

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, Pierre Sermanet

This paper collects unsupervised data of humans playing with robotic control systems, and uses that data to thread a needle between two problems in learning. One problem is that per-task demonstration data is costly, especially as number of tasks grows; the other is that randomly sampled control actions will rarely stumble across complex motor tasks in ways that allow robots to learn. The authors argue that human play data is a good compromise because humans at play tend to explore different ways of manipulating objects in ways that give robots nuggets of useful information like "how do I move this block inside a drawer", which can be composed into more complicated and intentional tasks.

The model works by learning to produce vectors that represent plans (or sequences of actions), and jointly learning to decode those vectors into action sequences. This architecture learns to generate plan vectors by using an autoencoder-like structure that uses KL divergence to align (1) a distribution of plan vectors predicted from the start and end state of a window of play data, and (2) a distribution of plan vectors predicted by looking back at all the actions taken in that window. Because we're jointly learning to unroll the (2) lookback-summarized vector such that it matches the actions actually taken, we'll ideally end up with a system that can take in a given plan vector and produce a sequence of actions to execute that plan. And, because we're learning to predict a vector that aligns with actions successfully taken to get to an end state from a starting one, the model at test time should be able to produce a play vector corresponding to feasible actions that will get it from its current state to a goal state we'd like it to reach.  The authors found that their Play-trained model was able to outperform single-task models on a range of manipulation tasks, even though those single-task models were trained with explicit demonstrations of the task.


I really liked this paper: it's a rare instance where I actually wanted to read the theory in the paper because it felt important for getting the high level insight. The theory is particularly straightforward and easy to understand (which usually seems to be true when it leads to high level insight). The design of the algorithm seems more principled than others, and the experiments suggest that this was actually fruitful. The algorithm is probably more computationally expensive per step compared to other algorithms, but that could likely be improved in the future.

One thing that felt strange is that the proposed solution is basically to prevent generalization between (s, a) pairs, but the whole point of DQL algorithms is to generalize between (s, a) pairs since you can't get experience from all of them. Of course, since they are only preventing generalization within a batch, they still generalize between (s, a) pairs that are not in the batch, but presumably that was because they only could prevent generalization within the batch. Empirically the algorithm does seem to work, but it's still not clear to me why it works.

Towards Characterizing Divergence in Deep Q-Learning

Joshua Achiam, Ethan Knight, Pieter Abbeel

Q-Learning algorithms use the Bellman equation to learn the Q\*(s, a) function, which is the long-term value of taking action a in state s. Tabular Q-Learning collects experience and updates the Q-value for each (s, a) pair independently. As long as each (s, a) pair is visited infinitely often, and the learning rate is decayed properly, the algorithm is guaranteed to converge to Q\*.

Once we get to complex environments where you can't enumerate all of the states, we can't explore all of the (s, a) pairs. The obvious approach is to approximate Q\*(s, a). Deep Q-Learning (DQL) algorithms use neural nets for this approximation, and use some flavor of gradient descent to update the parameters of the net such that it is closer to satisfying the Bellman equation. Unfortunately, this approximation can prevent the algorithm from ever converging to Q\*.

This paper studies the first-order Taylor expansion of the DQL update, and identifies three factors that affect the DQL update: the distribution of (s, a) pairs from which you learn, the Bellman update operator, and the _neural tangent kernel_, a property of the neural net that specifies how information from one (s, a) pair generalizes to other (s, a) pairs. The theoretical analysis shows that as long as there is limited generalization between (s, a) pairs, and each (s, a) pair is visited infinitely often, the algorithm will converge. Inspired by this, they design PreQN, which explicitly seeks to minimize generalization across (s, a) pairs _within the same batch_. They find that PreQN leads to competitive and stable performance, despite not using any of the tricks that DQL algorithms typically require, such as target networks.


I don't care very much about whether AlphaStar is superhuman or not -- it clearly is very good at StarCraft at both the micro and macro levels. Whether it hits the rather arbitrary level of "top human performance" is not as interesting as the fact that it is anywhere in the ballpark of "top human performance".

It's interesting to compare this to [OpenAI Five](https://blog.openai.com/openai-five/) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13)). While OpenAI solved the exploration problem using a combination of reward shaping and domain randomization, DeepMind solved it by using imitation learning on human games. While OpenAI relied primarily on self-play, DeepMind used population-based training in order to deal with catastrophic forgetting and in order to be robust to many different strategies. It's possible that this is because of the games they were playing -- it's plausible to me that StarCraft has more rock-paper-scissors-like cyclic mechanics than Dota, and so it's more important to be robust to many strategies in StarCraft. But I don't know either game very well, so this is pure speculation.

Exploring the full strategy space rather than finding the Nash equilibrium seems like the right thing to do, though I haven't kept up with the multiagent RL literature so take that with a grain of salt. That said, it doesn't seem like the full solution -- you also want some way of identifying what strategy your opponent is playing, so that you can choose the optimal strategy to play against them.

I often think about how you can build AI systems that _cooperate_ with humans. This can be significantly harder: in competitive games, if your opponent is more suboptimal than you were expecting, you just crush them even harder. However, in a cooperative game, if you make a bad assumption about what your partner will do, you can get significantly worse performance. (If you've played Hanabi, you've probably experienced this.) Self-play does not seem like it would handle this situation, but this kind of population-based training could potentially handle it, if you also had a method to identify how your partner is playing. (Without such a method, you would play some generic strategy that would hopefully be quite robust to playstyles, but would still not be nearly as good as being able to predict what your partner does.)

[Open-ended Learning in Symmetric Zero-sum Games](https://arxiv.org/abs/1901.08106), [AMA with AlphaStar creators and pro players](https://www.reddit.com/r/MachineLearning/comments/ajgzoc/we_are_oriol_vinyals_and_david_silver_from), and [Vox: StarCraft is a deep, complicated war strategy game. Google’s AlphaStar AI crushed it.](https://www.vox.com/future-perfect/2019/1/24/18196177/ai-artificial-intelligence-google-deepmind-starcraft-game)

AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

The AlphaStar system from DeepMind has beaten top human pros at StarCraft. You can read about the particular details of the matches in many sources, such as the blog post itself, this [Vox article](https://www.vox.com/future-perfect/2019/1/24/18196177/ai-artificial-intelligence-google-deepmind-starcraft-game), or [Import AI](https://jack-clark.net/2019/01/28/import-ai-131-ibm-optimizes-ai-with-ai-via-neunets-amazon-reveals-its-scout-delivery-robot-google-releases-300k-natural-questions-dataset/). The quick summary is that while there are some reasons you might not think it is conclusively superhuman yet (notably, it only won when it didn't have to manipulate the camera, and even then it may have had short bursts of very high actions per minute that humans can't do), it is clearly extremely good at StarCraft, both at the technically precise micro level and at the strategic macro level.

I want to focus instead on the technical details of how AlphaStar works. The key ideas seem to be a) using imitation learning to get policies that do something reasonable to start with and b) training a population of agents in order to explore the full space of strategies and how to play against all of them, without any catastrophic forgetting. Specifically, they take a dataset of human games and train various agents to mimic humans. This allows them to avoid the particularly hard exploration problems that happen when you start with a random agent. Once they have these agents to start with, they begin to do population-based training, where they play agents against each other and update their weights using an RL algorithm. The population of agents evolves over time, with well-performing agents splitting into two new agents that diversify a bit more. Some agents also have auxiliary rewards that encourage them to explore different parts of the strategy space -- for example, an agent might get reward for building a specific type of unit. Once training is done, we have a final population of agents. Using their empirical win probabilities, we can construct a Nash equilibrium of these agents, which forms the final AlphaStar agent. _(Note: I'm not sure if at the beginning of the game, one of the agents is chosen according to the Nash probabilities, or if at each timestep an action is chosen according to the Nash probabilities. I would expect the former, since the latter would result in one agent making a long-term plan that is then ruined by a different agent taking some other action, but the blog post seems to indicate the latter -- with the former, it's not clear why the compute ability of a GPU restricts the number of agents in the Nash equilibrium, which the blog posts mentions.)_

There are also a bunch of interesting technical details on how they get this to actually work, which you can get some information about in this [Reddit AMA](https://www.reddit.com/r/MachineLearning/comments/ajgzoc/we_are_oriol_vinyals_and_david_silver_from/). For example, "we included a policy distillation cost to ensure that the agent continues to try human-like behaviours with some probability throughout training, and this makes it much easier to discover unlikely strategies than when starting from self-play", and "there are elements of our research (for example temporally abstract actions that choose how many ticks to delay, or the adaptive selection of incentives for agents) that might be considered “hierarchical”". But it's probably best to wait for the journal publication (which is currently in preparation) for the full details.

I'm particularly interested by this [Balduzzi et al paper](https://arxiv.org/abs/1901.08106) that gives some more theoretical justification for the population-based training. In particular, the paper introduces the concept of "gamescapes", which can be thought of as a geometric visualization of which strategies beat which other strategies. In some games, like "say a number between 1 and 10, you get reward equal to your number - opponent's number", the gamescape is a 1-D line -- there is a scalar value of "how good a strategy is", and a better strategy will beat a weaker strategy. On the other hand, rock-paper-scissors is a cyclic game, and the gamescape looks like a triangle -- there's no strategy that strictly dominates all other strategies. Even the Nash strategy of randomizing between all three actions is not the "best", in that it fails to exploit suboptimal strategies, eg. the strategy of always playing rock. With games that are even somewhat cyclic (such as StarCraft), rather than trying to find the Nash equilibrium, we should try to explore and map out the entire strategy space. The paper also has some theoretical results supporting this that I haven't read through in detail.


I know that a lot of effort has gone into this project, and I expect that as a result this is probably the best educational resource on deep RL out there. The main other resource I know of is the [deep RL bootcamp](https://sites.google.com/view/deep-rl-bootcamp/), which probably supplements this resource nicely, especially with the lectures (though it is a year out of date).

Spinning Up in Deep RL

OpenAI has released an educational resource aimed to help software engineers become skilled at deep reinforcement learning. It includes simple implementations of many deep RL algorithms (as opposed to the relatively complex, highly optimized implementations in Baselines), educational exercises, documentation, and tutorials. OpenAI will host a workshop on the topic at their headquarters on Feb 2nd, and are also planning to hold a workshop at CHAI some time in early 2019.


This is a simple yet powerful idea which significantly reduces the number of samples required for good performance; I'm excited to see follow-up work. There are some confusing references in the paper to applications of MB-MPO to real-world robotics; it's unclear whether the authors already have results in this direction.

Model-Based Reinforcement Learning via Meta-Policy Optimization

Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, Pieter Abbeel

This paper introduces a new approach to model-based RL, called Model-Based Meta-Policy-Optimisation (MB-MPO), which doesn't require the dynamics models to be as accurate. It does so by learning an ensemble of dynamics models each trained on different subsets of the data, and then using meta-learning (specifically MAML) to find a policy which adapts well to any of these models within one step of gradient descent. This approach is a form of regularisation of policy learning, and achieves much greater sample efficiency without compromising performance: MB-MPO does just as well as top model-free algorithms in various Mujoco continuous-control environments, while requiring between 10 and 100 times fewer samples. Experiments suggest that it does so by having higher plasticity in regions with high dynamics model uncertainty. See also [Import AI](https://jack-clark.net/2018/09/25/import-ai-113-why-satellitesai-gives-us-a-global-eye-industry-pays-academia-to-say-sorry-for-strip-mining-it-and-kindred-researchers-seek-robot-standardization/).


This is an impressive result, since it's the first time a single agent has performed so well on a range of Atari games. It doesn't seem to have required any novel techniques except for a straightforward extension of PopArt to the multi-task setting, but this is still interesting since the results from the previous PopArt paper were very mixed (performance increased and decreased dramatically on different games, with the average remaining roughly stable).

One confusing aspect was that PopArt still benefitted slightly from being trained with reward clipping (110% vs. 101% in the unclipped case), even though the point of PopArt was to normalize rewards so that clipping wasn't necessary. I'm assuming the clipping happens after PopArt normalization, since if it happens before then you lose information as in the Pacman example. In this case, maybe it's that the reward distribution is fat-tailed, and so even after normalization there could be some extreme rewards that after normalization are still large enough that they would cause updates that are too large, and clipping alleviates this problem.

Preserving Outputs Precisely while Adaptively Rescaling Targets

When an agent is trained on multiple tasks across which the sizes of rewards vary greatly, it usually focuses on tasks which provide the largest or most frequent rewards at the expense of performance on others. Previous work dealt with this by clipping rewards outside a certain range, but this changes the optimal policy (eg. in Pacman, eating pellets is just as rewarding as eating ghosts). This paper uses PopArt (introduced in [this 2016 paper](https://arxiv.org/abs/1602.07714)) to normalise rewards from each task before using them to update the policy in an actor-critic RL algorithm. The authors use PopArt to train a single IMPALA agent which can play all 57 Atari games, achieving a median performance slightly higher than human performance.

To delve into more detail about PopArt, let's consider training a policy with an actor-critic algorithm. In this case, we need a critic that produces estimates of values V, and an actor that produces probabilities of actions. Both of these networks are trained by taking gradients of their outputs, and weighting them based on the observed rewards. Now, a key empirical fact about deep learning is that it works better if all the things are normalized, especially the gradients. (If nothing else, it makes it easier to choose the learning rate.) For the actor, this is easy -- probabilities are already normalized, and the weight of gradient is proportional to the reward, so we can just rescale the weight of the gradient based on the mean and standard deviation of the rewards we have observed so far. This is a bit harder for the critic, since it has to predict values, so we have to normalize both the outputs and the gradient weights. We can normalize the gradient weights in the same way as before. However, normalizing the outputs is tricky, because as time goes on the means and standard deviations change. To do this, at every timestep we modify the weights of the critic that is equivalent to unnormalizing based on the old statistics and then normalizing based on the new statistics. This gives the PopArt method.

Here's a simple example where I butcher types, ignore the difference between states and trajectories, and throw away the standard deviation. Suppose the first reward we see is 10, so we say that our mean is 10 and train our net to output a normalized reward of 0 for this state and action. Then, we see a reward of 100, so we update our mean to 55. On our previous (state, action) pair, we still output a normalized reward of 0, which now corresponds to a real reward of 55, even though it should correspond to 10! We then do the unnormalize-and-renormalize trick. After unnormalization, the critic would output 10, and after renormalization, the network would output -45, which when combined with the mean of 55 would give us the desired 10 reward.


This is a cool example of a relatively simple yet powerful idea -- instead of having a goal space over all states, learn a good latent representation and use that as your goal space. This enables unsupervised learning in order to figure out how to use a robot to generally affect the world, probably similarly to how babies explore and learn.

Visual Reinforcement Learning with Imagined Goals

This is a blog post explaining a paper by the same name that I covered in [AN #16](https://mailchi.mp/f2950ed2ac4b/alignment-newsletter-16). It's particularly clear and well-explained, and I continue to think the idea is cool and interesting. I've recopied my summary and opinion here, but you should read the blog post, it explains it very well.

[Hindsight Experience Replay](https://blog.openai.com/ingredients-for-robotics-research/) (HER) introduced the idea of accelerating learning with sparse rewards, by taking trajectories where you fail to achieve the goal (and so get no reward, and thus no learning signal) and replacing the actual goal with an "imagined" goal chosen in hindsight such that you actually achieved that goal, which means you get reward and can learn. This requires that you have a space of goals such that for any trajectory, you can come up with a goal such that the trajectory achieves that goal. In practice, this means that you are limited to tasks where the goals are of the form "reach this goal state". However, if your goal state is an image, it is very hard to learn how to act in order to reach any possible image goal state (even if you restrict to realistic ones), since the space is so large and unstructured. The authors propose to first learn a structured latent representation of the space of images using a variational autoencoder (VAE), and then use that structured latent space as the space of goals which can be achieved. They also use Q-learning instead of DDPG (which is what HER used), so that they can imagine any goal with a minibatch (s, a, s') and learn from it (whereas HER/DDPG is limited to states on the trajectory).


I agree with [Shimon Whiteson's take](https://twitter.com/shimon8282/status/979344417961250817), which is that this method gets improvements by creating a separation of concerns between modelling the world and learning a controller for the model, and evaluating on environments where this separation mostly holds. A major challenge in RL is learning the features that are important for the task under consideration, and this method instead learns features that allow you to reconstruct the state, which could be very different, but happen to not be different in their environments. That said, I really like the presentation of the paper and the fact that they did ablation studies.

Recurrent World Models Facilitate Policy Evolution

I read the [interactive version](https://worldmodels.github.io/) of the paper. The basic idea is to do model-based reinforcement learning, where the model is composed of a variational auto-encoder that turns a high-dimensional state of pixels into a low-dimensional representation, and a large RNN that predicts how the (low-dimensional) state will evolve in the future. The outputs of this model are fed into a very simple linear controller that chooses actions. Since the controller is so simple, they can train it using a black box optimization method (an evolutionary strategy) that doesn't require any gradient information. They evaluate on a racing task and on Doom, and set new state-of-the-art results. There are also other interesting setups -- for example, once you have a world model, you can train the controller completely within the world model without interacting with the outside world at all (using the number of timesteps before the episode ends as your reward function, since the world model doesn't predict standard rewards, but does predict whether the episode ends). There are a lot of cool visualizations that let you play with the models trained with their method.


Remarkably, this paper is both heavy on (useful) math, and very clear and well-explained. I actually didn't try to explain the technical details in my summary as much as I usually do, because you can just read the paper and actually understand what's going on, at least if you're familiar with probabilistic graphical models. Regarding the content, I've found the framework useful for one of my current projects, so I do recommend reading it.

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

I sent this out as a link in [AN #5](https://mailchi.mp/0ae5d69de63b/alignment-newsletter-5), but only just got around to reading it. This paper shows how you can fit the framework of reinforcement learning into the framework of inference within probabilistic graphical models. Specifically, the states s_t and actions a_t are now represented as nodes in the graphical model, and we add in new nodes O_t that represent whether or not an "event" happened at time t. By assigning the values of P(O_t | s_t, a_t) appropriately, we can encode a reward function. Then, by conditioning on the rewarding events happening, we can infer what actions must have been taken to get these events, which gives us a policy that achieves high reward. They later talk about the connection to variational inference, and how you can get IRL methods in this framework.


2nd Workshop on Meta-Learning at NeurIPS 2018

I really like these empirical papers that compare different methods and show their advantages and disadvantages. I was pretty surprised to see random features do as well as they did, especially to see that they transferred as well as learned features in one of the two cases they studied. There was of course a neural net that could learn how to use the arbitrary representation induced by the features, but then why couldn't it do the same for pixels? Perhaps the CNN was useful primarily for reducing the dimensionality of the pixels by combining nearby pixels together, and it didn't really matter how that was done since it still retains all the important information, but in a smaller vector?

I'm glad that the paper acknowledges that the good performance of curiosity is limited to environments that human designers have created. In a real world task, such as a house-cleaning robot, there are many other sources of uncertainty in the world that are unrelated to the task, and you need some form of specification to focus on it -- curiosity alone will not be enough.

Large-Scale Study of Curiosity-Driven Learning

Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell and Alexei A. Efros

One major challenge in RL is how to explore the environment sufficiently in order to find good rewards to learn from. One proposed method is curiosity, in which the agent generates an internal reward for taking any transition where the outcome was surprising, where surprisal is measured as the negative log probability assigned to the outcome by the agent. In this paper, a neural net that takes as input observation features φ(x) and action a, and predicts the features of the next state observation. The mean squared error with the actual features of the next state is then a measure of the surprisal, and is used as the curiosity reward. This is equivalent to treating the output of the neural net as the mean of a Gaussian distribution with fixed variance, and defining the reward to be the negative log probability assigned to the actual next state.

This still leaves the feature function φ undetermined. They consider using pixels directly, using a CNN with randomly chosen fixed weights, learned CNN features using a variational autoencoder (VAE) (which optimize for features that are useful for reconstructing the observation), and learned CNN features using inverse dynamics (IDF) (which optimize for features that are useful for reconstructing the action, biasing the features towards aspect of the environment that the agent can control). As you might expect, pixels don't work very well. However, random features do work quite well, often beating the VAE and IDF. This can happen because the random features stay fixed, leading to more stable learning, whereas with the VAE and IDF methods the features are changing over time, and the environment distribution is changing over time (as the agent explores more of it), leading to a harder learning problem.

Typically, curiosity is combined with an external reward. In this paper, the authors evaluate how well an agent can do with _only_ curiosity and no external reward. Intuitively, in game environments designed by humans, the designer sets up a good curriculum for humans to learn, which would align well with a curiosity reward. In fact, this is what happens, with a curiosity based reward leading to great performance (as measured by the external reward) on Atari games, Super Mario, Unity mazes, and Roboschool Pong, when using random features or IDF features. (The VAE features sometimes work well but were very unstable.) They evaluate transfer between levels in Super Mario, and find that the learned features transfer in more cases than random ones. Looking at the graphs, this seems like a very small effect to me -- I'm not sure if I'd agree with the claim, but I'd want to look at the behavior in videos and what the reward function rewards before making that claim strongly. They also investigate Pong with both players being driven by curiosity, and the players become so good at rallying that they crash the emulator.

Finally, they note one downside -- in any stochastic environment, or any environment where there will be lots of uncertainty about what will happen (eg. in multiagent settings), at convergence the reward for any action will be equal to the entropy of the next state distribution. While they don't demonstrate this flaw in particular, they show a related one -- if you add a TV to a Unity maze, and an action to change the channel, then the agent learns to stand in front of the TV and change the channel forever, rather than solving the maze.


If you do deep RL research regularly, you probably won't get too much out of it (though you might still get some handy tips on things you can do with Tensorflow), but I think everyone else should read it to get a more concrete sense of what deep RL research actually looks like and to be able to communicate more effectively with deep RL researchers.

Deep Reinforcement Learning Doesn’t Work Yet

Lessons Learned Reproducing a Deep Reinforcement Learning Paper

It's exactly what the title says. There were a lot of points that I can't easily summarize, but some highlights:
- Deep RL is very fiddly and tricky, and most of his time was spent debugging.
- Since each run takes a long time, it's very important to put a lot of effort into figuring out what hypotheses to test.
- The project took ~$850 of compute (~9,000 CPU hours and ~450 GPU hours)


I think this finally fell within my expectations, after two instances where I underestimated OpenAI Five. I expected that they would let the human team choose heroes in some limited way (~80%), that OpenAI Five would not be able to draft using just gradients via PPO (~60%), and (after having seen the first two games) that the human team would win after an adversarial draft (~70%). Of course, a draft did happen, but it was done by a tree search algorithm, not an algorithm learned using PPO.

The games themselves were pretty interesting (though I have not played Dota so take this with a grain of salt). It seemed to me like OpenAI Five had learned a particularly good strategy that plays to the advantages of computers, but hadn't learned some of the strategies and ideas that human players use to think about Dota. Since it uses the same amount of computation for each decision, it makes good decisions on all timescales, including ones where something surprising has occurred where humans would need some time to react, and also to coordinate. For example, as soon as a human hero entered within range of the bots (just to look and retreat), all of the bots would immediately unleash a barrage of attacks, killing the hero -- a move that humans could not execute, because of slower reaction times and worse communication and teamwork. Similarly, one common tactic in human gameplay is to teleport into a group of heroes and unleash an area-of-effect ability, but when they tried this against OpenAI Five, one of the bots hexed the hero as soon as he teleported in, rendering him unable to cast the spell. (That felt like the decisive moment in the first game.) On the other hand, there were some clear issues with the bots. At one point, two OpenAI bots were chasing Blitz, and Blitz used an ability that made him invisible while standing still. Any human player would have spammed area attacks, but the bots simply became confused and eventually left. Similarly, I believe (if I understood the commentary correctly) that a bot once used an ability multiple times, wasting mana, even though all uses after the first had no additional effect.

Other articles would have you believe that the games weren't even close, and if you look at the kill counts, that would seem accurate. I don't think that's actually right -- from what I understand, kills aren't as important as experience and gold, and you could see this in the human gameplay. OpenAI Five would often group most of its heroes together to push forward, which means they get less experience and gold. The human team continued to keep their heroes spread out over the map to collect resources -- and even though OpenAI Five got way more kills, the overall net worth of the two teams' heroes remained about equal for most of the early game. The big difference seemed to be that when the inevitable big confrontation between the two teams happened, OpenAI Five always came out on top. I'm not sure how, my Dota knowledge isn't good enough for that. Based on Blitz's comment, my guess is that OpenAI Five is particularly good at fights between heroes, and the draft reflects that. But I'd still guess that if you had pro human players who ceded control to OpenAI Five whenever a fight was about to happen, they would beat OpenAI Five (~70%). I used to put 80% on that prediction, but Blitz's comment updated me away from that.

One interesting thing was that the win probability seemed to be very strongly influenced by the draft, which in hindsight seems obvious. Dota is a really complicated game that is constantly tweaked to keep it balanced for humans, and even then the draft is very important. When you now introduce a new player (OpenAI Five) with very different capabilities (such as very good decision making under time pressure) and change the game conditions (such as a different pool of heroes), you should expect the game to become very imbalanced, with some teams far outshining others. And in fact we did see that Lion (the hero with the hexing ability) was remarkably useful (against humans, at least).

OpenAI Five Benchmark: Results

The OpenAI Five benchmark happened last Sunday, where OpenAI Five won two matches against the human team, and lost the last one when their draft was adversarially selected. They are now planning to play at The International in a couple of weeks (dates to be finalized). That will be a harder challenge, since they will be playing against teams that play and train professionally, and so will be better at communication and coordination than the human team here.

Blitz (one of the human players) [said](https://www.reddit.com/r/MachineLearning/comments/9533g8/n_openai_five_benchmark_results/e3qrnbt/): "The only noticeable difference in the mechanical skill aspect was the hex from the Lion, but even that was sorta irrelevant to the overall game flow. Got outdrafted and outmaneuvered pretty heavily, and from a strategy perspective it was just better then us. Even with the limitations in place it still 'felt' like a dota game, against a very good team. It made all the right plays I'd expect most top tier teams to make."

On the technical side, OpenAI implemented a brute-force draft system. With a pool of 18 heroes, you get some combinatorial explosion, but there are still only ~11 million possible matchups. You can then do a simple tree search over which hero to draft, where at the leaves (when you have a full draft) you choose which leaf you want based on the win probability (which OpenAI Five already outputs). Seeing this in action, it seems to me like it's a vanilla minimax algorithm, probably with alpha-beta pruning so that they don't have to evaluate all ~159 billion nodes in the tree. (Or they could have done the full search once, hardcoded the action it comes up with for the first decision, and run the full search for every subsequent action, which would have under 10 billion nodes in the tree.)

Besides the win probabilities, there are other ways to get insight into what the model is "thinking" -- for example, by asking the model to predict where the hero will be in 6 seconds, or by predicting how many last hits / denies / kills / deaths it will have.

The model that played the benchmark has been training since June 9th. Of course, in that time they've changed many things about the system (if for no other reason than to remove many of the restrictions in the original post). This is not a thing that you can easily do -- typically you would change your model architecture, which means your old parameters don't map over to the new architecture. I've been pretty curious about how they handle this, but unfortunately the blog post doesn't go into much detail, beyond saying that they can in fact handle these kinds of "surgery" issues.

They estimate that this particular model has used 190 petaflop/s-days of compute, putting it [just below AlphaZero](https://blog.openai.com/ai-and-compute/).


It's really refreshing to read a paper with a negative result about their own method (specifically, that the learned behaviors on Toddler do not look natural). It makes me trust the rest of their paper so much more. (A very gameable instinct, I know.) While they were able to find a fairly diverse set of options, and could interpolate between them, their experiments found that using this for hierarchical RL was about as good as training hierarchical RL from scratch. I guess I'm just saying things they've already said -- I think they've done such a great job writing this paper that they've already told me what my opinion about the topic should be, so there's not much left for me to say.

Variational Option Discovery Algorithms

Joshua Achiam, Harrison Edwards, Dario Amodei, Pieter Abbeel

We can hope to do hierarchical reinforcement learning by first discovering several useful simple policies (or "options") by just acting in the environment without any reward function, and then using these options as primitive actions in a higher level policy that learns to do some task (using a reward function). How could we learn the options without a reward function though? Intuitively, we would like to learn behaviors that are different from each other. One way to frame this would be to think of this as an encoder-decoder problem. Suppose we want to learn K options. Then, we can give the encoder a number in the range [1, K], have it "encode" the number into a trajectory τ (that is, our encoder is a policy), and then have a decoder take τ and recover the original number. We train the encoder/policy and decoder jointly, optimizing them to successfully recover the original number (called a _context_). Intuitively, the encoder/policy wants to have very different behaviors for each option, so that it easy for decoder to figure out the context from the trajectory τ. However, a simple solution would be for the encoder/policy to just take a particular series of actions for each context and then stop, and the decoder learns an exact mapping from final states to contexts. To avoid this, we can decrease the capacity of the decoder (i.e. don't give it too many layers), and we also optimize for the _entropy_ of the encoder/policy, which encourages the encoder/policy to be more stochastic, and so it is more likely to learn overall behaviors that can still have some stochasticity, while still allowing the decoder to decode them. It turns out that this optimization problem has a one-to-one correspondence with variational autoencoders, motivating the name "variational option discovery". To stabilize training, they start with a small K, and increase K whenever the decoder becomes powerful enough. They evaluate in Gym environments, a simulated robotic hand, and a new "Toddler" environment. They find that the scheme works well (in terms of maximizing the objective) in all environments, but that the learned behaviors no longer look natural in the Toddler environment (which is the most complex). They also show that the learned policies can be used for hierarchical RL in the AntMaze problem.

This is very similar to the recent [Diversity Is All You Need](https://arxiv.org/abs/1802.06070). DIAYN aims to decode the context from _every state_ along a trajectory, which incentivizes it to find behaviors of the form "go to a goal state", whereas VALOR (this work) decodes the context from the entire trajectory (without actions, which would make the decoder's job too easy), which allows it to learn behaviors with motion, such as "go around in a circle".


This is pretty exciting -- transferring a policy from simulation to the real world is notoriously hard, but it turns out that as long as you use domain randomization (and 30x the compute) it actually is possible to transfer the policy. I wish they had compared the success probability in simulation to the success probability in the real world -- right now I don't know how well the policy transferred. (That is, I want to evaluate how well domain randomization solved the distribution shift problem.) Lots of other exciting things too, but they are pretty similar to the exciting things about OpenAI Five, such as the ability to learn higher level strategies like finger pivoting and sliding (analogously, fighting over mid or 5-man push).

Learning Dexterity

Most current experiments with robotics work on relatively small state spaces (think 7 degrees of freedom, each a real number) and are trained in simulation. If we could throw a lot of compute at the problem, could we do significantly better? Yes! Using the same general approach as with [OpenAI Five](https://blog.openai.com/openai-five/), OpenAI has built a system called Dactyl, which allows a physical real-world dexterous hand to manipulate a block. It may not seem as impressive as the videos of humanoids running through obstacle courses, but this is way harder than your typical Mujoco environment, especially since they aim to get it working on a real robot. As with OpenAI Five, they only need a reward function (I believe not even a shaped reward function in this case), a simulator, and a good way to explore. In this setting though, "exploration" is actually domain randomization, where you randomly set parameters that you are uncertain about (such as the coefficient of friction between two surfaces), so that the learned policy is robust to distribution shift from the simulator to the real world. (OpenAI Five also used domain randomization, but in that case it was not because we were uncertain about the parameters in the simulator, but because the policy was too specialized to the kinds of characters and heroes it was seeing, and randomizing those properties exposed it to a wider variety of scenarios so it had to learn more general policies.) They use 6144 CPU cores and 8 GPUs, which is _much_ less than for OpenAI Five, but _much_ more than for a typical Mujoco environment.

They do separate the problem into two pieces -- first, they learn how to map from camera pictures to a 3D pose (using convolutional nets), and second, they use RL to choose actions based on the 3D pose. They can also get better estimates of the 3D pose using motion tracking. They find that the CNN is almost as good as motion tracking, and that the domain randomization is crucial for getting the system to actually work.

They also have a couple of sections on surprising results and things that didn't work. Probably the most interesting part was that they didn't need to use the tactile sensors to get these results. They couldn't get these sensors in simulation, so they just did without and it seems to have worked fine. It also turns out that the robot's reaction time wasn't too important -- there wasn't a big difference in changing from 80ms reaction time to 40ms reaction time; in fact, this just increased the required training time without much benefit.

Probably the most interesting part of the post is the last paragraph (italics indicates my notes): "This project completes a full cycle of AI development that OpenAI has been pursuing for the past two years: we’ve developed a new learning algorithm _(PPO)_, scaled it massively to solve hard simulated tasks _(OpenAI Five)_, and then applied the resulting system to the real world _(this post)_. Repeating this cycle at increasing scale is the primary route we are pursuing to increase the capabilities of today’s AI systems towards safe artificial general intelligence."


This seems like a great example of the approach suggested in the previous paper.

Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning

Pedro A. Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J. Gershman, Joshua B. Tenenbaum

Deep reinforcement learning algorithms require many more samples to learn a new game than a human would need: humans have rich priors and theories of how games work that allow them to perform directed exploration and quickly learn the rules of the game. This paper hypothesizes that by providing agents with this rich prior knowledge, we can create agents that learn to play new games as quickly as humans do. The two main ingredients are (1) allowing agents to reason directly over objects, agents, physics and goals (rather than pixels) and (2) using algorithms designed to exploit this prior knowledge. In particular, given this well structured space, they propose EMPA, which uses three main algorithms to exploit the prior knowledge:

**Model learning:** The agent maintains a distribution over possible game mechanics and updates it using Bayes Rule as it takes more actions. This allows it to quickly learn that certain objects tend to kill you, whereas deep RL may require thousands of interactions in order to do the same.

**Exploration:** Exploration is important to the extent that it allows the agent to reduce its uncertainty over the game mechanics. Since we have a distribution over the game mechanics, we could explore in a way that best reduces the uncertainty in that distribution. But in fact our prior knowledge allows us to do something simpler: we just set “exploration subgoals” that seek to cause a collision between two objects (one of which could be the agent’s avatar).

**Planning:** The planning module chooses actions to take in order to achieve some goal or subgoal (note that the subgoals can be set by the exploration algorithm). It uses search algorithms to find such plans.

They evaluate the agent on a variety of games similar to those in Atari. (I assume they could not evaluate on Atari because they can’t easily extract the required prior knowledge from the Atari game engine.) They find that the agent learns to play the games about as fast as humans do, which in turn is much faster than deep RL algorithms. In addition, the gameplay looks more human-like: for example, both EMPA and humans don’t collide with walls very much, whereas deep RL algorithms collide a lot.


This is quite impressive, since they are learning from the binary win-loss reward signal. I'm surprised that the agents generalized well enough to play alongside humans -- I would have expected that to cause a substantial distributional shift preventing good generalization. They only had 30 agents in their population, so it seems unlikely a priori that this would induce a distribution that included humans. Perhaps Quake III is simple enough strategically that there aren't very many viable strategies, and most strategies are robust to having slightly worse allies? That doesn't seem right though.

DeepMind did a _lot_ of different things to analyze what the agents learned and how they are different from humans -- check out the [paper](https://arxiv.org/pdf/1807.01281.pdf) for details. For example, they showed that the agents are much better at tagging (shooting) at short ranges, while humans are much better at long ranges.

Capture the Flag: the emergence of complex cooperative agents

Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Harris, and Thore Graepel

DeepMind has trained FTW (For The Win) agents that can play Quake III Arena Capture The Flag from raw pixels, given _only_ the signal of whether they win or not. They identify three key ideas that enable this -- population based training (instead of self play), learning an internal reward function, and operating at two timescales (enabling better use of memory). Their ablation studies show that all of these are necessary, and in particular it even outperforms population based training with manual reward shaping. The trained agents can cooperate and compete with a wide range of agents (thanks to the population based training), including humans.

But why are these three techniques so useful? This isn't as clear, but I can speculate. Population based training works well because the agents are trained against a diversity of collaborators and opponents, which can fix the issue of instability that afflicts self-play. Operating at two timescales gives the agent a better inductive bias. They say that it enables the agent to use memory more effectively, but my story is that it lets it do something more hierarchical, where the slow RNN makes "plans", while the fast RNN executes on those plans. Learning an internal reward function flummoxed me for a while, it really seemed like that should not outperform manual reward shaping, but then I found out that the internal reward function is computed from the game points screen, not from the full trajectory. This gives it a really strong inductive bias (since the points screen provides really good features for defining reward functions) that allows it to quickly learn an internal reward function that's more effective than manual reward shaping. It's still somewhat surprising, since it's still learning this reward function from the pixels of the points screen (I assume), but more believable.


I might be more excited by an approach that was able to learn from human games (which are plentiful), and perhaps finetune with RL, in order to develop an approach that could generalize to more tasks in the future, where human data is available but a simulator is not. (Given the ridiculous sample complexity, pure RL with PPO can only be used in tasks with a simulator.) On the other hand, an approach that leveraged human data would necessarily be at least somewhat specific to Dota. A dependence on human data is unlikely to get us to _general_ intelligence, whereas this result suggests that we can solve tasks that have a simulator, exploration strategy, and a dense reward function, which really is pushing the boundary on generality. This seems to be [gdb's take](https://news.ycombinator.com/item?id=17392802): "We are very encouraged by the algorithmic implication of this result — in fact, it mirrors closely the story of deep learning (existing algorithms at large scale solve otherwise unsolvable problems). If you have a very hard problem for which you have a simulator, our results imply there is a _real, practical path_ towards solving it. This still needs to be proven out in real-world domains, but it will be very interesting to see the full ramifications of this finding."

OpenAI Five

OpenAI has trained a team of five neural networks to play a particular set of Dota heroes in a mirror match (playing against the same set of heroes) with a few restrictions, and have started to beat amateur human players. They are aiming to beat a team of top professionals at The International in August, with the same set of five heroes, but without any other restrictions. Salient points:
- The method is remarkably simple -- it's a scaled up version of PPO with training data coming from self-play, with reward shaping and some heuristics for exploration, where each agent is implemented by an LSTM.
- There's no human data apart from the reward shaping and exploration heuristics.
- Contrary to most expectations, they didn't need anything fundamentally new in order to get long-term strategic planning. I was particularly surprised by this. Some interesting thoughts from OpenAI researchers in [this thread](https://news.ycombinator.com/item?id=17392700) -- in particular, assuming good exploration, the variance of the gradient should scale linearly with the duration, and so you might expect you only need linearly more samples to counteract this.
- They used 256 dedicated GPUs and 128,000 preemptible CPUs. A [Hacker News comment](https://news.ycombinator.com/item?id=17394150) estimates the cost at $2500 per hour, which would put the likely total cost in the millions of dollars.
- They simulate 900 years of Dota every day, which is a ratio of ~330,000:1, suggesting that each CPU is running Dota ~2.6x faster than real time. In reality, it's probably running many times faster than that, but preemptions, communication costs, synchronization etc. all lead to inefficiency.
- There was no explicit communication mechanism between agents, but they all get to observe the full Dota 2 state (_not_ pixels) that any of the agents could observe, so communication is not really necessary.
- A version of the code with a serious bug was still able to train to beat humans. Not encouraging for safety.
- Alex Irpan covers some of these points in more depth in [Quick Opinions on OpenAI Five](https://www.alexirpan.com/2018/06/27/dota-2-five.html).
- Gwern [comments](https://www.reddit.com/r/reinforcementlearning/comments/8tqzvq/openai_dota_update_ppo_lstm_reaches_amateurlevel/) as well.


The whole blog post is worth reading, but I particularly agree with his point that data augmentation generally seems like a no-brainer, since you can think of it either as increasing the size of your dataset by some constant factor, or as a way of eliminating spurious correlations that your model might otherwise learn.

A Reinforcement Learning Potpourri

This blog post summarizes several recent papers in RL (including the data augmentation papers I summarized above, as well as [First Return Then Explore](https://arxiv.org/abs/2004.12919), the successor to <@Go-Explore@>(@Montezuma’s Revenge Solved by Go-Explore, a New Algorithm for Hard-Exploration Problems@).


On the one hand, this work feels like the achievement of an important milestone in DeepMind's ongoing research agenda towards building more general agents. On the other hand, it has the flavour of engineered sophistry: a remarkable collection of building blocks arranged together to patch specific known weaknesses, but lacking in core insights about how to make agents more general, without, say, making them more complex.

The work is well presented and accessible, especially the blogpost that contains a snapshot of the functional development of deep reinforcement learning capabilities over time. There are several open questions from here on out; personally, I hope this progresses to a single instance of an agent that is proficient at multiple games, and to the design of agents that do not require extensive hyperparameter tuning. The scale of DeepMind's experiments continues to grow, with 256 actors, and 10s of billions of frames, suggesting that, for now, this work is only suitable for simulated environments.

Paper: Agent57: Outperforming the Atari Human Benchmark

Agent57: Outperforming the human Atari benchmark

Adrià Puigdomènech Badia*, Bilal Piot*, Steven Kapturowski*, Pablo Sprechmann*, Alex Vitvitskyi, Daniel Guo, Charles Blundell

This blogpost and its associated [arxiv publication](https://arxiv.org/abs/2003.13350) present Agent57, DeepMind's latest RL agent created for the purpose of achieving human-level performance in a suite of 57 Atari games. Notably, Agent57 is the first agent that is able to surpass average human performance, as measured by Human Normalized Score or HNS, on every individual game in the suite, with the same set of hyperparameters. The blogpost details the evolution of DeepMind's Atari agents from DQN up to Agent57, and the paper elaborates on the improvements made in Agent57.

Specifically, Agent57 builds on a recent agent 'Never Give Up' (NGU), which itself augments R2D2 with episodic memory for curiosity-driven exploration. Agent57 introduces (i) a new parameterization of state-action value function that decomposes into intrinsic and extrinsic rewards, and (ii) a meta-controller which selects which of its numerous distributed policies to prioritize during learning, allowing the agent to control the exploration/exploitation trade-off.


I think this is pretty strong evidence that catastrophic interference is in fact a problem with the Atari games. On the other hand, <@OpenAI Five@> also has many, many subtasks, that in theory should interfere with each other, and it still seems to train well. Some guesses at how to reconcile these facts:

1) the tasks in Dota are more correlated than in (say) Montezuma’s Revenge, and so interference is less of a problem (seems plausible)
2) the policy in OpenAI Five was large enough that it could easily allocate separate capacity for various subtasks (seems unlikely, I believe the policy was relatively small), or
3) with sufficiently large-scale training, there is more “exploration” in weight-space until a configuration is found where interference doesn’t happen (seems unlikely given that large batch sizes help, since they tend to reduce weight-space exploration).

On Catastrophic Interference in Atari 2600 Games

William Fedus*, Dibya Ghosh*, John D. Martin, Marc G. Bellemare, Yoshua Bengio, Hugo Larochelle

One common worry with deep learning is the possibility of _catastrophic interference_: as the model uses gradients to learn a new behaviour, those same gradients cause it to forget past behaviours. In model-free deep RL, this would be particularly harmful in long, sequential tasks as in hard exploration problems like Montezuma’s Revenge: after the model learns how to do the first few subtasks, as it is trying to learn the next subtask, it would “forget” the first subtasks, degrading performance. The authors set out to test this hypothesis.

If this hypothesis were true, there would be an easy way to improve performance: once you have learned to perform the first subtask, just create a brand new neural net for the next subtask, so that training for this next subtask doesn’t interfere with past learning. Since the new agent has no information about what happened in the past, and must just “pick up” from wherever the previous agent left off, it is called the Memento agent (a reference to the movie of the same name). One can then solve the entire task by executing each agent in sequence.

In practice, they train an agent until its reward plateaus. They train a new Memento agent starting from the states that the previous agent reached, and note that it reliably makes further progress in hard exploration games like Montezuma’s Revenge, and not in “steady-state” games like Pong (where you wouldn’t expect as much catastrophic interference). Of course, with the Memento agent, you get both twice the training time and twice the model size, which could explain the improvement. They compare against giving the original agent twice the compute and model capacity, and find that Memento still does significantly better. They also present some fine-grained experiments which show that for a typical agent, training on specific contexts adversely affects performance on other contexts that are qualitatively different.


One of the dangers of training powerful AI to maximize a reward function is that optimizing the function to extreme values may no longer correlate with what we want, as in the classic paperclip maximizer example. I think RCP provides an interesting solution to that problem; if we can instead specify a good, but reasonable, value, we may be able to avoid those extreme cases. We can then gradually increase the desired reward without retraining while continuously monitoring for issues. I think there are likely flaws in the above scheme, but I am optimistic in general about the potential of finding alternate ways to communicate goals to an agent.

One piece I am still curious about is whether the policy remembers how to achieve lower rewards as its training dataset updates towards higher rewards. They show in a heatmap that the target and actual rewards do match up well, but the target rewards are all sampled quite near each other; it would be interesting to see how well the final policy generalizes to the entire spectrum of target rewards.

Reward-Conditioned Policies

Aviral Kumar, Xue Bin Peng, Sergey Levine

Standard RL algorithms create a policy that maximizes a reward function; the *Reward-Conditioned Policy* algorithm instead creates a policy that can achieve a particular reward value passed in as an input. This allows the policy to be trained via supervised regression on a dataset. Each example in the dataset consists of a state, action, and either a return or an advantage, referred to as *Z*. The network then predicts the action based on the state and *Z*. The learned model is able to generalize to policies for larger returns. During training, the target value is sampled from a distribution that gradually increases so that it continues to learn higher rewards.

During evaluation, they then feed in the state and a high target value of *Z* (set one standard deviation above the average in their paper.) This enables them to achieve solid - but not state of the art - performance on a variety of the OpenAI Gym benchmark tasks. They also run ablation studies showing, among other things, that the policy is indeed accurate in achieving the target reward it aims for.


This seems like a useful benchmark. I find it particularly interesting that their experiment testing non-procedurally generated levels as training samples implies huge overfitting effects in existing agents trained in video-game environments.

Paper: Leveraging Procedural Generation to Benchmark Reinforcement Learning

Procgen Benchmark

Karl Cobbe, Christopher Hesse, Jacob Hilton, John Schulman

Existing game-based benchmarks for reinforcement learners suffer from the problem that agents constantly encounter near-identical states, meaning that the agents may be overfitting and memorizing specific trajectories rather than learning a general set of skills. In an attempt to remedy this, in this post OpenAI introduces Procgen Benchmark, 16 procedurally-generated video game environments used to measure how quickly a reinforcement learning agent learns generalizable skills.

The authors conduct several experiments using the benchmark. Notably, they discover that:
- Agents strongly overfit to small training sets and need access to as many as 10,000 levels to generalize appropriately.
- After a certain threshold, training performance improves as the training set grows, counter to trends in other supervised learning tasks.
- Using a fixed series of levels for each training sample (as other benchmarks do) makes agents fail to generalize to randomly generated series of levels at test time.
- Larger models improve sample efficiency and generalization.


This work warrants a visit for the easy-to-absorb animations and charts. On the other hand, they make a few innocent-sounding observations that made me uncomfortable because they weren't rigourously proved nor labelled as speculation, e.g. a) "At higher peek probabilities, the learned dynamics model is not needed to solve the task thus is never learned.", and b) "Here, the world model clearly only learns reliable transition maps for moving down and to the right, which is sufficient."

While this is a neat bit of work well presented, it is nevertheless still unlikely this (and most other current work in deep model-based RL) will scale to more complex alignment problems such as <@Embedded World-Models@>; these world models do not capture the notion of an agent, and do not model the agent as an entity making long-horizon plans in the environment.

Learning to Predict Without Looking Ahead: World Models Without Forward Prediction

One [critique](https://twitter.com/shimon8282/status/979344417961250817) of the <@World Models@>(@Recurrent World Models Facilitate Policy Evolution@) paper was that in any realistic setting, you only want to learn the features that are important for the task under consideration, while the VAE used in the paper would learn features for state reconstruction. This paper instead studies world models that are trained directly from reward, rather than by supervised learning on observed future states, which should lead to models that only focus on task-relevant features. Specifically, they use _observational dropout_ on the environment percepts, where the true state is passed to the policy with a peek probability _p_, while a neural network, **M**, generates a proxy state with probability _1 - p_. At the next time-step, **M**  takes the same input as the policy, plus the policy's action, and generates the next proxy state, which then may get passed to the controller, again with probability _1 - p_.

They investigate whether the emergent 'world model' **M** behaves like a good forward predictive model. They find that even with very low peek probability e.g. _p_ = 5%, **M** learns a good enough world model that enables the policy to perform reasonably well. Additionally, they find that world models thus learned can be used to train policies that sometimes transfer well to the real environment. They claim that the world model only learns features that are useful for task performance, but also note that interpretability of those features depends on inductive biases such as the network architecture.


From what I understand, much of the difficulty of poker lies in being careful not to reveal information. For decades, computers have already had an upper hand in being silent, computing probabilities, and choosing unpredictable strategies, which makes me a bit surprised that this result took so long. Nonetheless, I found it interesting how little compute was required to accomplish superhuman play.

Let's Read: Superhuman AI for multiplayer poker

Superhuman AI for multiplayer poker

In July, this paper presented the first AI that can play six-player no-limit Texas hold’em poker better than professional players. Rather than using deep learning, it works by precomputing a blueprint strategy using a novel variant of Monte Carlo linear counterfactual regret minimization, an iterative self-play algorithm. To traverse the enormous game tree, the AI buckets moves by abstracting information in the game. During play, the AI adapts its strategy by modifying its abstractions according to how the opponents play, and by performing real-time search through the game tree. It used the equivalent of $144 of cloud compute to calculate the blueprint strategy and two server grade CPUs, which was much less hardware than what prior AI game milesones required.


I think this is a very impressive display of how powerful current ML methods are at a very complex game. StarCraft poses many challenges that are not present in board games such as chess and go, such as limited visibility, a large state and action space, and strategies that play out over very long time horizons. I found it particularly interesting how they used imitation learning and human examples to avoid trying to find new strategies by exploration, but then attained higher performance by training on top of that.

I do believe progress on games is becoming less correlated with progress on AGI. Most of the key innovations in this paper revolve around the League training, which seems quite specific to StarCraft. In order to continue making progress towards AGI, I think we need to focus on being able to learn in the real world on tasks that are not as easy to simulate.

Paper: Grandmaster level in StarCraft II using multi-agent reinforcement learning

AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning

<@AlphaStar@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@), DeepMind’s StarCraft II AI, has now defeated a top professional player and is better than 99.8% of players. While previous versions were limited to only a subset of the game, it now plays the full game and has limitations on how quickly it can take actions similar to top human players. It was trained initially via supervised learning on human players and then afterwards trained using RL.

A challenge in learning StarCraft via self-play is that strategies exhibit non-transitivity: Stalker units beat Void Rays, Void Rays beat Immortals, but Immortals beat Stalkers. This can lead to training getting stuck in cycles. In order to avoid this, they set up a League of exploiter agents and main agents. The exploiter agents train only against the current iteration of main agents, so they can learn specific counter-strategies. The main agents then train against a mixture of current main agents, past main agents, and exploiters, prioritizing opponents that they have a lower win rate against.


The videos look impressive, even though the robot hand still has some clunkiness to it. My intuition is that model-based approaches can be very useful in robotics and similar domains, where the randomness in transitions can easily be approximated by Gaussians. In other tasks where transitions follow more complicated, multimodal distributions, I am more sceptical.

Deep Dynamics Models for Dexterous Manipulation

Anusha Nagabandi, Kurt Konoglie, Sergey Levine, Vikash Kumar

For hard robotic tasks like manipulating a screwdriver, model-free RL requires large amounts of data that are hard to generate with real-world hardware. So, we might want to use the more sample-efficient model-based RL, which has the additional advantage that the model can be reused for similar tasks with different rewards. This paper uses an ensemble of neural networks to predict state transitions, and plans by sampling trajectories for different policies. With this, they train a real anthropomorphic robot hand to be able to rotate two balls in its hand somewhat reliably within a few hours. They also trained for the same task in a simulation and were able to reuse the resulting model to move a single ball to a target location.


As usual, Alex's analysis is spot on, and I have nothing to add beyond strong agreement.

Let's Discuss OpenAI's Rubik's Cube Result

This post makes many points about <@OpenAI's Rubik's cube result@>(@Solving Rubik’s Cube with a Robot Hand@), but I'm only going to focus on two. First, the result is a major success for OpenAI's focus on design decisions that encourage long-term research success. In particular, it relied heavily on the engineering-heavy model surgery and policy distillation capabilities that allow them to modify e.g. the architecture in the middle of a training run (which we've seen with <@OpenAI Five@>(@OpenAI Five Benchmark: Results@)). Second, the domain randomization doesn't help as much as you might think: OpenAI needed to put a significant amount of effort into improving the simulation to get these results, tripling the number of successes on a face rotation task. Intuitively, we still need to put in a lot of effort to getting the simulation to be "near" reality, and then domain randomization can take care of the last little bit needed to robustly transfer to reality. Given that domain randomization isn't doing that much, it's not clear if the paradigm of zero-shot sim-to-real transfer is the right one to pursue. To quote the post's conclusion:  _I see two endgames here. In one, robot learning reduces to building rich simulators that are well-instrumented for randomization, then using ludicrous amounts of compute across those simulators. In the other, randomization is never good enough to be more than a bootstrapping step before real robot data, no matter what the compute situation looks like. Both seem plausible to me, and we’ll see how things shake out._


My impression is that this is a very impressive robotics result, largely because the problem of transferring training in simulation to real life ("sim2real") is extremly difficult. I also think it's quite novel if as the authors hypothesize, the system is exhibiting emergent meta-learning. It's worth noting that the hand is still not quite at human-level -- in the hardest configurations, it only succeeds 20% of the time, and for most experiments, the hand gets some of the state of the cube via Bluetooth sensors inside the cube, not just via vision.

Vox: Watch this robot solve a Rubik’s Cube one-handed

Solving Rubik’s Cube with a Robot Hand

Historically, researchers have had limited success making general purpose robot hands. Now, OpenAI has successfully trained a pair of neural networks to solve a Rubik's cube with a human-like robot hand (the learned portion of the problem is manipulating the hand -- solving the Rubik's cube is specified via a classical algorithm). The hand is able to solve the Rubik's cube even under a variety of perturbations, including having some of its fingers tied together, or having its view of the cube partially occluded. The primary innovation presented is a new method called _Automatic Domain Randomization_ (ADR). ADR automatically generates progressively more difficult environments to train on in simulation that are diverse enough to capture the physics of the real world. ADR performs better than existing domain randomization methods, which require manually specifying randomization ranges. The post speculates that ADR is actually leading to _emergent meta-learning_, where the network learns a learning algorithm that allows itself to rapidly adapt its behavior to its environment.


This is somewhat similar to <@AI-GAs@>(@AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence@): both depend on _environment design_, which so far has been relatively neglected. However, AI-GAs are hoping to create _learning algorithms_, while multiagent autocurricula leads to _tool use_, at least in this case. Another point of similarity is that they both require vast amounts of compute, as discovering new strategies can take significant exploration. That said, it seems that we might be able to drastically decrease the amount of compute needed by solving the exploration problem using e.g. human play data or demonstrations (discussed in two different papers above).

More speculatively, I hypothesize that it will be useful to have environments where you need to identify _what strategy your opponent is using_. In this environment, each strategy has the property that it beats _all_ of the strategies that preceded it. As a result, it was fine for the agent to undergo catastrophic forgetting: even though it was trained against past agents, it only needed to learn the current strategy well; it didn't need to remember previous strategies. As a result, it may have forgotten prior strategies and skills, which might have reduced its ability to learn new tasks quickly.

[Paper: Emergent Tool Use from Multi-Agent Autocurricula](https://d4mucfpksywv.cloudfront.net/emergent-tool-use/paper/Multi_Agent_Emergence_2019.pdf), [Vox: Watch an AI learn to play hide-and-seek](https://www.vox.com/future-perfect/2019/9/20/20872672/ai-learn-play-hide-and-seek)

Emergent Tool Use from Multi-Agent Interaction

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, Igor Mordatch

We have such a vast diversity of organisms and behaviors on Earth because of evolution: every time a new strategy evolved, it created new pressures and incentives for other organisms, leading to new behaviors. The multiagent competition led to an _autocurriculum_. This work harnesses this effect: they design a multiagent environment and task, and then use standard RL algorithms to learn several interesting behaviors. Their task is hide-and-seek, where the agents are able to move boxes, walls and ramps, and lock objects in place. The agents find _six_ different strategies, each emerging from incentives created by the previous strategy: seekers chasing hiders, hiders building shelters, seekers using ramps to get into shelters, hiders locking ramps away from seekers, seekers surfing boxes to hiders, and hiders locking both boxes and ramps.

The hope is that this can be used to learn general skills that can then be used for specific tasks. This makes it a form of unsupervised learning, with a similar goal as e.g. <@curiosity@>(@Large-Scale Study of Curiosity-Driven Learning@). We might hope that multiagent autocurricula would do better than curiosity, because they automatically tend to use features that are important for control in the environment (such as ramps and boxes), while intrinsic motivation methods often end up focusing on features we wouldn't think are particularly important. They empirically test this by designing five tasks in the environment and checking whether finetuning the agents from the multiagent autocurricula learns faster than direct training and finetuning curiosity-based agents. They find that the multiagent autocurricula agents do best, but only slightly. To explain this, they hypothesize that the learned skill representations are still highly entangled and so are hard to finetune, whereas learned feature representations transfer more easily.


This article is a solid and useful version of what I would expect out of a review article: mostly useful as a way to get thinking in the direction of the intersection of RL and language, and makes me more interested in digging more into some of the mentioned techniques, since by design this review didn't go very deep into any of them.

A Survey of Reinforcement Learning Informed by Natural Language

Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, Tim Rocktäschel

Humans use language as a way of efficiently storing knowledge of the world and instructions for handling new scenarios; this paper is written from the perspective that it would be potentially hugely valuable if RL agents could leverage information stored in language in similar ways. They look at both the case where language is an inherent part of the task (example: the goal is parameterized by a language instruction) and where language is used to give auxiliary information (example: parts of the environment are described using language). Overall, the authors push for more work in this area, and, in particular, more work using external-corpus-pretrained language models and with research designs that use human-generated rather than synthetically-generated language; the latter is typically preferred for the sake of speed, but the former has particular challenges we'll need to tackle to actually use existing sources of human language data.