I'm interested by this paper's finding that you can generate apparently altruistic behavior by incentivizing agents to influence others, rather than necessarily help others. I also appreciate the point that was made to train in a decentralized way. I'd love to see more work on a less asymmetric version of influence reward; currently influencers and influencees are separate groups due to worries about causal feedback loops, and this implicitly means there's a constructed group of quasi-altruistic agents who are getting less concrete reward because they're being incentivized by this auxiliary reward.

Social Influence as Intrinsic Motivation for Multi-Agent Deep RL

Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A. Ortega, DJ Strouse, Joel Z. Leibo, Nando de Freitas

An emerging field of common-sum multi-agent research asks how to induce groups of agents to perform complex coordination behavior to increase general reward, and many existing approaches involve centralized training or hardcoding altruistic behavior into the agents. This paper suggests a new technique that rewards agents for having a causal influence over the actions of other agents, in the sense that the actions of the pair of agents agents have high mutual information. The authors empirically find that having even a small number of agents who act as "influencers" can help avoid coordination failures in partial information settings and lead to higher collective reward.  In one sub-experiment, they only add this influence reward to the agents' communication channels, so agents are incentivized to provide information that will impact other agents' actions (this information is presumed to be truthful and beneficial since otherwise it would subsequently be ignored).


On the Convergence of Competitive, Multi-Agent Gradient-Based Learning


I'm not sure what this is adding over two-player game theory (for which we can compute equilibria) but again I only skimmed the paper so it's quite likely that I missed something.

Reinforcement Learning under Threats

Víctor Gallego, Roi Naveiro, David Ríos Insua

Due to lack of time, I only skimmed this paper for 5 minutes, but my general sense is that it takes MDPs and turns them into two player games by positing the presence of an adversary. It modifies the Bellman update equations to handle the adversary, but runs into the usual problems of simulating an adversary that simulates you. So, it formalizes level-k thinking (simulating an opponent that thinks about you at level k-1), and evaluates this on matrix games and the friend-or-foe environment from [AI safety gridworlds](https://deepmind.com/blog/specifying-ai-safety-problems/).


Multiagent settings seem very important (since there does happen to be more than one human in the world). This looks like a useful generalization from the single agent case to the multiagent case, though it's not clear to me that this deals with the major challenges that come from multiagent scenarios. One major challenge is that there is no longer a single optimal equilibrium when there are multiple agents, but they simply assume in their theoretical analysis that there is only one solution. Another one is that it seems more important that the policies take history into account somehow, but they don't do this. (If you don't take history into account, then you can't learn strategies like tit-for-tat in the iterated prisoner's dilemma.) But to be clear I think this is the standard setup for multiagent RL -- it seems like field is not trying to deal with this issue yet (even though they could using eg. a recurrent policy, I think?)

Multi-Agent Generative Adversarial Imitation Learning

Jiaming Song, Hongyu Ren, Dorsa Sadigh, Stefano Ermon

This paper generalizes [GAIL](http://www.jonathanho.me/files/HoErmon_NIPS2016.pdf) (which was covered [last week](https://mailchi.mp/ad852629e45a/alignment-newsletter-17)) to the multiagent setting, where we want to imitate a group of interacting agents. They want to find a Nash equilibrium in particular. They formalize the Nash equilibrium constraints and use this to motivate a particular optimization problem for multiagent IRL, that looks very similar to their optimization problem for regular IRL in GAIL. After that, it is quite similar to GAIL -- they use a regularizer ψ for the reward functions, show that the composition of multiagent RL and multiagent IRL can be solved as a single optimization problem involving the convex conjugate of ψ, and propose a particular instantiation of ψ that is data-dependent, giving an algorithm. They do have to assume in the theory that the multiagent RL problem has a unique solution, which is not typically true, but may not be too important. As before, to make the algorithm practical, they structure it like a GAN, with discriminators acting like reward functions. What if we have prior information that the game is cooperative or competitive? In this case, they propose changing the regularizer ψ, making it keep all the reward functions the same (if cooperative), making them negations of each other (in two-player zero-sum games), or leaving it as is. They evaluate in a variety of simple multiagent games, as well as a plank environment in which the environment changes between training and test time, thus requiring the agent to learn a robust policy, and find that the correct variant of MAGAIL (cooperative/competitive/neither) outperforms both behavioral cloning and single-agent GAIL (which they run N times to infer a separate reward for each agent).


Weak identifiability and its consequences in strategic settings

Inverse reinforcement learning suffers from the problem of unidentifiability: even given large amounts of data, it is not possible to uniquely recover the true reward function. This can lead to poor predictions if assumptions change (e.g. if there is some distributional shift, or if you are trying to correct for some bias like hyperbolic discounting).

This post demonstrates how a similar failure can affect multiagent settings as well, using the ultimatum game as an example. In the ultimatum game, there are two players: the Proposer and the Responder. The Proposer suggests a way to split $10 between the two players, and the Responder decides either to accept or reject the offer. If the offer is accepted, then the players get money according to the proposed split. If the offer is rejected, neither player gets anything.

Let’s suppose we get to observe how a particular Responder plays in an iterated ultimatum game, where we see as much data as we want. We figure out that the Responder will reject any split where it gets under $4. We could posit two explanations for this behavior:

1. Reputation-building: The Responder is building a reputation of refusing unfair splits (defined as a split where it gets < $4), so that it is offered better splits in the future.
2. Commitment: The Responder may have committed in advance to always refuse unfair splits (for the same reason, or perhaps because the Responder intrinsically dislikes unfair deals).

Note that both explanations perfectly account for all the data (no matter how much data we get).

Suppose the Responder has committed to rejecting unfair deals, but we incorrectly believe that it does it for reputation-building. Let’s say we now play a _one-shot_ ultimatum game with the Responder. We reason that it no longer needs to build reputation, and so it will accept a 9/1 split. However, in fact it has _committed_ to avoid unfair splits, and so rejects our offer. The post also gives a mathematical formalization of this example.


This is an interesting formalization of friendly and adversarial behavior. It feels somewhat weird to model the environment as having a prior strategy that it can then update. This has the implication that a "somewhat friendly" environment is unable to change its strategy to help the agent, even though it would "want" to, whereas when I think of a "somewhat friendly" environment, I think of a group of agents that share some of your goals but not all of them, so a limited amount of cooperation is possible. These feel quite different.

Modeling Friends and Foes

Multiagent scenarios are typically modeled using game theory. However, it is hard to capture the intuitive notions of "adversarial", "neutral" and "friendly" agents using standard game theory terminology. The authors propose that we model the agent and environment as having some prior mixed strategy, and then allow them to "react" by changing the strategies to get a posterior strategy, but with a term in the objective function for the change (as measured by the KL divergence). The sign of the environment's KL divergence term determines whether it is friendly or adversarial, and the magnitude determines the magnitude of friendliness or adversarialness. They show that there are always equilibria, and give an algorithm to compute them. They then show some experiments demonstrating that the notions of "friendly" and "adversarial" they develop actually do lead to behavior that we would intuitively call friendly or adversarial.

Some notes to understand the paper: while normally we think of multiagent games as consisting of a set of agents, in this paper there is an agent that acts, and an environment in which it acts (which can contain other agents). The objective function is neither minimized nor maximized -- the sign of the environment's KL divergence changes whether the stationary points are maxima or minima (which is why it can model both friendly and adversarial environments). There is only one utility function, the agent's utility function -- the environment is only modeled as responding to the agent, rather than having its own utility function.


Learning Social Conventions in Markov Games


I’m a big fan of moving towards modeling humans as part of the environment, since we will eventually have AI systems working with and interacting with humans -- they won’t be “external to the AI’s universe” as it is often modeled currently.

Multi-agent Social Reinforcement Learning Improves Generalization

Kamal Ndousse, Douglas Eck, Sergey Levine, Natasha Jaques

We’ve previously seen that in sparse reward settings where exploration is hard, it’s very useful to have expert demonstrations to avoid having to do all the exploration yourself (<@1@>(@Learning Montezuma’s Revenge from a Single Demonstration@), <@2@>(@Making Efficient Use of Demonstrations to Solve Hard Exploration Problems@), <@3@>(@Playing hard exploration games by watching YouTube@)). However, this assumes that the demonstrator is “external” to the environment, whereas really we’d like to model them as part of the environment, as in <@assistance games@>(@Cooperative Inverse Reinforcement Learning@). This then looks like _social learning_, in which agents learn how to perform tasks by looking at cues from other agents within the environment.

But how can we do this in high-dimensional environments? This paper looks at one approach: adding an auxiliary loss in which the agent must predict the next state of the environment. Since the environment itself contains experts that do useful things, the agent implicitly must learn what those experts are doing and what effects their actions have.

They find that such agents learn to follow the cues of the experts and thus achieve significantly improved reward relative to agents that are trained in isolation. In fact, these agents can be transferred to novel environments, where they continue to follow expert cues to achieve high reward. However, this means that they don’t learn how to act when experts aren’t present, and so fail in the solo setting. This can be fixed by training on a mixture of solo settings and settings with experts present.


Comparing Utilities

This is a reference post about preference aggregation across multiple individually rational agents (in the sense that they have [VNM-style](https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem) utility functions), that explains the following points (among others):

1. The concept of “utility” in ethics is somewhat overloaded. The “utility” in hedonic utilitarianism is very different from the VNM concept of utility. The concept of “utility” in preference utilitarianism is pretty similar to the VNM concept of utility.
2. Utilities are not directly comparable, because affine transformations of utility functions represent exactly the same set of preferences. Without any additional information, concepts like “utility monster” are type errors.
3. However, our goal is not to compare utilities, it is to aggregate people’s preferences. We can instead impose constraints on the aggregation procedure.
4. If we require that the aggregation procedure produces a Pareto-optimal outcome, then Harsanyi’s utilitarianism theorem says that our aggregation procedure can be viewed as maximizing some linear combination of the utility functions.
5. We usually want to incorporate some notion of fairness. Different specific assumptions lead to different results, including variance normalization, Nash bargaining, and Kalai-Smorodinsky.


This is a cool theoretical framework. Its relevance depends on how likely you think it is that social cognition will be a core component of AGI, as opposed to just another task to be solved using general-purpose reasoning. I imagine that most AI safety researchers lean towards the latter, but there are some reasons to give credence to the former.

Theory of Minds: Understanding Behavior in Groups Through Inverse Planning

Michael Shum, Max Kleiman-Weiner, Michael L. Littman, Joshua B. Tenenbaum

This paper introduces Composable Team Hierarchies (CTH), a representation designed for reasoning about how agents reason about each other in collaborative and competitive environments. CTH uses two "planning operators": the Best Response operator returns the best policy in a single-agent game, and the Joint Planning operator returns the best team policy when all agents are cooperating. Competitive policies can then be derived via recursive application of those operations to subsets of agents (while holding the policies of other agents fixed). CTH draws from ideas in level-K planning (in which each agent assumes all other agents are at level K-1) and cooperative planning, but is more powerful than either approach.

The authors experiment with using CTH to probabilistically infer policies and future actions of agents participating in the stag-hunt task; they find that these judgements correlate well with human data.


It's great to see work on the multiagent setting! This setting does seem quite a bit more complex, and hasn't been explored very much from the AI safety standpoint. One major question I have is how this relates to the work already done in academia for different settings (typically groups of humans instead of AI agents). Quick takes on how each failure mode is related to existing academic work: Accidental steering is novel to me (but I wouldn't be surprised if there has been work on it), coordination failures seem like a particular kind of (large scale) prisoner's dilemma, adversarial misalignment is a special case of the principal-agent problem, input spoofing and filtering and goal co-option seem like special cases of adversarial misalignment (and are related to ML security as the paper points out).

Multi-Agent Overoptimization, and Embedded Agent World Models

This post and the associated [paper](https://arxiv.org/abs/1810.10862) argue for the complexity of multiagent settings, where you must build a model of how other agents act, even though they have models of how you act. While game theory already deals with this setting, it only does so by assuming that the agents are perfectly rational, an assumption that doesn't hold in practice and doesn't grapple with the fact that your model of the opponent cannot be perfect. The paper lists a few failure modes. Accidental steering happens when one agent takes action without the knowledge of what other agents are doing. Coordination failures are exactly what they sound like. Adversarial misalignment happens when one agent chooses actions to mislead a victim agent into taking actions that benefit the first agent. Input spoofing and filtering happen when one agent doctors the training data for a victim agent. Goal co-option occurs when one agent takes control over the other agent (possibly by modifying their reward function).


CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning

Jiachen Yang, Alireza Nakhaei, David Isele, Hongyuan Zha, Kikuo Fujimura


CLR's recent work on multi-agent systems

This post summarizes recent work by the Center for Long-Term Risk (CLR). The general theme is cooperative AI, with a focus on research that helps avert s-risks. See <@this research agenda@>(@Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda@) for an explanation of what might cause these s-risks. I’ve summarized some of the individual pieces of research below.


Generative Multi-Agent Behavioral Cloning


A multi-agent reinforcement learning model of common-pool resource appropriation

Inequity aversion resolves intertemporal social dilemmas

Edward Hughes, Joel Z. Leibo, Matthew G. Phillips, Karl Tuyls, Edgar A. Duéñez-Guzmán, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin R. McKee, Raphael Koster, Heather Roff, Thore Graepel


M^3RL: Mind-aware Multi-agent Management Reinforcement Learning


Negative Update Intervals in Deep Multi-Agent Reinforcement Learning

Gregory Palmer, Rahul Savani, Karl Tuyls


Emergent Communication through Negotiation

Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z Leibo, Karl Tuyls, Stephen Clark


Conference on Robot Learning (CoRL) 2018

Learning to Share and Hide Intentions using Information Regularization

DJ Strouse, Max Kleiman-Weiner, Josh Tenenbaum, Matt Botvinick, David Schwab


Adaptive Mechanism Design: Learning to Promote Cooperation

Tobias Baumann, Thore Graepel, John Shawe-Taylor


1st Workshop on Goal Specifications for Reinforcement Learning

Multi-Agent Deep Reinforcement Learning with Human Strategies

Thanh Nguyen, Ngoc Duy Nguyen, Saeid Nahavandi


Coordination-driven learning in multi-agent problem spaces

Sean L. Barton, Nicholas R. Waytowich, Derrik E. Asher


Norms for beneficial A.I.: A computational analysis of the societal value alignment problem

Pedro Fernandes, Francisco C. Santos, Manuel Lopes

This paper presents a simple quantitative model to argue for the following two observations:
1. Unless they are willing to “fall behind” others, individual actors will need to use AI systems to stay competitive.
2. Those AI systems will optimize for their owner’s goals, even though a better outcome could be achieved if all AI systems optimized for the average welfare across all actors.


Multi-winner Voting: a question of Alignment


**Overall, this paper is compelling because the method presented is both promising and provides natural ideas for future work.** The method presented here is arguably simpler than HRL and the ability to generalize to new environments is simple to implement. The idea of introducing competition at an information theoretic level seems natural and the evidence for better generalization capability is compelling. It'd be interesting to see what would happen if more complex primitives were used.

Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives

Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, Yoshua Bengio

Learning policies that generalize to new environments is a fundamental challenge in reinforcement learning. In particular, humans seem to be adept at learning skills and understanding the world in a way that is compositional, hinting at the source of the discrepancy. Hierarchical reinforcement learning (HRL) has partially addressed the discrepancy by decomposing policies into options/primitives/subpolicies that a top-level controller selects from. However, generalization is limited because the top-level policy must work for all states.

**In this paper, the authors explore a novel decentralized approach where policies are still decomposed into primitives, but without a top-level controller.** The key idea is to incentivize each primitive to work on a different cluster of states. Every primitive has a variational information bottleneck between the state and predicted action, that allows us to quantify how much information about the state the primitive uses in selecting actions. Intuitively, a primitive that knows how to open gates is going to extract a lot of information about gates from the state to choose an appropriate action, and won’t extract much information in states without gates. So, our high-level controller can just be: check which primitive is using the most state information, and let that primitive choose the action.

The reward R from a trajectory is split amongst the primitives in proportion to how likely each primitive was to be chosen. This is what incentivizes the primitives to use information from the state. The primitives also get a cost in proportion to how much information they use, incentivizing them to specialize to a particular cluster of states. Finally, there is a regularization term that also incentivizes specialization, and in particular prevents a collapse where a single primitive is always active.

To demonstrate effectiveness, the authors compare the baseline HRL methods option-critic and [Meta-learning Shared Hierarchy](https://blog.openai.com/learning-a-hierarchy/) to their method in grid-world and motion imitation transfer tasks. They show that using an ensemble of primitives can outperform more traditional HRL methods in generalization across tasks.


Data-Efficient Hierarchical Reinforcement Learning

Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine


Hierarchical Reinforcement Learning with Deep Nested Agents


Hierarchical Reinforcement Learning with Hindsight


I found this paper interesting, but also found it a bit tricky/unintuitive to read, since it used a different RL frame than I'm used to (the idea of minimizing the KL divergence between your trajectory distribution and the optimal trajectory distribution). Overall, seems like a reasonable method, but is a bit hard to intuitively tell how strong the theoretical advantages are on these relatively simple tasks.

Multitask Soft Option Learning

Maximilian Igl, Andrew Gambardella, Nantas Nardelli, N. Siddharth, Wendelin Böhmer, Shimon Whiteson

This paper is a mix of variational inference and hierarchical reinforcement learning, in the context of learning skills that can be reused across tasks. Instead of learning a fixed set of options (read: skills/subpolicies), and a master task-specific policy to switch between them, this method learns cross-task priors for each skill, and then learns a task-specific posterior using reward signal from the task, but regularized towards the prior. The hope is that this will allow for an intermediary between cross-task transfer and single-task specificity.


Model Primitive Hierarchical Lifelong Reinforcement Learning

Bohan Wu, Jayesh K. Gupta, Mykel J. Kochenderfer


Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization


Hierarchical visuomotor control of humanoids

Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Greg Wayne


Imagine [Twitch plays](https://www.twitch.tv/directory/game/Twitch%20Plays), but you use a reputation to buy and sell your actions. The actual idea in the paper is slightly more mundane than this because the primitives are bidders. <@Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives@> is a similar piece of work that also uses primitives as the basic level of selection. However, their incentive mechanism is different: agents pay according to how much information from the environment they use and then get a reward back for their actions. However, there’s good reason to think options could work as well since in both of these papers there’s evidence that primitives that learn sub-tasks are useful in new tasks.

Paper: Decentralized Reinforcement Learning: Global Decision-Making via Local Economic Transactions

Michael Chang, Sidhant Kaushik, S. Matthew Weinberg, Thomas L. Griffiths, Sergey Levine

Increasing the scalability of learning systems is a central challenge to machine learning. One framework is to organize RL agents as ‘super’ agents, large collections of simpler agents that each make decisions according to their own incentives. If it were possible to get the incentives correct, the dominant equilibria would be identical to the optimal solution for the original RL problem.

In this paper, the authors introduce a framework for decentralizing decision-making by appealing to auction theory. There is a separate simple agent for each action. At every a timestep, a Vickrey auction is run in which each agent can bid for the superagent executing their particular action. The trick is that when an agent successfully wins a bid and acts on a state, it then ‘owns’ the produced next state, and ‘earns’ the result of the auction in the next round. (At the end of an episode, the owner of the state earns the reward of the trajectory.) Intuitively, the agent wants to bid on states in which it can make progress towards earning the final reward, as those will be states that other agents want to buy. The authors show that this scheme incentivizes each agent to bid the Q-value of their action in the given state, which would then lead to an optimal policy.

The authors test out this approach with some simple MDPs. They also investigate a task where they try to get the agents to rotate MNIST images so that a classifier will recognize them. Finally, they investigate task transfer by training agents on simple sub-tasks and then reusing those agents to learn a related task making use of both sub-tasks.


To summarize my summary, this paper assumes we have a POMDP with a labeling function L that extracts important state information from transitions. Given this, they learn a (hopefully perfect) reward machine from experience, and then use the reward machine to learn a policy more efficiently.

I see two main limitations to this method. First, they require a good labeling function L, which doesn’t seem easy to specify (at least if you want a high-level labeling function that only extracts the relevant information). Second, I think their heuristic of using every <o, a, o’> transition as a separate experience for every possible u would not usually work -- even if you learn a perfect reward model (such that the combination of o and u together form a “state” in an MDP), it’s not necessarily true that for every possible state in which you get observation o, when taking action a, you get observation o’. The authors acknowledge this limitation with an example of a gridworld with a button that changes how transitions work. But it seems to me that in practice, the underlying state in a POMDP will often affect the next observation you get. For example, in Minecraft, maybe you get some experience where you chop down a tree, in which your next observation involves you having wood. If you generalize it to all possible states with identical initial observations, you’d also generalize it to the case where there is an enemy behind you who is about to attack. Then, your policy would learn to chop down trees, even when it _knows_ that there is an enemy behind it.

It seems pretty important in RL to figure out how to infer underlying states when working in a POMDP, as it seems like a useful inductive bias for our agents to assume that there is a (Markovian) world “out there”, and I’m excited that people are thinking about this. Due to the two limitations above, I don’t expect that reward machines are the way to go (at least as developed so far), but it’s exciting to see new ideas in this area. (I’m currently most excited about learning a latent state space model, as done in e.g. <@Dreamer@>(@Dream to Control: Learning Behaviors by Latent Imagination@).)

Learning Reward Machines for Partially Observable Reinforcement Learning

Rodrigo Toro Icarte, Ethan Waldie, Toryn Q. Klassen, Richard Valenzano, Margarita P. Castro, Sheila A. McIlraith

Typically in reinforcement learning, the agent only gets access to a reward signal: it sees a single number saying how well it has done. The problem might be simpler to solve if the agent could get a more holistic view of the problem through a structured representation of the reward. This could allow it to infer things like “if I went left, I would get 5 reward, but if I went right, I would get 10 reward”. Under the current RL paradigm, it has to try both actions in separate episodes to learn this.

Model-based RL tries to recover some of this structured representation: it learns a model of the world and the reward function, such that you can ask queries of the form “if I took this sequence of actions, what reward would I get?” The hope is that the learned models will generalize to new sequences that we haven’t previously seen, allowing the agent to learn from fewer environment interactions (i.e. higher sample efficiency).

This work does something similar using _reward machines_. The key idea is to represent both the reward and some aspects of the dynamics using a finite state machine, which can then be reasoned about without collecting more experience. In particular, given a POMDP, they propose learning a set of states U such that when combining the observation o with the state u, we have an MDP instead of a POMDP. This is called a _perfect_ reward machine. To make this feasible, they assume the existence of a labeling function L that, given a transition <o, a, o’>, extracts all of the relevant state information. (Since POMDPs can be reduced to belief-space MDPs, it is always possible to extract a perfect reward machine by having U be the set of possible beliefs and L be the identity function, but the hope is that U and L can be much simpler in most cases.)

They provide a formulation of an optimization problem over finite state machines such that a perfect reward machine would be an optimal solution to that problem (though I believe other imperfect reward machines could also be optimal). Since they are searching over a discrete space, they need to use a discrete optimization algorithm, and end up using Tabu search.

Once they have learned a reward machine from experience and a labeling function L, how can they use it to improve policy learning? They propose a very simple idea: when we get experience <o, a, o’>, treat it as a separate experience for every possible u, so that you effectively multiply the size of your dataset. They can then learn optimal policies that are conditioned on the state u (which can be inferred at test time using the learned state machine). Experiments show that this works in some simple gridworlds.


I think unsupervised learning is likely to be key in getting more powerful and general AI systems without requiring a truly staggering amount of expert data, and this is a great example of what that might look like. Note though that the learned primitives are certainly not what you'd expect of a human: for example, the humanoid learns to vaguely shuffle in a direction, rather than walking. In addition, they did require specifying an "x-y prior" that required skills to be diverse based on x-y coordinates, which is why the skills learned navigation primitives, as opposed to e.g. distinct types of flailing.

Paper: Dynamics-Aware Unsupervised Discovery of Skills

DADS: Unsupervised Reinforcement Learning for Skill Discovery

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, Karol Hausman

Reinforcement learning in robotics typically plans directly on low-level actions. However, it sure seems like there are a simple set of _primitives_ like walking, running, shuffling, etc. that are inherent to the robot morphology. What if we could learn these primitives, and then plan using those primitives? This paper introduces a method for learning these primitives _without a reward function_. They simply optimize skills for _predictability_ and _diversity_ (by optimizing the mutual information between the current state and next state, conditioned on which skill is being executed).

They can then use these primitives for _model-based planning_ for a downstream task. You can think of this as a regular RL problem, except that an action in their "action space" takes the form "execute skill X for T timesteps". They use _model-predictive control_ (MPC), in which you sample a bunch of trajectories, and execute the first action of the trajectory that gets the highest reward. Since each of their high-level actions determines the policy for T timesteps, they can scale to much longer horizon tasks than MPC can usually be used for. They show that this approach is competitive with regular model-based RL.


Unsupervised Discovery of Decision States for Transfer in Reinforcement Learning

Nirbhay Modhe, Prithvijit Chattopadhyay, Mohit Sharma, Abhishek Das, Devi Parikh, Dhruv Batra, Ramakrishna Vedantam


Neural Modular Control for Embodied Question Answering

Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra


I really liked this paper. The method is simple (although computationally expensive), and gives novel insights. I think understanding how deep learning training works is important as it can help us design better training processes, not just for better performance but for other properties we want the training process to induce. I think there's a lot of future work which could be done with this method, in making it more efficient and then applying it to larger models in domains other than vision. I'd also be interested in seeing if this can be used to understand which parts of the training set help and hurt training; for example seeing whether there's any correlation between the points of synchronised learning and the datapoints in the minibatch at that training iteration. Note: I'd recommend reading the paper (including the appendices) to see the graphs and visualisations the authors produced to demonstrate their arguments, as they're much easier to understand than a textual description.

**Rohin's opinion:** I also really enjoyed this paper, it has great empirical evidence about how neural networks work. I'd be inclined to analyze the results somewhat differently. In particular, suppose that when calculating LCA, we made the following changes:

1. We used the loss on the training batches instead of the full training set.
2. We didn't improve the approximation error (i.e. we just used the point estimate of the gradient calculated during training).
3. We trained using stochastic gradient descent (SGD) (as opposed to say Adam or Momentum-SGD).

Then all LCA values would be negative (explanation in [this comment](https://www.alignmentforum.org/posts/TmHRACaxXrLbXb5tS/rohinmshah-s-shortform?commentId=dWkj6mK2uZtnXJQiq)). So, when the paper shows experiments where LCA values are positive (i.e. the parameters / layers are anti-learning), we can attribute those effects to some combination of these three factors.

Take the observation that learning is very noisy. I would guess that this is primarily because of the first point: there are many many ways to improve the loss on a tiny little minibatch, but only a tiny fraction of those are capturing "real effects" that would improve the loss on the full large training dataset. Likely in the large majority of cases, the update doesn't capture a "real effect", and so it's a coin flip whether or not it will help with the loss on the full training dataset. A large probability of a coin flip + a small probability of a "real effect" gets you to an improvement slightly over half the time. This explanation applies across parameters, iterations, layers, etc.

Similarly, they find that learning is synchronized across layers. I think this is also primarily because of the first point. My guess is that there are some batches of data that are more "canonical" than others, that are easiest to learn from. In the case where we see synchronization for each class, this could be as simple as that particular training batch having more examples of that class than other training batches.

I’d be interested in seeing experiments in which we start with the version of LCA where everything is negative, and made only one of the changes. This would allow us to narrow down which particular change causes a given effect, kind of like an ablation study.

LCA: Loss Change Allocation for Neural Network Training

Janice Lan, Rosanne Liu, Hattie Zhou, Jason Yosinski

This paper introduces the _Loss Change Allocation_ (LCA) method. The method's purpose is to gain insight and understanding into the training process of deep neural networks. The method calculates an allocation of the change in overall loss (on the whole training set) between every parameter at each training iteration, which is iteratively refined until the approximation error is less than 1% overall. This loss change allocation can be either positive or negative; **if it's negative, then the parameter is said to have helped training at that iteration, and if it's positive then the parameter hurt training**. Given this measurement is per-parameter and per-iteration, it can be aggregated to per-layer LCA, or any other summation over parameters and training iterations.

The authors use the method to gain a number of insights into the training process of several small neural networks (trained on MNIST and CIFAR-10).

First, they validate that learning is very noisy, with **on average only half of the parameters helping at each iteration**. The distribution is heavier-tailed than a normal distribution, and is fairly symmetrical. However, parameters tend to alternate between helping and hurting, and each parameter only tends to help approximately 50% of the time.

Second, they look at the LCA aggregated per-layer, summed over the entire training process, and show that in the CIFAR ResNet model **the first and last layers hurt overall** (i.e. have positive LCA). In an attempt to remedy this and understand the causes, the authors try freezing these layers, or reducing their learning rate. The first layer can't be fixed (freezing makes it's LCA 0, but later layers' LCA is increased in turn so the overall final loss stays the same). However, for the last layer, **freezing or reducing the learning rate increases the overall performance of the network**, as the last layer's LCA is decreased more than all the other layer's LCAs are increased. They also hypothesize that by reducing the momentum for the last layer, they can give it fresher information and make it more likely to learn. They find that this does work, though in this setting previous layers’ LCA increases to compensate, leaving overall performance unchanged.

Finally, the authors show that **learning seems to be synchronised across layers**; layers get local LCA minima at the same training iterations, in a statistically significant way. They show this must be a combination of parameter motion and the gradient, as neither on their own explains this phenomenon.


While the authors present this as a method for understanding how neurons interact, it seems to me that the key insight is about looking at and explaining the behavior of the neural network _on data points in-distribution_. Most possible inputs are off-distribution, and there is not much to be gained by understanding what the network does on these points. Techniques that aim to gain a global understanding of the network are going to be "explaining" the behavior of the network on such points as well, and so will be presenting data that we won't be able to interpret. By looking specifically at activations corresponding to in-distribution images, we can ensure that the data we're visualizing is in-distribution and is expected to make sense to us.

I'm pretty excited that interpretability techniques have gotten good enough that they allow us to construct adversarial examples "by hand" -- that seems like a clear demonstration that we are learning something real about the network. It feels like the next step would be to use interpretability techniques to enable us to actually fix the network -- though admittedly this would require us to also develop methods that allow humans to "tweak" networks, which doesn't really fit within interpretability research as normally defined.

[OpenAI blog post](https://openai.com/blog/introducing-activation-atlases/) and [Google AI blog post](https://ai.googleblog.com/2019/03/exploring-neural-networks.html)

Exploring Neural Networks with Activation Atlases

Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah

Previous work by this group of people includes [The Building Blocks of Interpretability](https://distill.pub/2018/building-blocks/) and [Feature Visualization](https://distill.pub/2017/feature-visualization/), both of which apparently came out before this newsletter started so I don't have a summary to point to. Those were primarily about understanding what individual neurons in an image classifer were responding to, and the key idea was to "name" each neuron with the input that would maximally activate that neuron. This can give you a global view of what the network is doing.

However, such a global view makes it hard to understand the interaction between neurons. To understand these, we can look at a specific input image, and use techniques like attribution. Rather than attribute final classifications to the input, you could attribute classifications to neurons in the network, and then since individual neurons now had meanings (roughly: "fuzzy texture neuron", "tennis ball neuron", etc) you can gain insight to how the network is making decisions _for that specific input_.

However, ideally we would like to see how the network uses interactions between neurons to make decisions in general; not on a single image. This motivates activation atlases, which analyze the activations of a network on a _large dataset_ of inputs. In particular, for each of a million images, they randomly choose a non-border patch from the image, and compute the activation vector at a particular layer of the network at that patch. This gives a dataset of a million activation vectors. They use standard dimensionality reduction techniques to map each activation vector into an (x, y) point on the 2D plane. They divide the 2D plane into a reasonably sized grid (e.g. 50x50), and for each grid cell they compute the average of all the activation vectors in the cell, visualize that activation vector using feature visualization, and put the resulting image into the grid cell. This gives a 50x50 grid of the "concepts" that the particular neural network layer we are analyzing can reason about. They also use attribution to show, for each grid cell, which class that grid cell most supports.

The paper then goes into a lot of detail about what we can infer from the activation atlas. For example, we can see that paths in activation vector space can correspond to human-interpretable concepts like the number of objects in an image, or moving from water to beaches to rocky cliffs. If we look at activation atlases for different layers, we can see that the later layers seem to get much more specific and complex, and formed of combinations of previous features (e.g. combining sand and water features to get a single sandbar feature).

By looking at images for specific classes, we can use attribution to see which parts of an activation atlas are most relevant for the class. By comparing across classes, we can see how the network makes decisions. For example, for fireboats vs. streetcars, the network looks for windows for both, crane-like structures for both (though less than windows), and water for fireboats vs. buildings for streetcars. This sort of analysis can also help us find mistakes in reasoning -- e.g. looking at the difference between grey whales and great white sharks, we can see that the network looks for the teeth and mouth of a great white shark, including an activation that looks suspiciously like a baseball. In fact, if you take a grey whale and put a patch of a baseball in the top left corner, this becomes an adversarial example that fools the network into thinking the grey whale is a great white shark. They run a bunch of experiments with these human-found adversarial examples and find they are quite effective.


I take a pretty different view from this post -- I've highlighted it because I think this is an important disagreement that's relevant for alignment. In particular, it's not clear to me that "from-within" interpretability is doomed -- while I agree that humans basically only do "from-without" rationalizations, we also aren't able to inspect a human brain in the same way that we can inspect a neural net. For example, we can't see the output of each individual neuron, we can't tell what input would each neuron would respond maximally to, and we can't pose counterfactuals with slightly different inputs to see what changes. In fact, I think that "from-within" interpretability techniques, such as [Building Blocks of Interpretability]() have already seen successes in identifying biases that image classifiers suffer from, that we wouldn't have known about otherwise.

We could also consider whether post-hoc rationalization is sufficient for alignment. Consider a thought experiment where a superintelligent AI is about to take a treacherous turn, but there is an explainer AI system that post-hoc rationalizes the output of the AI that could warn us in advance. If the explainer AI only gets access to the output of the superintelligent AI, I'm very worried -- it seems way too easy to come up with some arbitrary rationalization for an action that makes it seem good, you'd have to be have a much more powerful explainer AI to have a hope. On the other hand, if the explainer AI gets access to all of the weights and activations that led to the output, it seems more likely that this could work -- as an analogy, I think a teenager could tell if I was going to betray them, if they could constantly eavesdrop on my thoughts.

Interpretability and Post-Rationalization

Neuroscience suggests that most explanations that we humans give for a decision are post-hoc rationalizations, and don't reflect the messy underlying true reasons for the decision. It turns out that decision making, perception, and all the other tasks we're hoping to outsource to neural nets are inherently complex and difficult, and are not amenable to easy explanation. We can aim for "from-without" explanations, which post-hoc rationalize the decisions a neural net makes, but "from-within" explanations, which aim for a mechanistic understanding, are intractable. We could try to design models that are more interpretable (in the "from-within" sense), but this would lead to worse performance on the actual task, which would hurt everyone, including the people calling for more accountability.


I'm guessing this is limited to tasks where your data points have a reasonable number of features (< 1000, I'd guess) and you are only analyzing a small set of test data points (around tens of thousands), due to computational constraints. That said, for those tasks, this seems incredibly useful to actually get a good model that you can debug and eventually deploy.

It's worth noting that this is an engineering achievement. Researchers are considering even stronger (but more computationally difficult) techniques, such as finding which part of the training set most influenced a particular decision, whereas the What-If tool doesn't talk about the training set and training process at all, instead only allowing you to ask queries about the final trained model.

The What-If Tool: Code-Free Probing of Machine Learning Models

When you train an ML model, it is often hard to understand what your model is doing and why. This post introduces the What-If tool, which allows you to ask counterfactual queries about the decision rule implemented by your final trained model, for classification and regression tasks. For example, you can take a particular data point, edit it slightly, and see how that changes the model prediction. Or you can graph the data points by L2 distance from a particular point. For classification tasks, you can find the "closest counterfactual", that is, the data point closest to the current point where the decision of the model is reversed. I played around with some of the demos, and apparently for a particular person and a particular model trained on census data, the probability that they had a salary of over $50k depended much more strongly on their marital status than their age, which was the opposite of my prediction. I figured this out by choosing a point, finding the closest counterfactual, and then making each of the changes in the delta individually and seeing which affected the model probability most.


While [OpenAI Five](https://blog.openai.com/openai-five/) suggests that the main thing we need to do is think of a reward function and an exploration strategy, this suggests that ML requires not just a good loss function, but lots of other things in order to work well. We have particular examples where changing things other than the loss function leads to different results. (This is probably also true for OpenAI Five, but the variations may not matter much, or OpenAI hasn't talked about the ML engineering behind the scenes -- I'm not sure.) These generally seem to be changing the inductive bias of the neural nets encoding the images. I think that if you expect to get very capable AI systems within the current paradigm, you will have to think about how inductive bias will affect what your AI system will do (and consequently its safety).

Also, the paper is very clear and approachable, and filled with great visualizations, as I've come to expect from Distill. I almost forgot to mention this, because I take it as a given for any Distill paper.

Differentiable Image Parameterizations

Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah

There are lots of techniques for generating images using neural nets. A common approach is to take a neural net trained to classify images, and then use gradient descent to optimize _the input image_ instead of the weights of the neural net. You might think that the only way to affect the generated input image would be to change the loss function on which you run gradient descent, but in reality the way in which you represent the image makes a huge difference. They describe why this might be the case, and go through several examples:

1. Suppose you want to see how two neurons interact. You could optimize an image to maximize the sum of the activations of the neurons. Even better, you could create an animation of how the image changes as you trade off how much you care about each neuron. Done naively, this doesn't look good, because there's a lot of randomness that changes between each image in the animation, which swamps out the differences we actually care about. To fix this, we can generate each frame in the animation as the sum of two images, one shared across all frames, and one that is frame-specific. Despite changing neither the loss function nor the space of input images, this is sufficient to remove the randomness between frames.

2. You've probably seen [style transfer](https://medium.com/data-science-group-iitr/artistic-style-transfer-with-convolutional-neural-network-7ce2476039fd) before, but did you know it only works with the VGG architecture? We can get it to work with other architectures by representing images in Fourier-space instead of pixel-space, again without any change in the loss function or expressible space of images.

3. If you generate the pixel-space representation of an image from a lower-dimensional representation using a Compositional Pattern Producing Network (CPPN), then gradient descent will optimize the lower-dimensional representation. It turns out that this produces images vaguely reminiscent of light-paintings. (I believe in this case, while the loss function doesn't change, the space of expressible images does change.)

4. Often when we see the feature visualization for a neuron, there are a lot of areas of the image that don't actually matter for the neuron's activation. So, we can add transparency, and add a term in the loss function that encourages transparency. We also have to change the representation of the image to include a transparency channel in addition to the normal RGB channels. Then, the generated image will be transparent wherever the pixels don't matter, but will still have the visualization wherever it does matter for activating the neuron.

5+6. We can even use a representation of 3D objects, and then write a (differentiable) algorithm that converts that into a 2D image that then goes through the standard image classifier neural net. This lets us optimize over the 3D object representation itself, letting us do both feature visualization and style transfer on 3D objects.


I liked this paper especially for its experimental design; it seems like it does a good job of keeping human priors from influencing the results. The results themselves are very much a first step, showing that you’ve gotten at least some understanding and interpretability, but ideally we’d do much much better on these axes. For example, if we “understand” the model, one would hope that we’d be able to get scores of 95+% on the simulation experiment (bullet point 1 above), rather than the current 72% / 57%. It might be interesting to have benchmarks that use these sorts of experiments as their evaluation method. Given that this method just uses feature visualization on the penultimate layer, it seems like there should be room for improvement by studying other layers as well.

_Editorial note:_ I summarized this work because I saw and liked the blog post about it. I don't generally follow the interpretability literature (it's huge), and so it's plausible that there are lots of more useful papers that I happen to not have seen. Most of the time, the highlighted papers can at least be understood as "this is what Rohin thinks is most useful for alignment researchers to read within this field"; that's not the case here.

Debuggable Deep Networks: Usage and Evaluation

Eric Wong*, Shibani Santurkar*, Aleksander Madry

One simple approach to make neural nets more understandable is to make just the final layer sparse. Neurons in the penultimate layer can be visualized using [existing techniques](https://distill.pub/2017/feature-visualization/), and the sparsity of the final layer means that it is relatively easy to understand how they are combined together to make predictions. For example, in ImageNet, the final logit for an individual class becomes a weighted combination of around 20 features, instead of 2048 as you would get with a dense model. The authors' core claim is that this makes the model more understandable and debuggable, at the cost of a small drop in performance (about 1-5 percentage points). They show this using several experiments, many with real humans:

1. The most basic test is simulation: can humans predict what the model would say (regardless of whether or not it is correct)? Unfortunately, if you show people a picture of an airplane, they are probably going to predict that the model says “airplane”, on priors. To avoid this sort of prior knowledge, they first sample a class like “airplane” that they _don’t_ reveal. Instead, they reveal feature visualizations of five randomly chosen features that the model uses to identify images of that class. They then choose three images and ask humans which of the three images will have the highest probability of being assigned to that class, according to the model. They find that when using a sparse final layer, humans have non-trivial performance (72% when the best image really is from the sampled class, and 57% when the best image is from some different class), whereas with a dense final layer they are only slightly better than random chance (44% and 31%, where random chance would be 33%).

2. They can study biases and spurious correlations in models. For example, Toxic-BERT identifies toxic sentences, but does so by searching for identity groups like “christianity”. Debiased-BERT was meant to solve this, but by looking at the feature visualizations (word clouds) below a sparse decision layer, they find that it simply learns a strong _negative_ weight for identity groups. Thus, they are able to fool the model into thinking a toxic comment is non-toxic simply by adding an identity group like “christianity” somewhere in the sentence. (This also applies to the version that uses a dense final layer.)

3. The identified biases or spurious correlations can then be used to generate counterfactuals: for example, in a sentiment analysis system, they can visualize word clouds that represent positive and negative influences on the final sentiment reported by the model. Then, by simply exchanging a positive word for a negative word (or vice versa), they can flip the label that the model assigns to the sentence. (Usually this is correct behavior – if you change “a _marvel_ like you’ve never seen” to “a _failure_ like you’ve never seen”, the sentiment really is different. The point is that the sparse model allows you to create these examples automatically.)

4. In cases where the model makes a mistake, can humans identify why the model made a mistake? The authors note that over 30% of misclassifications can be explained by a single problematic feature, i.e. if you intervene to set that feature to zero, then the model no longer makes a mistake. So one way to check human understanding is to see whether they can reproduce this misclassification. Specifically, we take some image whose true label is y\* but which the model incorrectly labels as y’. We then take the highest-activating feature in support of y\* and the corresponding feature for y’, and ask humans which of the two features is more present in the image. They find that annotators prefer the feature for y’ 60% of the time – more than random chance (50%). Since the annotators don’t know which feature corresponds to the ground truth and which corresponds to the incorrect model prediction, they probably were not using prior knowledge in answering this question. Thus, doing better than random suggests that even according to humans the feature that the model picked up on really was present in the image.


This post is great. Especially to the extent that you like summaries of papers (and according to the survey I recently ran, you probably do like summaries), I would recommend reading through this post. You could also read through the highlights from each section, bringing it down to 13 summaries instead of 70.

Opinions on Interpretable Machine Learning and 70 Summaries of Recent Papers

This is basically 3 months worth of Alignment Newsletters focused solely on interpretability, wrapped up into a single post. The authors provide summaries of 70 (!) papers on the topic, and include links to another 90. I’ll focus on their opinions about the field in this summary.

The theory and conceptual clarity of the field of interpretability has improved dramatically since its inception. There are several new or clearer concepts, such as simulatability, plausibility, (aligned) faithfulness, and (warranted) trust. This seems to have had a decent amount of influence over the more typical “methods” papers.

There have been lots of proposals for how to evaluate interpretability methods, leading to the [problem of too many standards](https://xkcd.com/927/). The authors speculate that this is because both “methods” and “evaluation” papers don’t have sufficient clarity on what research questions they are trying to answer. Even after choosing an evaluation methodology, it is often unclear which other techniques you should be comparing your new method to.

For specific methods for achieving interpretability, at a high level, there has been clear progress. There are cases where we can:
1. identify concepts that certain neurons represent,
2. find feature subsets that account for most of a model's output,
3. find changes to data points that yield requested model predictions,
4. find training data that influences individual test time predictions,
5. generate natural language explanations that are somewhat informative of model reasoning, and
6. create somewhat competitive models that are inherently more interpretable.

There does seem to be a problem of disconnected research and reinventing the wheel. In particular, work at CV conferences, work at NLP conferences, and work at NeurIPS / ICML / ICLR form three clusters that for the most part do not cite each other.


I really like the Circuits line of research; it has made pretty incredible progress in the last few years. I continue to be amazed at the power of “naming” neurons with their feature visualizations; it feels a bit shocking how helpful this is. (Though like most good ideas, in hindsight it’s blatantly obvious.) The fact that it is possible to understand 50,000 parameters well enough to reimplement them from scratch seems like a good validation of this sort of methodology. Note though that we are still far from the billions of parameters in large language models.

One thing that consistently impresses me is how incredibly _thorough_ the authors are. I often find myself pretty convinced by the first three distinct sources of evidence they show me, and then they just continue on and show me another three distinct sources of evidence. This sort of care and caution seems particularly appropriate when attempting to make claims about what neural networks are and aren’t doing, given how hard it has historically seemed to be to make reasonable claims of that sort.

I’m pretty curious what equivariance in natural language looks like. Is it that equivariance is common in nearly all neural nets trained on “realistic” tasks, or is equivariance a special feature of vision?

Circuits Thread

We’ve <@previously@>(@Thread: Circuits@) summarized the three main claims of the Circuits thread. Since then, several additional articles in the series have been published, so now seems like a good time to take another look at the work in this space.

The three main claims of the Circuits agenda, introduced in [Zoom In](https://distill.pub/2020/circuits/zoom-in/), are:
1.  Neural network *features* - the activation values of hidden layers - are understandable.
2.  *Circuits* - the weights connecting these features - are also understandable.
3.  *Universality* - when training different models on different tasks, you will get analogous features.

To support the first claim, there are seven different techniques that we can use to understand neural network features that all seem to work quite well in practice:

1. **Feature visualization** produces an input that maximally activates a particular neuron, thus showing what that neuron is “looking for”.
2. **Dataset examples** are the inputs in the training dataset that maximally activate a particular neuron, which provide additional evidence about what the neuron is “looking for”.
3. **Synthetic examples** created independently of the model or training set can be used to check a particular hypothesis for what the neuron activates on.
4. **Tuning** involves perturbing an image and seeing how the neuron activations change. For example, we might rotate images and see how that impacts a curve detector.
5. By looking at the circuit used to create a neuron, we can read off the algorithm that **implements the feature** and make sure that it makes intuitive sense.
6. We can look to see **how the feature is used** in future circuits.
7. We can **handwrite circuits** that implement the feature after we understand how it works, in order to check our understanding and make sure it performs as well as the learned circuit.

Note that since techniques 5-7 are based on understanding circuits, their empirical success also supports claim 2.

The third claim, that the features learned are universal, is the most speculative. There isn’t direct support for it, but anecdotally it does seem to be the case that many vision networks do in fact learn the same features, especially in the earlier layers that learn lower-level features.

In order to analyze circuits, we need to develop interpretability tools that work with them. Just as activations were the primary object of interest in understanding the behavior of a model on a given input, weights are the primary object of interest in circuits. [Visualizing Weights](https://distill.pub/2020/circuits/visualizing-weights/) explains how we can adapt all of the techniques from [Building Blocks](https://distill.pub/2018/building-blocks/) to this setting. Just as in Building Blocks, a key idea is to “name” each neuron with its feature visualization: this makes each neuron meaningful to humans, in the same way that informative variable names make the variable more meaningful to a software engineer.

Once we have “named” our neurons, our core operation is to visualize the matrix of weights connecting one neuron in layer L to another in layer L+1. By looking at this visualization, we can see the “algorithm” (or at least part of the algorithm) for producing the layer L+1 neuron given the layer L neuron. The other visualizations are variations on this core operation: for example, we might instead visualize weights for multiple input neurons at once, or for a group of neurons together, or for neurons that are separated by more than one layer.

We then apply our machinery to _curve detectors_, a collection of 10 neurons that detect curves. We first use techniques 1-4 (which don’t rely on circuits) to understand what the neurons are doing in [Curve Detectors](https://distill.pub/2020/circuits/curve-detectors/). This is probably my favorite post of the entire thread, because it spends ~7000 words and many, many visualizations digging into _just 10 neurons_, and it is still consistently interesting. It gives you a great sense of just how complex the behavior learned by neural networks can be. Unfortunately, it’s hard to summarize (both because there’s a lot of content and because the visualizations are so core to it); I recommend reading it directly.

[Curve Circuits](https://distill.pub/2020/circuits/curve-circuits/) delves into the analysis of how the curve detectors actually work. The key highlight of this post is that one author set values for weights without looking at the original network, and recreated a working curve detection algorithm (albeit not one that was robust to colors and textures, as in the original neural network). This is a powerful validation that at least that author really does “understand” the curve detectors. It’s not a small network either -- it takes around 50,000 parameters. Nonetheless, it can be described relatively concisely:

_Gabor filters turn into proto-lines which build lines and early curves. Finally, lines and early curves are composed into curves. In each case, each shape family (eg conv2d2 line) has positive weight across the tangent of the shape family it builds (eg. 3a early curve). Each shape family implements the rotational equivariance motif, containing multiple rotated copies of approximately the same neuron._

(Yes, that’s a lot of jargon; it isn’t really meant to be understood just from that paragraph. The post goes into much more detail.)

So why is it amenable to such a simple explanation? [Equivariance](https://distill.pub/2020/circuits/equivariance/) helps a lot in reducing the amount we need to explain. Equivariance occurs when there is a kind of symmetry across the learned neurons, such that there’s a single motif copied across several neurons, except transformed each time. A typical example of equivariance is _rotational equivariance_, where the feature is simply rotated some amount. Curve detectors show rotational equivariance: we have 10 different detectors that are all implemented in roughly the same way, except that the orientation is changed slightly. Many of the underlying neurons also show this sort of equivariance. In this case, we only need to understand one of the circuits and the others follow naturally. This reduces the amount to understand by about 50x.

This isn’t the only type of equivariance: you can also have e.g. neuron 1 that detects A on the left and B on the right, and neuron 2 that detects A on the right and B on the left, which is a reflection equivariance. If you trained a multilayer perceptron (MLP), that would probably have _translational equivariance_, where you’d see multiple neurons computing the same function but for different spatial locations of the image. (CNNs build translational equivariance into the architecture because it is so common.)

The authors go into detail by showing many circuits that involve some sort of equivariance. Note that equivariance is a property of _features_ (neurons), rather than the circuits that connect them. Nonetheless, this is reflected in the structure of the weights that make up the circuit. The post gives several examples of such circuits. These can be of three types: invariant-equivariant circuits (building a family of equivariant features starting from invariant features), equivariant-invariant circuits, and equivariant-equivariant circuits.

[High-Low Frequency Detectors](https://distill.pub/2020/circuits/frequency-edges/) applies all of these tools to understand a family of features that detect areas with high frequencies on one side, and low frequencies on the other. It’s mostly interesting as another validation of the tools, especially because these detectors were found through interpretability techniques and weren’t hypothesized before (whereas we could have and maybe did predict a priori that an image classifier would learn to detect curves).


I think work which targets specific neurons and what they're doing is interesting as it can give us a very low-level understanding of the model, which I feel is necessary to achieve the level of understanding required by alignment solutions which use interpretability (i.e. those in <@An overview of 11 proposals for building safe advanced AI@>). The main limitation of this approach is that it currently requires a large amount of dense human labelling of the datasets, and if a concept isn't in the labels of the dataset, then the method won't be able to explain a neuron using this concept. Also, the fact that their interpretability method is able to give insights (in the form of creating copy-paste examples) is a useful sign it's actually doing something meaningful, which I think some other interpretability methods lack.

Compositional Explanations of Neurons

Network dissection is an interpretability technique introduced in 2017, which uses a dataset of images with dense (i.e. pixel) labels of concepts, objects and textures. The method measures the areas of high activation of specific channels in a convolutional neural network, then compares these areas with the labelled areas in the dataset. If there's a high similarity for a particular channel (measured by the intersection divided by the union of the two areas), then we can say this channel is recognising or responding to this human-interpretable concept.

This paper introduces an extension of this idea, where instead of just using the basic concepts (and matching areas in the dataset), they search through logical combinations of concepts (respectively areas) to try and find a compositional concept which matches the channel's activations. For example, a channel might respond to (water OR river) AND NOT blue. This is still a concept humans can understand (bodies of water which aren't blue), but enables us to explain the behaviour of a larger number of neurons than in the original network dissection method. Their work also extends the method to natural language inference (NLI), and they interpret neurons in the penultimate layer of a BiLSTM-based network trained to know whether a sentence entails, contradicts, or is neutral with respect to another. Here they create their own features based on words, lexical similarity between the two sentences, and part-of-speech tags.

Using their method, they find that channels in image classifiers do learn compositional concepts that seem useful. Some of these concepts are semantically coherent (i.e. the example above), and some seem to have multiple unrelated concepts entangled together (i.e. operating room OR castle OR bathroom). In the NLI network, they see that many neurons seem to learn shallow heuristics based on bias in the dataset - i.e. the appearance of single words (like nobody) which are highly informative about the classification.

Finally, they use their method to create copy-paste adversarial examples (like in Activation Atlas (AN #49)). In the Places365 dataset (where the goal is to classify places), they can crudely add images which appear in compositional concepts aligned with highly contributing neurons, to make that neuron fire more, and hence change the classification. Some of these examples generalise across classifier architectures, implying a bias present in the dataset.


Even from only the first three posts, I am largely convinced that most of neural networks can be understood in this way. The main open question to me is the scalability of this approach. As neural networks get more powerful, do they become more interpretable or less interpretable? Or does it follow a more complex pattern like the one suggested <@here@>(@Chris Olah’s views on AGI safety@). I’d love to see some quantitative metric of how interpretable a model is and see how that has changed for the vision state of the art each year. Another related topic I am very interested in is how these visualizations change over training. Do early layers develop first? Does finetuning affect some layers more than others? What happens to these features if the model is overfit?

The other thing I found very exciting about all of these posts is the visualization tools that were used (omitting these is a major shortcoming of this summary). For example, you can click on any of the neurons mentioned in the paper and it opens up a [Microscope](https://microscope.openai.com/models) page that lets you see all the information on that feature and its circuits. I hope that as we get better and more generic tools for analyzing neural networks in this way, this could become very useful for debugging and improving neural network architectures.

Thread: Circuits

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter

The (currently incomplete) Circuits thread of articles builds a case around 3 main claims:

1.  Neural network *features* - the activation values of hidden layers - are understandable.
2.  *Circuits* - the weights connecting these features - are also understandable.
3.  *Universality* - when training different models on different tasks, you will get analogous features.

[Zoom In](https://distill.pub/2020/circuits/zoom-in/) provides an overview of the argument. The next two articles go into detail on particular sets of layers or neurons.

**Claim 1:** Neural Network Features - the activation values of hidden layers - are understandable.

They make seven arguments for this claim in [Zoom In](https://distill.pub/2020/circuits/zoom-in/) which are expanded upon in subsequent articles.

**1.  Feature Visualization**: By optimizing the input to maximize the activation of a particular neuron, they can obtain an image of what that neuron reacts most strongly to. They create and analyze these for all 1056 neurons in the first five layers of the *InceptionV1* image classification model. While some of them were difficult to understand, they were able to classify and understand the purpose of most of the neurons. A simple example is that curve detecting neurons produce feature visualizations of curves of a particular orientation. A more complex example is neurons detecting boundaries between high and low frequency, which often are helpful for separating foreground and background.

**2.  Dataset examples**: They also look at the examples in the dataset that maximize a particular neuron. These align with the feature visualizations. Neurons with a particular curve in the feature visualization also fire strongest on dataset examples exhibiting that curve.

**3.  Synthetic Examples**: They also create synthetic examples and find that neurons fire on the expected synthetically generated examples. For example, they generate synthetic curves with a wide range of orientations and curvatures. Curve detectors respond most strongly to a particular orientation and curvature that matches the feature visualizations and highest activation dataset examples. [Curve Detectors](https://distill.pub/2020/circuits/curve-detectors/#synthetic-curves) includes many more experiments and visualizations of curve detectors on the full distribution of curvature and orientation.

**4.  Joint Tuning**: In the case of [curve detectors](https://distill.pub/2020/circuits/curve-detectors/#joint-tuning-curves), they rotate the maximal activation dataset examples and find that as the curves change in orientation, the corresponding curve detector neurons increase and decrease activations in the expected pattern.

**5.  Feature Implementation**: By looking at the circuit used to create a neuron, they can read off the algorithm for producing that feature. For example, curve detectors are made up of line detectors and earlier curve detectors being combined in a way that indicates it would only activate on curves of a particular orientation and curvature.

**6.  Feature Use**: In addition to looking at the inputs to the neuron, they also look at the outputs to see how the feature is used. For example, curves are frequently used in neurons that recognize circles and spirals.

**7.  Handwritten Circuits**: After understanding existing curve detectors, they can implement their own curve detectors by hand-coding all the weights, and those reliably detect curves.

**Claim 2**: Circuits - the weights connecting the features - are also understandable

They provide a number of examples of neurons, both at deep and shallow layers of the network, that are composed of earlier neurons via clear algorithms. As mentioned above, curve detectors are excited by earlier curve detectors in similar orientations and inhibited by ones of opposing orientations.

A large part of ImageNet is focused on distinguishing a hundred species of dogs. A pose-invariant dog head and neck detector can be shown to be composed of two earlier detectors for dogs facing left and right. These in turn are constructed from earlier detectors of fur in a particular orientation.

They also describe circuits for dog head, car, [boundary](https://distill.pub/2020/circuits/early-vision/#mixed3b), [fur](https://distill.pub/2020/circuits/early-vision/#mixed3b), [circle](https://distill.pub/2020/circuits/early-vision/#mixed3a), and [triangle](https://distill.pub/2020/circuits/early-vision/#mixed3a) detectors.

**Claim 3:** Universality: when training different models on different tasks, you will get analogous features.

This is the most speculative claim and most of the articles so far have not addressed it directly. However, the early layers of vision (edges, etc), are believed to be common to many computer vision networks. They describe in detail the first five layers of InceptionV1 and categorize all of the neurons.

**Layer 1** is the simplest: 85% of the neurons either detect simple edges or contrasts in colors.
**Layer 2** starts to be more varied and detects edges and color contrasts with some invariance to orientation, along with low frequency patterns and multiple colors.
In **Layer 3,** simple shapes and textures begin to emerge, such as lines, curves, and hatches, along with color contrasts that are more invariant to position and orientation than those in the earlier layers.
**Layer 4** has a much more diverse set of features. 25% are textures, but there are also detectors for curves, high-low frequency transitions, brightness gradients, black and white, fur, and eyes.
**Layer 5** continues the trend of having features with more variety and complexity. One example is boundary detectors, which combine a number of low-level features into something that can detect boundaries between objects.

They also highlight a few phenomena that are not yet fully understood:

*Polysemantic neurons* are neurons that respond to multiple unrelated inputs, such as parts of cars and parts of cats. What is particularly interesting is that these are often constructed from earlier features that are then spread out across multiple neurons in a later layer.

The *combing phenomenon* is that curve and line detectors on multiple models and datasets tend to be excited by small lines that are perpendicular to the curve. Potential hypotheses are that many curves in the data have them (e.g. spokes on a wheel), that it is helpful for fur detection, that it provides higher contrast between the orientation of the curve and the background, or that it is just a side effect rather than an intrinsically useful feature.


It's nice to see an interpretability method with demonstrable and measurable use cases. Many methods aim at improving insight, but often don't demonstrate this aim; I think this paper does this well in showing how its method can be used for model surgery. I think methods that allow us to investigate and understand individual neurons and their contributions are useful in building up a fine grained picture of how neural networks work. This links to previous work such as [Network Dissection](http://netdissect.csail.mit.edu/) as well as the recent [Circuits Thread](https://distill.pub/2020/circuits/) on Distill, and I'd love to see how these methods interact. They all give different kinds of understanding, and I think it would be interesting to see if given the results of the circuits tools we were able to predict which neurons where most responsible for different metrics (Neuron Shapley) or aligned to which relevant features (Network Dissection).

Neuron Shapley: Discovering the Responsible Neurons

This paper presents a novel method, Neuron Shapley, that uses the [Shapley value framework](https://en.wikipedia.org/wiki/Shapley_value) to measure the importance of different neurons in determining an arbitrary metric of the neural net output. (Shapley values have been applied to machine learning before to [measure the importance of features to a model's output](https://christophm.github.io/interpretable-ml-book/shapley.html), but here the authors use them to calculate neuron importance.) Due to several novel approaches and optimisations in calculating these Shapley values, **the top k most responsible neurons (k ~ 30) can be feasibly found for large networks such as Inception-v3**.

The authors demonstrate that finding these neurons enables the performance of model surgery. Removing the top 30 neurons that contribute to accuracy completely destroys the accuracy, whereas in expectation removing 30 neurons at random from the network barely moves the accuracy at all. Since the method can be applied to an arbitrary metric, this kind of surgery can be performed for other metrics we care about. For example, removing the neurons which are most responsible for vulnerability to adversarial attacks makes the network more robust, and removing the neurons most responsible for the class-accuracy imbalance (a fairness metric) makes the classes much more even, while only reducing the overall accuracy a small amount.


This seems to me like a very plausible story for how AI systems get incorporated into human decision-making in the near-term future. I do worry that further down the line, AI systems where AIs are merely advising will get outcompeted by AI systems doing the entire decision-making process. From an interpretability perspective, it also seems to me like having 'advice' that represents complicated multivariate data still hides a lot of reasoning that could be important if we were worried about misaligned AI. I like that the paper emphasizes having humans-in-the-loop during training and presents an effective mechanism for doing gradient descent with human choices.

**Rohin's opinion:** One interesting thing about this paper is its similarity to [Deep RL from Human Preferences](https://deepmind.com/blog/article/learning-through-human-feedback): it also trains a human model, that is improved over time by collecting more data from real humans. The difference is that DRLHP produces a model of the human reward function, whereas the model in this paper predicts human _actions_.

Learning Representations by Humans, for Humans

Sophie Hilgard*, Nir Rosenfeld*, Mahzarin R. Banaji, Jack Cao, David C. Parkes

Historically, interpretability approaches have involved machines acting as **experts**, making decisions and generating explanations for their decisions. This paper takes a slightly different approach, instead using machines as **advisers** who are trying to give the best possible advice to humans, the final decision makers. Models are given input data and trained to generate visual representations based on the data that cause humans to take the best possible actions. In the main experiment in this paper, humans are tasked with deciding whether to approve or deny loans based on details of a loan application. Advising networks generate realistic-looking faces whose expressions represent multivariate information that's important for the loan decision. Humans do better when provided the facial expression 'advice', and furthermore can justify their decisions with analogical reasoning based on the faces, e.g. "x will likely be repaid because x is similar to x', and x' was repaid".


SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Maithra Raghu, Justin Gilmer, Jason Yosinski, Jascha Sohl-Dickstein


Neural Networks seem to follow a puzzlingly simple strategy to classify images

This is a blog post explaining the paper [Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet](https://openreview.net/pdf?id=SkfMWhAqYQ), which was summarized in [AN #33](https://mailchi.mp/b6dc636f6a1b/alignment-newsletter-33).


Deeper Interpretability of Deep Networks

Tian Xu, Jiayu Zhan, Oliver G.B. Garrod, Philip H.S. Torr, Song-Chun Zhu, Robin A.A. Ince, Philippe G. Schyns


GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, Antonio Torralba


What the hell is going on inside neural networks

This podcast covers a significant chunk of work in understanding neural networks, including <@circuits@>(@Circuits Thread@) and <@multimodal neurons@>(@Multimodal Neurons in Artificial Neural Networks@), as well as high-level thoughts such as [advantages of neural net interpretability over neuroscience](http://colah.github.io/notes/interp-v-neuro/) and <@why larger models may be more interpretable@>(@Chris Olah’s views on AGI safety@). Some interesting points I haven’t made in this newsletter before:

1. Interpretability as a field is fractured into several different mini-paradigms. The author’s paradigm might be described as “mechanistic interpretability”, where you try to “fully understand” the neural network from the ground up. An ML-based paradigm is interested in defining good “interpretability metrics” that can then be optimized. An HCI-based paradigm is interested in developing techniques that show good results based on user evaluations (e.g. people can better predict network outputs).

2. Scaling up mechanistic interpretability does seem possible, because (a) as models get larger their features plausibly get crisper and easier to understand, and (b) there are motifs (such as equivariance in curve circuits) that allow you to reduce the number of neurons you have to understand by over an order of magnitude. However, neurons can be _polysemantic_, where they encode multiple features at once; this could pose a significant challenge for mechanistic interpretability. (While current features encoded in polysemantic neurons will probably become crisper as models scale up, we might expect that the scaled up models will have new polysemantic neurons that encode multiple more abstract features.)

3. One aesthetically pleasing aspect of the mechanistic interpretability approach is that, in the world where we succeed, humans could plausibly “keep up” with the neural nets and understand these advanced concepts that the networks have, rather than living happy lives but being unable to comprehend what is going on in the world around them. See also [Using Artificial Intelligence to Augment Human Intelligence](https://distill.pub/2017/aia/).

You may also want to check out [this followup podcast](https://80000hours.org/podcast/episodes/chris-olah-unconventional-career-path/) in which Chris talks about his unconventional career path.


The rigour of the study in terms of it's relatively large sample size of participants, pre-registered hypotheses, and follow up experiments is very positive. It's a good example for other researchers wanting to make and test empirical claims about what kind of interpretability can be useful for different goals. The results suggest that exposing the internals of models isn't always useful for increasing the interpretability of the model. This is useful information for the designers of systems who aim to provide interpretation of models to end-users.

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, Hanna Wallach

This paper performs a rigorous, pre-registered experiment investigating to what degree transparent models are more useful for participants. They investigate how well participants can estimate what the _model predicts_, as well as how well the participant can make predictions given access to the model information. The task they consider is prediction of house prices based on 8 features (such as number of bathrooms and square footage). They manipulate two independent variables. First, CLEAR is a presentation of the model where the coefficients for each feature are visible, whereas BB (black box) is the opposite. Second, **-8** is a setting where all 8 features are used and visible, whereas in **-2** only the 2 most important features (number of bathrooms and square footage) are visible. (The model predictions remain the same whether 2 or 8 features are revealed to the human.) This gives 4 conditions: CLEAR-2, CLEAR-8, BB-2, BB-8.

They find a significant difference in ability to predict model output in the CLEAR-2 setting vs all other settings, supporting their pre-registered hypothesis that showing the few most important features of a transparent model is the easiest for participants to simulate. However, counter to another pre-registered prediction, they find no significant difference in deviation from model prediction based on transparency or number of features. Finally, they found that participants shown the clear model were less likely to correct the model's inaccurate predictions on "out of distribution" examples than participants with the black box model.


I'm in favour of work on interpretability in reinforcement learning, and it's good to see the team at OpenAI working on it. I think this is a(nother) demonstration from them that interpretability research is often mostly about engineering and user interface design, followed by extended use of the produced interface; none of the methods proposed here are especially novel, but the combined interface and subsequent insights gained from its use are.

I also think the diversity hypothesis seems (in the abstract) plausible, and seems to have some supporting evidence from supervised learning (in particular computer vision): harder tasks tend to lead to better representations, and adversarially robust networks produce more interpretable representations, while also generalising better. One problem with verifying this hypothesis in other settings (or even more formally in this setting) is having to measure what it means for a representation to be "more interpretable". In general, I think this is related to the phenomena of [shortcut learning in deep learning](https://arxiv.org/abs/2004.07780): shortcuts in tasks will tend to mean that the network won't have learned a robust or interpretable feature set, whereas if there are no shortcuts and the network needs to do the task "as a human would", then it's more likely that the representations will be more robust.

Understanding RL Vision

Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah

This work presents an interface for interpreting the vision of a reinforcement learning agent trained with PPO on the CoinRun game. This game is procedurally generated, which means the levels are different in every episode of playing. The interface primarily uses attribution from a hidden layer to the output of the value function. This interface is used in several ways.

First, they use the interface to dissect failed trajectories of the policy (it fails in 1 out of 200 levels). They're able to understand why the failures occurred using their interface: for example, in one case the view of the agent at the top of its jump means it can't see any platforms below it, so doesn't move to the right fast enough to reach the platform it was jumping for, leading it to miss the platform and fail the level. Second, they use the interface to discover "hallucinations", where the value function mistakes one element of the environment for another, causing its value to drop or rise significantly. Often these hallucinations only last a single time-step, so they don't affect performance.

Finally, they use the attributions specifically to hand-edit the weights of the model to make it "blind" to buzzsaws (one of the hazards) by zeroing the feature which recognises them. After doing this, they show that the edited agent fails a lot more from buzzsaw failures but no more from other types of failures, which gives a quantitative justification for their interpretation of the feature as buzzsaw-recognising.

From using this interface, they propose the **diversity hypothesis:** _Interpretable features tend to arise (at a given level of abstraction) if and only if the training distribution is diverse enough (at that level of abstraction)._ This is based on the fact that interpretable features arise more when the agent is trained on a wider variety of levels. There also seems to be a qualitative link to generalisation - a wider distribution of training levels leads to better interpretability (measured qualitatively) and better generalisation (measured quantitatively).


The framework presented in this paper is simple, but I like the idea of combining it with the fully adjustable ATARI simulator to enable meaningful interventions, which enable us to falsify claims made using saliency maps. This is one way to validate whether the methods we're using are producing good insights.

I think this paper points more at the fact that our interpretation of saliency maps is incorrect, due to us imposing anthropomorphic biases on the agent's reasoning, and trying to infer general behaviour from specific explanations. I think many of the claims they reference could be reworded to be much more specific, and then would probably hold true (i.e. instead of "The agent understands tunnels and attends to and builds them" say "The agent knows destroying bricks consistently on the right hand side of the screen leads to higher reward and so attends to that location when it's able to bounce the ball there".)

Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning

Akanksha Atrey, Kaleigh Clary, David Jensen

This paper presents an analysis of the use of saliency maps in deep vision-based reinforcement learning on ATARI. They consider several types of saliency methods, all of which produce heatmaps on the input image. They show that all (46 claims across 11 papers) uses of saliency maps in deep RL literature interpret them as representing the agent's "focus", 87% use the saliency map to generate a claim about the agent's behaviour or reasoning, but only 7% validate their claims with additional or more direct evidence.

They go on to present a framework to turn subjective and under-defined claims about agent behaviour generated with saliency maps into falsifiable claims. This framework effectively makes the claim more specific and targeted at specific semantic concepts in the game's state space. Using a fully parameterized version of the ATARI environment, they can alter the game's state in ways which preserve meaning (i.e. the new state is still a valid game state). This allows them to perform interventions in a rigorous way, and falsify the claims made in their framework.

Using their framework, they perform 3 experimental case studies on popular claims about agent behaviour backed up by saliency maps, and show that all of them are false (or at least stated more generally than they should be). For example, in the game Breakout, agents tend to build tunnels through the bricks to get a high score. Saliency maps show that the agent attends to these tunnels in natural games. However, shifting the position of the tunnel and/or the agent's paddle and/or the ball all remove the saliency on the tunnel's location. Even flipping the whole screen vertically (which still results in a valid game state) removes the saliency on the tunnel's location. This shows that the agent doesn’t understand the concept of tunnels generally or robustly, which is often what is claimed.