Branch Specialization

Chelsea Voss
Gabriel Goh
Nick Cammarata
Michael Petrov
Ludwig Schubert
Chris Olah

Abstract: This article is part of the Circuits thread,  an experimental format collecting invited short articles and critical  commentary delving into the inner workings of neural networks.

![](Branch%20Specialization_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

[Visualizing Weights](https://distill.pub/2020/circuits/visualizing-weights/)

[Weight Banding](https://distill.pub/2020/circuits/weight-banding/)

Introduction

------------

 If we think of interpretability as a kind of "anatomy of neural

networks," most of the circuits thread has involved studying tiny little

 veins – looking at the small-scale, at individual neurons and how they

connect. However, there are many natural questions that the small-scale

approach doesn't address.

 In contrast, the most prominent abstractions in biological anatomy

 involve larger-scale structures: individual organs like the heart, or

entire organ systems like the respiratory system. And so we wonder: is

there a "respiratory system" or "heart" or "brain region" of an

artificial neural network? Do neural networks have any emergent

structures that we could study that are larger-scale than circuits?

 have separate dedicated articles.) Branch specialization occurs when

neural network layers are split up into branches. The neurons and

circuits tend to self-organize, clumping related functions into each

branch and forming larger functional units – a kind of "neural network

brain region." We find evidence that these structures implicitly exist

in neural networks without branches, and that branches are simply

reifying structures that otherwise exist.

 AlexNet is famous as a jump in computer vision, arguably starting the

deep learning revolution, but buried in the paper is a fascinating,

rarely-discussed detail.

 The first two layers of AlexNet are split into two branches which

can't communicate until they rejoin after the second layer. This

structure was used to maximize the efficiency of training the model on

two GPUs, but the authors noticed something very curious happened as a

result. The neurons in the first layer organized themselves into two

groups: black-and-white Gabor filters formed on one branch and

low-frequency color detectors formed on the other branch.

![](Branch%20Specialization_files/Figure_1.png)

 observed the phenomenon we call branch specialization in the first

layer of AlexNet by visualizing their weights to RGB channels; here, we

 Although the first layer of AlexNet is the only example of branch

specialization we're aware of being discussed in the literature, it

seems to be a common phenomenon. We find that branch specialization

happens in later hidden layers, not just the first layer. It occurs in

both low-level and high-level features. It occurs in a wide range of

models, including places you might not expect it – for example, residual

 blocks in resnets can functionally be branches and specialize. Finally,

 branch specialization appears to surface as a structural phenomenon in

plain convolutional nets, even without any particular structure causing

it.

 Is there a large-scale structure to how neural networks operate?

How are features and circuits organized within the model? Does network

architecture influence the features and circuits that form? Branch

specialization hints at an exciting story related to all of these

questions.

What is a branch?

-----------------

InceptionV1

 has nine sets of four-way branches called "Inception blocks."

 has several two-way branches.

AlexNet

Residual Networks

 In more recent years, these have become less common, but residual

networks – which can be seen as implicitly having branches in their

residual blocks – have become very common. We also sometimes see

branched architectures develop automatically in neural architecture

search, an approach where the network architecture is learned.

 The implicit branching of residual networks has some important

nuances. At first glance, every layer is a two-way branch. But because

the branches are combined together by addition, we can actually rewrite

the model to reveal that the residual blocks can be understood as

branches in parallel:

+

+

+

+

+

[3](#figure-3). Residual blocks as branches in parallel.

 One hypothesis for why is that, in these models, the exact depth of a

layer doesn't matter and the branching aspect becomes more important

than the sequential aspect.

 One of the conceptual weaknesses of normal branching models is

that although branches can save parameters, it still requires a lot of

parameters to mix values between branches. However, if you buy the

branch interpretation of residual networks, you can see them as a

strategy to sidestep this: residual networks intermix branches (e.g.

block sparse weights) with low-rank connections (projecting all the

blocks into the same sum and then back up). This seems like a really

elegant way to handle branching. More practically, it suggests that

analysis of residual networks might be well-served by paying close

attention to the units in the blocks, and that we might expect the

residual stream to be unusually polysemantic.

Why does branch specialization occur?

-------------------------------------

 Branch specialization is defined by features organizing between

branches. In a normal layer, features are organized randomly: a given

feature is just as likely to be any neuron in a layer. But in a branched

 layer, we often see features of a given type cluster to one branch. The

 branch has specialized on that type of feature.

A1

B1

C1

D1

A2

B2

D2

D2

 Another way to think about this is that if you need to cut a

neural network into pieces that have limited ability to communicate with

 each other, it makes sense to organize similar features close together,

 because they probably need to share more information.

Branch specialization beyond the first layer

--------------------------------------------

 So far, the only concrete example we've shown of branch

specialization is the first and second layer of AlexNet. What about

later layers? AlexNet also splits its later layers into branches, after

all. This seems to be unexplored, since studying features after the

 Unfortunately, branch specialization in the later layers of

AlexNet is also very subtle. Instead of one overall split, it's more

like there's dozens of small clusters of neurons, each cluster being

assigned to a branch. It's hard to be confident that one isn't just

seeing patterns in noise.

 But other models have very clear branch specialization in later

layers. This tends to happen when a branch constitutes only a very small

 fraction of a layer, either because there are many branches or because

one is much smaller than others. In these cases, the branch can

specialize on a very small subset of the features that exist in a layer

and reveal a clear pattern.

 For example, most of InceptionV1′s layers have a branched

structure. The branches have varying numbers of units, and varying

convolution sizes. The 5x5 branch is the smallest branch, and also has

the largest convolution size. It's often very specialized:

mixed3a\_5x5:

mixed3b\_5x5:

mixed4a\_5x5:

3D Geometry / Complex Shapes

Curve Related

BW vs Color

Fur/Eye/Face Related

Other

Boundary Detectors

Other

Other

Brightness

Other Color Contrast

 This is exceptionally unlikely to have occurred by chance.

 It's worth noting one confounding factor which might be

influencing the specialization. The 5x5 branches are the smallest

branches, but also have larger convolutions (5x5 instead of 3x3 or 1x1)

than their neighbors.There is something

which suggests that the branching plays an essential role: mixed3a and

mixed3b are adjacent layers which contain relatively similar features

and are at the same scale. If it was only about convolution size, why

Why is branch specialization consistent?

----------------------------------------

 Perhaps the most surprising thing about branch specialization is

that the same branch specializations seem to occur again and again,

across different architectures and tasks.

 For example, the branch specialization we observed in AlexNet –

the first layer specializing into a black-and-white Gabor branch vs. a

low-frequency color branch – is a surprisingly robust phenomenon. It

occurs consistently if you retrain AlexNet. It also occurs if you train

other architectures with the first few layers split into two branches.

It even occurs if you train those models on other natural image

datasets, like Places instead of ImageNet. Anecdotally, we also seem to

see other types of branch specialization recur. For example, finding

branches that seem to specialize in curve detection seems to be quite

 So, why do the same branch specializations occur again and again?

 One hypothesis seems very tempting. Notice that many of the same

features that form in normal, non-branched models also seem to form in

branched models. For example, the first layer of both branched and

non-branched models contain Gabor filters and color features. If the

same features exist, presumably the same weights exist between them.

 Could it be that branching is just surfacing a structure that

already exists? Perhaps there's two different subgraphs between the

weights of the first and second conv layer in a normal model, with

relatively small weights between them, and when you train a branched

model these two subgraphs latch onto the branches.

 (This would be directionally similar to work finding modular

substructures within neural networks.)

 To test this, let's look at models which have non-branched first

and second convolutional layers. Let's take the weights between them and

 perform a singular value decomposition (SVD) on the absolute values of

the weights. This will show us the main factors of variation in which

neurons connect to different neurons in the next layer (irrespective of

whether those connections are excitatory or inhibitory).

 Sure enough, the singular vector (the largest factor of variation)

 of the weights between the first two convolutional layers of

InceptionV1 is color.

first convolutional layer

Neurons in the

organized by the left singular vectors of |W|.

InceptionV1 (tf-slim version) trained on ImageNet.

Singular Vector 1 (frequency?)

Singular Vector 0 (color?)

InceptionV1 trained on Places365

Singular Vector 1 (frequency?)

Singular Vector 0 (color?)

Singular Vector 1 (frequency?)

Singular Vector 0 (color?)

Singular Vector 1 (frequency?)

Singular Vector 0 (color?)

second convolutional layer

Neurons in the  organized by the right singular vectors of |W|.

[6](#figure-6). Singular

vectors for the first and second convolutional layers of InceptionV1,

trained on ImageNet (above) or Places365 (below). One can think of

neurons being plotted closer together in this diagram as meaning they

likely tend to connect to similar neurons.

 This suggests an interesting prediction: perhaps if we were to split

the layer into more than two branches, we'd also observe specialization

in frequency in addition to color.

 This seems like it may be true. For example, here we see a

high-frequency black-and-white branch, a mid-frequency mostly

black-and-white branch, a mid-frequency color branch, and a

low-frequency color branch.

![](Branch%20Specialization_files/Figure_7.png)

[7](#figure-7). We

constructed a small ImageNet model with the first layer split into four

branches. The rest of the model is roughly an InceptionV1 architecture.

Parallels to neuroscience

-------------------------

 We've shown that branch specialization is one example of a

structural phenomenon — a larger-scale structure in a neural network. It

 happens in a variety of situations and neural network architectures,

and it happens with *consistency* – certain motifs of

specialization, such as color, frequency, and curves, happen

consistently across different architectures and tasks.

 Returning to our comparison with anatomy, although we hesitate to

claim explicit parallels to neuroscience, it's tempting to draw

analogies between branch specialization and the existence of regions of

the brain focused on particular tasks.

 The visual cortex, the auditory cortex, Broca's area and

Wernicke's area

 The subspecialization within the V2 area of the primate visual

cortex is another strong example from neuroscience. One type of stripe

within V2 is sensitive to orientation or luminance, whereas the other

type of stripe contains color-selective neurons.

 We are grateful to Patrick Mineault for noting this analogy, and

 for further noting that the high-frequency features are consistent with

 some of the known representations of high-level features in the primate

 V2 area.

 – these are all examples of brain areas with such consistent

specialization across wide populations of people that neuroscientists

and psychologists have been able to characterize as having remarkably

consistent functions.

 As researchers without expertise in neuroscience, we're uncertain

how useful this connection is, but it may be worth considering whether

branch specialization can be a useful model of how specialization might

emerge in biological neural networks.

![](Branch%20Specialization_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

[Visualizing Weights](https://distill.pub/2020/circuits/visualizing-weights/)

[Weight Banding](https://distill.pub/2020/circuits/weight-banding/)

 .comment {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 margin-top: 1em;

 }

 .comment h3 {

 font-size: 100%;

 font-weight: bold;

 text-transform: uppercase;

 margin-top: 0px;

 }

 .comment .commenter-description {

 font-style: italic;

 margin-bottom: 1em;

 margin-top: 0px;

 }

 .comment .commenter-description, .comment .commenter-description a {

 color: #777;

 font-size: 80%;

 line-height: 160%;

 }

 .comment p {

 margin-bottom: 0px;

 }

 .comment div {

 margin-top: 1em;

 }

### Comment

 is Professor of Neural Circuits and Computation at the Centre for

Discovery Brain Sciences and Simons Initiative for the Developing Brain,

 University of Edinburgh.

As neuroscientists we're excited by this work as it offers

fresh theoretical perspectives on long-standing questions about how

brains are organised and how they develop. Branching and specialisation

are found throughout the brain. A well studied example is the dorsal and

 ventral visual streams, which are associated with spatial and

non-spatial visual processing. At the microcircuit level neurons in each

 pathway are similar. However, recordings of neural activity demonstrate

 remarkable specialisation; classic experiments from the 1970s and 80s

established the idea that the ventral stream enables identification of

objects whereas the dorsal stream represents their location. Since then,

 much has been learned about signal processing in these pathways but

fundamental questions such as why there are multiple streams and how

they are established remain unanswered.

From the perspective of a neuroscientist, a striking result

from the investigation of branch specialization by Voss and her

colleagues is that robust branch specialisation emerges in the absence

of any complex branch specific design rules. Their analyses show that

specialisation is similar within and across architectures, and across

different training tasks. The implication here is that no specific

instructions are required for branch specialisation to emerge. Indeed,

their analyses suggest that it even emerges in the absence of

predetermined branches. By contrast, the intuition of many

neuroscientists would be that specialisation of different areas of the

neocortex requires developmental mechanisms that are specific to each

area. For neuroscientists aiming to understand how perceptual and

cognitive functions of the brain arise, an important idea here is that

developmental mechanisms that drive the separation of cortical pathways,

 such as the dorsal and ventral visual streams, may be absolutely

critical.

While the parallels between branch specialization in

artificial neural networks and neural circuits in the brain are

striking, there are clearly major differences and many outstanding

questions. From the perspective of building artificial neural networks,

we wonder if branch specific tuning of individual units and their

connectivity rules would enhance performance? In the brain, there is

good evidence that the activation functions of individual neurons are

fine-tuned between and even within distinct neural circuits. If this

fine tuning confers benefits to the brain then we might expect similar

benefits in artificial networks. From the perspective of understanding

the brain, we wonder whether branch specialisation could help make

experimentally testable predictions? If artificial networks can be

engineered with branches that have organisation similar to branching

pathways in the brain, then manipulations to these networks could be

compared to experimental manipulations achieved with optogenetic and

chemogenetic strategies. Given that many brain disorders involve changes

 to specific neural populations, similar strategies could give insights

into how these pathological changes alter brain functions. For example,

very specific populations of neurons are disrupted in early stages of

Alzheimer's disease. By disrupting corresponding units in neural network

 models one could explore the resulting computational deficits and

possible strategies for restoration of cognitive functions.

Imagenet classification with deep convolutional neural networks

Visualizing higher-layer features of a deep network

Deep inside convolutional networks: Visualising image classification models and saliency maps

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Feature Visualization

Going deeper with convolutions

Neural architecture search with reinforcement learning

Neural networks are surprisingly modular

Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks

Segregation of form, color, and stereopsis in primate area 18

Representation of Angles Embedded within Contour Stimuli in Area V2 of Macaque Monkeys


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'

Logan Engstrom
Justin Gilmer
Gabriel Goh
Dan Hendrycks
Andrew Ilyas
Aleksander Madry
Reiichiro Nakano
Preetum Nakkiran
Shibani Santurkar
Brandon Tran
Dimitris Tsipras
Eric Wallace

Abstract: On May 6th, Andrew Ilyas and colleagues published a paper  outlining two sets of experiments.             Firstly, they showed that models trained on adversarial examples can transfer to real data,             and secondly that models trained on a dataset derived from the representations of robust neural networks             seem to inherit non-trivial robustness.             They proposed an intriguing interpretation for their results:             adversarial examples are due to “non-robust features” which are highly predictive but imperceptible to             humans.

 outlining two sets of experiments.

 seem to inherit non-trivial robustness.

 They proposed an intriguing interpretation for their results:

 humans.

 The paper was received with intense interest and discussion

 on social media, mailing lists, and reading groups around the world.

 How should we interpret these experiments?

 Would they replicate?

 disciplines of machine learning,

 because it requires researchers to play both attack and defense.

 It's easy for even very rigorous researchers to accidentally use a weak attack.

 However, as we'll see, Ilyas et al's results have held up to initial scrutiny.

 And if non-robust features exist… what are they?

 Why not just have everyone write private blog posts like Ferenc?

 can give more researchers license to invest energy in discussing other's work

 published.

 We invited a number of researchers

 The Machine Learning community

 [sometimes](https://www.machinelearningdebates.com/program)

 that peer review isn't thorough enough.

 In contrast to this, we were struck by how deeply respondents engaged.

 deeply about the original paper.

 and forth!

 even running new experiments in response to comments.

 discussion articles in the future.

Discussion Themes

-----------------

**Clarifications**:

 Discussion between the respondents and original authors was able

 to surface several misunderstandings or opportunities to sharpen claims.

 The original authors summarize this in their rebuttal.

**Successful Replication**:

 the non-robust dataset experiments.

 Preetum reproduced the D^det\widehat{\mathcal{D}}\_{det}D

det non-robust dataset experiment as described in the

 paper, for L∞L\_\inftyL∞ and L2L\_2L2 attacks.

 Gabriel repproduced both D^det\widehat{\mathcal{D}}\_{det}D

det and D^rand\widehat{\mathcal{D}}\_{rand}D

rand for L2L\_2L2

 attacks.

 Preetum also replicated part of the robust dataset experiment by

 and hyperparameters he tried.

**Exploring the Boundaries of Non-Robust Transfer**:

 where training on adversarial examples transfers to real data.

 When, how, and why does it happen?

 Gabriel Goh explores an alternative mechanism for the results,

 Preetum Nakkiran shows a special construction where it doesn't happen,

**Properties of Robust and Non-Robust Features**:

 distribution shift,

Comments

--------

 Distill collected six comments on the original paper.

 They are presented in alphabetical order by the author's last name,

###

[Adversarial Example Researchers Need to Expand What is Meant by

 "Robustness"](https://distill.pub/2019/advex-bugs-discussion/response-1/)

### Authors

### Affiliations

[Justin Gilmer](https://www.linkedin.com/in/jmgilmer)

[Google Brain Team](https://g.co/brain)

[Dan Hendrycks](https://people.eecs.berkeley.edu/~hendrycks/)

[UC Berkeley](https://www.berkeley.edu/)

 Justin and Dan discuss "non-robust features" as a special case

 of models being non-robust because they latch on to superficial correlations,

 a view often found in the distributional robustness literature.

 They emphasize we should think about a broader notion of robustness.

#### Comment from original authors:

 appears "meaningless" to humans.

 to rely on.

###

### Authors

### Affiliations

[Gabriel Goh](https://gabgoh.github.io/)

[OpenAI](https://openai.com/)

 results.

 D^rand\widehat{\mathcal{D}}\_{rand}D

rand experiment,

 but finds no evidence for it effecting the D^det\widehat{\mathcal{D}}\_{det}D

det experiment.

#### Comment from original authors:

 motivations for designing the D^det\widehat{\mathcal{D}}\_{det}D

det dataset.

###

### Authors

### Affiliations

[Gabriel Goh](https://gabgoh.github.io/)

[OpenAI](https://openai.com/)

 He provides two constructions:

 and "ensembles" that could be candidates for true useful non-robust features.

#### Comment from original authors:

 features for real datasets (and thus a neat corroboration of their existence).

 interesting direction of developing a more fine-grained definition of features.

###

### Authors

###

[Reiichiro Nakano](https://reiinakano.com/)

 Reiichiro shows that adversarial robustness makes neural style transfer

 work by default on a non-VGG architecture.

 to humans.

#### Comment from original authors:

 trained models will have in neural network art!

 interesting links between robustness and style transfer.

###

### Authors

### Affiliations

[Preetum Nakkiran](https://preetum.nakkiran.org/)

[OpenAI](https://openai.com/) &

 [Harvard University](https://www.harvard.edu/)

 has no "non-robust features".

#### Comment from original authors:

 example of adversarial examples that arise from "bugs".

###

### Authors

### Affiliations

[Eric Wallace](https://www.ericswallace.com/)

[Allen Institute for AI](https://allenai.org/)

 Eric shows that training on a model's training

errors,

 or on how it predicts examples form an unrelated

dataset,

 can both transfer to the true test set.

 These experiments are analogous to the original

paper's non-robust transfer results — all three results are examples of a

 kind of "learning from incorrectly labeled data."

#### Comment from original authors:

 settings.

Original Author Discussion and Responses

----------------------------------------

###

### Authors

### Affiliations

[Logan Engstrom](http://loganengstrom.com/),

 [Andrew Ilyas](http://andrewilyas.com/),

 [Aleksander Madry](https://people.csail.mit.edu/madry/),

 [Shibani Santurkar](http://people.csail.mit.edu/shibani/),

 Brandon Tran,

 [Dimitris Tsipras](http://people.csail.mit.edu/tsipras/)

MIT

 conversation.

 This article also contains their responses to each comment.

Adversarial examples are not bugs, they are features


Understanding RL Vision

Jacob Hilton
Nick Cammarata
Shan Carter
Gabriel Goh
Chris Olah

### Contents

[Introduction](#introduction)

[Our CoinRun model](#coinrun)

[Model analysis](#analysis)

* [Dissecting failure](#dissecting-failure)

* [Hallucinations](#hallucinations)

* [Model editing](#model-editing)

[The diversity hypothesis](#diversity-hypothesis)

[Feature visualization](#feature-visualization)

[Attribution](#attribution)

[Questions for further research](#questions)

 In this article, we apply interpretability techniques to a

reinforcement learning (RL) model trained to play the video game CoinRun

 . Using attribution  combined with dimensionality reduction as in ,

 we build an interface for exploring the objects detected by the model,

and how they influence its value function and policy. We leverage this

interface in several ways.

* **[Dissecting failure](#dissecting-failure).**

 We perform a step-by-step analysis of the agent's behavior in cases

where it failed to achieve the maximum reward, allowing us to understand

 what went wrong, and why. For example, one case of failure was caused

by an obstacle being temporarily obscured from view.

* **[Hallucinations](#hallucinations).** We find

 situations when the model "hallucinated" a feature not present in the

observation, thereby explaining inaccuracies in the model's value

function. These were brief enough that they did not affect the agent's

behavior.

* **[Model editing](#model-editing).** We

hand-edit the weights of the model to blind the agent to certain

hazards, without otherwise changing the agent's behavior. We verify the

effects of these edits by checking which hazards cause the new agents to

 fail. Such editing is only made possible by our previous analysis, and

thus provides a quantitative validation of this analysis.

 for interpretability. If it is correct, then we can expect RL models to

 become more interpretable as the environments they are trained on

become more diverse. We provide evidence for our hypothesis by measuring

 the relationship between interpretability and generalization.

Our CoinRun model

-----------------

 CoinRun is a side-scrolling platformer in which the agent must

dodge enemies and other traps and collect the coin at the end of the

level.

 CoinRun is procedurally-generated, meaning that each new level

encountered by the agent is randomly generated from scratch. This

incentivizes the model to learn how to spot the different kinds of

objects in the game, since it cannot get away with simply memorizing a

|  |  |  |  |  |  |  |  |

| --- | --- | --- | --- | --- | --- | --- | --- |

| Full resolution |

 |  |  |

 |

 |  |  |

| Model resolution |

 |  |  |

 |

 |  |  |

 in the velocity info allows the model to infer the agent's motion from a

 single frame. The shade of the left square indicates the agent's

horizontal velocity (black for left at full speed, white for right at

full speed), and the shade of the right square indicates the agent's

vertical velocity (black for down at full speed, white for up at full

speed). In this example, the agent is moving forward and about to land

(and is thus moving right and down). |

 There are 9 actions available to the agent in CoinRun:

|  |  |  |  |

| --- | --- | --- | --- |

| ← | → |  | Left and right change the agent's horizontal

velocity. They still work while the agent is in mid-air, but have less

of an effect. |

| ↑ | ↖ | ↗ | Up causes the agent to jump after the next

non-up action. Diagonal directions have the same effect as both

component directions combined. |

| A | B | C | A, B and C do nothing.The

 original version of CoinRun only has 1 "do nothing" action, but our

version ended up with 3 when "A" and "B" actions were added to be used

in other games. For consistency, we have relabeled the original "do

nothing" action as "C". |

 except that we used twice as many copies of the environment per worker

and twice and many workers. The effect of these changes was to increase

the effective batch size, which seemed to be necessary to reach the same

 We used a non-recurrent network, to avoid any need to visualize

multiple frames at once. Thus our model observes a single downsampled

64x64 image, and outputs a value function (an estimate of the total

future time-discounted reward) and a policy (a probability distribution

over the actions, from which the next action is sampled).

observation

CNN

value function

logits

softmax

policy

Model analysis

--------------

 we developed an interface for examining trajectories of the agent

playing the game. This incorporates attribution from a hidden layer that

 recognizes objects, which serves to highlight objects that positively

or negatively influence a particular network output. By applying

dimensionality reduction, we obtain attribution vectors whose components

 correspond to different types of object, which we indicate using

different colors.

 Here is our interface for a typical trajectory, with the value

function as the network output. It reveals the model using obstacles,

coins, enemies and more to compute the value function.

| Value function attribution |

| --- |

| Observation

  Video game pixels seen by the model | Positive attribution

 (bad news)  Color overlay shows objects predictive of failure |

| --- | --- | --- |

|     |      |      |

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

  Timeline ← ►

 Speed:  FPS →

        Value function

 Estimated discounted win probability 93.9%

### Dissecting failure

 Our fully-trained model fails to complete around 1 in every 200

levels. We explored a few of these failures using our interface, and

found that we were usually able to understand why they occurred.

 The failure often boils down to the fact that the model has no

memory, and must therefore choose its action based only on the current

observation. It is also common for some unlucky sampling of actions from

 the agent's policy to be partly responsible.

|  |  |

| --- | --- |

|

 **Buzzsaw obstacle obscured by enemy**

 **Stepping down to avoid jumping**

 **Landing platform moving off-screen**

 |

 The agent moves too far to the right while in mid-air as a

 result of a buzzsaw obstacle being temporarily hidden from view by a

moving enemy. The buzzsaw comes back into view, but too late to avoid a

collision.

 The agent presses down in a bid to delay a jump. This

causes the agent to inadvertently step down from a box and onto an

enemy.

 The agent fails to move far enough to the right while in

mid-air, as a result of the platform where it was intending to land

moving below the field of view.

 |

Prev

Start

►

Next

End

Timestep **1** of **18**

 The agent jumps by releasing up, analyzing something (◦) on the wall.

(The "residual" feature can be triggered by a number of different

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO).**Timesteps 4–6**:

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO).**Timesteps 4–6**:

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

 Having adjusted its horizontal motion, the agent returns to a policy

 Having adjusted its horizontal motion, the agent returns to a policy

 Having adjusted its horizontal motion, the agent returns to a policy

 Having adjusted its horizontal motion, the agent returns to a policy

 Having adjusted its horizontal motion, the agent returns to a policy

 becomes clear to the agent again. The agent tries to move left to avoid

 it, and also prepares to immediately jump upon landing. Pulling off the

 jump actually matters more to its chances of survival, since it is

already moving right.**Timesteps 16–17**: The danger of the buzzsaw obstacle (•)

 becomes clear to the agent again. The agent tries to move left to avoid

 it, and also prepares to immediately jump upon landing. Pulling off the

 jump actually matters more to its chances of survival, since it is

 immediately in front of it. The agent cannot jump when it is in

mid-air, but jumping is only triggered when up is released, and it is

allowed to start pressing up before landing.**Timesteps 2–3**:

 The agent's policy is dominated by the 3 upward directions (which delay

 the jump) and down (which cancels it). Any other action would trigger

the jump. So the agent is trying to delay the jump, partly because of

 The agent's policy is dominated by the 3 upward directions (which delay

 the jump) and down (which cancels it). Any other action would trigger

the jump. So the agent is trying to delay the jump, partly because of

 The agent's policy is still dominated by the 3 updward directions and

down, and at this point the agent happens to move down. Unfortunately,

the agent does not seem to have taken into account that this will cause

 across the other side of the chasm ahead of it. Even though it is not

yet at the edge of the platform it is currently on, it is timing its

jump so as to land on the nearest platform up ahead, perhaps because it

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO). Unfortunately, it happens

to move left at every timestep during this period.**Timesteps 4–6**:

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO). Unfortunately, it happens

to move left at every timestep during this period.**Timesteps 4–6**:

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO). Unfortunately, it happens

 has been reduced, and tries to move right to compensate. (Several

features read from the velocity info as a secondary purpose.) With some

 has been reduced, and tries to move right to compensate. (Several

features read from the velocity info as a secondary purpose.) With some

bad luck, it moves up but not right on its first move.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

bias is not reflected in the actions sampled.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

bias is not reflected in the actions sampled.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

bias is not reflected in the actions sampled.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

bias is not reflected in the actions sampled.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

|

 Attribution for: | → | show choices |  |

 Policy: (probabilities and sampled action) | ←→↓↑↗↖ABC |

| --- | --- | --- | --- | --- | --- |

| Choose attribution: |

 Value function |  ←  →  ↓  ↑  ↗  ↖  A  B  C  |

| --- | --- | --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

|

 Attribution for: | Value function | show choices |  |

 Policy: (probabilities and sampled action) | ←→↓↑↗↖ABC |

| --- | --- | --- | --- | --- | --- |

| Choose attribution: |

 Value function |  ←  →  ↓  ↑  ↗  ↖  A  B  C  |

| --- | --- | --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

|

 Attribution for: | Value function | show choices |  |

 Policy: (probabilities and sampled action) | ←→↓↑↗↖ABC |

| --- | --- | --- | --- | --- | --- |

| Choose attribution: |

 Value function |  ←  →  ↓  ↑  ↗  ↖  A  B  C  |

| --- | --- | --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

### Hallucinations

 which measures how successful each action turned out relative to the

agent's expectations. An unusually high or low GAE indicates that either

 something unexpected occurred, or the agent's expectations were

miscalibrated. Filtering for such timesteps can therefore find problems

with the value function or policy.

 Using our interface, we found a couple of cases in which the model

 "hallucinated" a feature not present in the observation, causing the

value function to spike.

|  |  |

| --- | --- |

|

 **Coin hallucination**

 **Buzzsaw hallucination**

 |

 At one point the value function spiked upwards from 95% to

 98% for a single timestep. This was due to a curved yellow-brown shape

in the background, which happened to appear next to a wall, being

mistaken for a coin.

 At another point the value function spiked downwards from

94% to 85% for a single timestep. This was due to the agent, colored in

gray-blue and crouching against a mottled background, being mistaken for

 a buzzsaw obstacle. An actual buzzsaw was also present in the

observation, but the main effect was from the misjudged agent, as shown

by the larger red circle around the agent (hover over the first legend

item to isolate).

 |

        Value function  98.0%

| Value function attribution |

| --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

        Value function  85.0%

| Value function attribution |

| --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

### Model editing

 Our analysis so far has been mostly qualitative. To quantitatively

 validate our analysis, we hand-edited the model to make the agent blind

 to certain features identified by our interface: buzzsaw obstacles in

one case, and left-moving enemies in another. Our method for this can be

 We evaluated each edit by measuring the percentage of levels that

the new agent failed to complete, broken down by the object that the

agent collided with to cause the failure. Our results show that our

edits were successful and targeted, with no statistically measurable

effects on the agent's other abilities.The data for this plot are as follows.

- Original model: 0.37% / 0.16% / 0.12% / 0.08%

- Buzzsaw obstacle blindness: 12.76% / 0.16% / 0.08% / 0.05%

- Enemy moving left blindness: 0.36% / 4.69% / 0.97% / 0.07%

Each model was tested on 10,000 levels.

 We did not manage to achieve complete blindness, however: the

buzzsaw-edited model still performed significantly better than the

Original model, invisible buzzsaws: 32.20% / 0.05% / 0.05% / 0.05%

We tested the model on 10,000 levels.

We

 experimented briefly with iterating the editing procedure, but were not

 able to achieve more than around 50% buzzsaw blindness by this metric

 Here are the original and edited models playing some cherry-picked levels.

|  |  |  |  |

| --- | --- | --- | --- |

|

 **Level 1**

 **Level 2**

 **Level 3**

►

►

►

 |

 |

 |

 |

The diversity hypothesis

------------------------

 All of the above analysis uses the same hidden layer of our

network, the third of five convolutional layers, since it was much

harder to find interpretable features at other layers. Interestingly,

the level of abstraction at which this layer operates – finding the

locations of various in-game objects – is exactly the level at which

CoinRun levels are randomized using procedural generation. Furthermore,

we found that training on many randomized levels was essential for us to

 be able to find any interpretable features at all.

 This led us to suspect that the diversity introduced by CoinRun's

randomization is linked to the formation of interpretable features. We

call this the diversity hypothesis:

>

>  Interpretable features tend to arise (at a given level of

> abstraction) if and only if the training distribution is diverse enough

> (at that level of abstraction).

>

 Our explanation for this hypothesis is as follows. For the forward

 implication ("only if"), we only expect features to be interpretable if

 they are general enough, and when the training distribution is not

diverse enough, models have no incentive to develop features that

generalize instead of overfitting. For the reverse implication ("if"),

we do not expect it to hold in a strict sense: diversity on its own is

not enough to guarantee the development of interpretable features, since

 they must also be relevant to the task. Rather, our intention with the

reverse implication is to hypothesize that it holds very often in

practice, as a result of generalization being bottlenecked by diversity.

 of each level is randomized, and correspondingly, we were only able to

find interpretable features at the level of abstraction of objects. At a

 lower level, there are only a handful of visual patterns in the game,

and the low-level features of our model seem to consist mostly of

memorized color configurations used for picking these out. Similarly,

the game's high-level dynamics follow a few simple rules, and

accordingly the high-level features of our model seem to involve

mixtures of combinations of objects that are hard to decipher. To

### Interpretability and generalization

 To test our hypothesis, we made the training distribution less

diverse, by training the agent on a fixed set of 100 levels. This

dramatically reduced our ability to interpret the model's features. Here

 we display an interface for the new model, generated in the same way as

 the one [above](#interface). The smoothly increasing value

function suggests that the model has memorized the number of timesteps

until the end of the level, and the features it uses for this focus on

irrelevant background objects. Similar overfitting occurs for other

video games with a limited number of levels .

| Value function attribution |

| --- |

| Observation

  Video game pixels seen by the model | Positive attribution

 (bad news)  Color overlay shows objects predictive of failure |

| --- | --- | --- |

|     |      |      |

 Attribution legend

 (hover to isolate)

    ???   ???   ???   ???   ???   ???   ???  Agent, walls

and velocity

info? hover

to show residual

 Everything

else

  Timeline ← ►

 Speed:  FPS →

        Value function

 Estimated discounted win probability 94.4%

 We attempted to quantify this effect by varying the number of

levels used to train the agent, and evaluating the 8 features identified

 Features were scored based on how consistently they focused on the same

 objects, and whether the value function attribution made sense – for

example, background objects should not be relevant. This process was

subjective and noisy, but that may be unavoidable. We also measured the

generalization ability of each model, by testing the agent on unseen

levels .The data for this plot are as follows.

Comparison of models trained on different numbers of

levels. Two models were trained for each number of levels, and two

researchers independently evaluated how interpretable the features of

each model were, without being shown the number of levels.Our

 methodology had some flaws. Firstly, the researchers were not

completely blind to the number of levels: for example, it is possible to

 infer something about the number of levels from the smoothness of

graphs of the value function, since with fewer levels the model is

better able to memorize the number of timesteps until the end of the

level. Secondly, since evaluations are somewhat tedious, we stopped them

 once we thought the trend had become clear, introducing some selection

bias. Therefore these results should be considered primarily

illustrative. Each model was tested on 10,000 train and

10,000 test levels sampled with replacement. Shaded areas in the left

plot show the range of values over both models, though these are mostly

too narrow to be visible. Error bars in the right plot show ±1

population standard deviation over all four model–researcher pairs.

 Our results illustrate how diversity may lead to interpretable

features via generalization, lending support to the diversity

hypothesis. Nevertheless, we still consider the hypothesis to be highly

unproven.

Feature visualization

---------------------

[Feature visualization](https://distill.pub/2017/feature-visualization/)

 answers questions about what certain parts of a network are looking for

 by generating examples. This can be done by applying gradient descent

to the input image, starting from random noise, with the objective of

activating a particular neuron or group of neurons. While this method

works well for an image classifier trained on ImageNet ,

 for our CoinRun model it yields only featureless clouds of color. Only

for the first layer, which computes simple convolutions of the input,

does the method produce comparable visualizations for the two models.

|  | ImageNet | CoinRun |

| --- | --- | --- |

|

 First layer

  |

 |

 |

|

 Intermediate layer

  |

 |

 |

 Each image was chosen to activate a neuron in the center, with the 3

images corresponding to the first 3 channels. Jittering was applied

between optimization steps of up to 2 pixels for the first layer, and up

 To try to get it to work for CoinRun, we varied the method in a number

of ways. Nothing we tried had any noticeable effect on the quality of

the visualizations.

* **Transformation robustness.** This is the method of

stochastically jittering, rotating and scaling the image between

optimization steps, to search for examples that are robust to these

transformations . We tried both

increasing and decreasing the size of the jittering. Rotating and

scaling are less appropriate for CoinRun, since the observations

themselves are not invariant to these transformations.

* **Penalizing extremal colors.**By

 an "extremal" color we mean one of the 8 colors with maximal or minimal

 RGB values (black, white, red, green, blue, yellow, cyan and magenta).

 Noticing that our visualizations tend to use extremal colors towards

the middle, we tried including in the visualization objective an L2

penalty of various strengths on the activations of the first layer,

which successfully reduced the size of the extremally-colored region but

 did not otherwise help.

 caricature objective is to maximize the dot product between the

activations of the input image and the activations of a reference image.

 Caricatures are often an especially easy type of feature visualization

to make work, and helpful for getting a first glance into what features a

* **Low-level visual diversity.** In an attempt to

broaden the distribution of images seen by the model, we retrained it on

 a version of the game with procedurally-generated sprites. We

additionally tried adding noise to the images, both independent

per-pixel noise and spatially-correlated noise. Finally, we experimented

 As shown [below](#dataset-examples), we were able to

use dataset examples to identify a number of channels that pick out

human-interpretable features. It is therefore striking how resistant

gradient-based methods were to our efforts. We believe that this is

because solving CoinRun does not ultimately require much visual ability.

 Even with our modifications, it is possible to solve the game using

simple visual shortcuts, such as picking out certain small

configurations of pixels. These shortcuts work well on the narrow

distribution of images on which the model is trained, but behave

unpredictably in the full space of images in which gradient-based

optimization takes place.

 In support of the hypothesis, we have examples of features that are

hard to interpret in the absence of diversity. But there is also

evidence that the hypothesis may need to be refined. Firstly, it seems

to be a lack of diversity at a low level of abstraction that harms our

ability to interpret features at all levels of abstraction, which could

be due to the fact that gradient-based feature visualization needs to

back-propagate through earlier layers. Secondly, the failure of our

efforts to increase low-level visual diversity suggests that diversity

may need to be assessed in the context of the requirements of the task.

### Dataset example-based feature visualization

 As an alternative to gradient-based feature visualization, we use

dataset examples. This idea has a long history, and can be thought of as

 a heavily-regularized form of feature visualization .

 In more detail, we sample a few thousand observations infrequently from

 the agent playing the game, and pass them through the model. We then

apply a dimensionality reduction method known as non-negative matrix

factorization (NMF) to the activation channels .More

 precisely, we find a non-negative approximate low-rank factorization of

 the matrix obtained by flattening the spatial dimensions of the

activations into the batch dimension. This matrix has one row per

 For each of the resulting channels (which correspond to weighted

combinations of the original channels), we choose the observations and

spatial positions with the strongest activation (with a limited number

of examples per position, for diversity), and display a patch from the

observation at that position.

![](Understanding%20RL%20Vision_files/layer_2b_feature_6.png)Coin

![](Understanding%20RL%20Vision_files/layer_2b_feature_9.png)Step

![](Understanding%20RL%20Vision_files/layer_2b_feature_11.png)Left edge of box

![](Understanding%20RL%20Vision_files/layer_2b_feature_13.png)Buzzsaw obstacle

 Unlike gradient-based feature visualization, this method finds

some meaning to the different directions in activation space. However,

it may still fail to provide a complete picture for each direction,

since it only shows a limited number of dataset examples, and with

limited context.

### Spatially-aware feature visualization

 CoinRun observations differ from natural images in that they are

much less spatially invariant. For example, the agent always appears in

the center, and the agent's velocity is always encoded in the top left.

As a result, some features detect unrelated things at different spatial

positions, such as reading the agent's velocity in the top left while

detecting an unrelated object elsewhere. To account for this, we

developed a spatially-aware version of dataset example-based feature

visualization, in which we fix each spatial position in turn, and choose

 the observation with the strongest activation at that position (with a

limited number of reuses of the same observation, for diversity). This

creates a spatial correspondence between visualizations and

observations.

 Here is such a visualization for a feature that responds strongly

to coins. The white squares in the top left show that the feature also

responds strongly to the horizontal velocity info when it is white,

corresponding to the agent moving right at full speed.

![](Understanding%20RL%20Vision_files/feature_vis_spatial.png)

 Transparency (revealing the diagonally-striped background) indicates a

weak response, so the left half of the visualization is mostly

transparent because coins never appear in the left half of observations.

Attribution

-----------

 Attribution

 answers questions about the relationships between neurons. It is most

commonly used to see how the input to a network affects a particular

 how we applied this method a hidden layer, and how positive value

function attribution can be thought of as "good news" and negative value

 function attribution can as "bad news".

### Dimensionality reduction for attribution

 We showed [above](#dataset-examples) that a

dimensionality reduction method known as non-negative matrix

factorization (NMF) could be applied to the channels of activations to

 before, we obtain the NMF directions by sampling a few thousand

observations infrequently from the agent playing the game, computing the

 attributions, flattening the spatial dimensions into the batch

 workaround is to separate out the positive and negative parts of the

attributions and concatenate them along the batch dimension. We could

also have concatenated them along the channel dimension.).

Both methods tend to produce NMF directions that are close to one-hot,

and so can be thought of as picking out the most relevant channels.

However, when reducing to a small number of dimensions, using

attributions usually picks out more salient features, because

 Following , after applying

NMF to attributions, we visualize them by assigning a different color to

 each of the resulting channels. We overlay these visualizations over

 This gives a basic version of our interface, which allows us to see the

 effect of the main features at different spatial positions.

| --- | --- | --- |

|

 |

 |

 |

Legend (hover to isolate)

Buzzsaw

obstacle

Coin

Enemy

moving

left

Agent

or enemy

moving right

 of our CoinRun model, reduced to 4 channels using attribution-based

NMF. The dataset example-based feature visualizations of these

directions reveal more salient features than the visualizations of the

first 4 activation-based NMF directions from the preceding section.

 For the [full version](#interface) of our interface, we

 simply repeat this for an entire trajectory of the agent playing the

game. We also incorporate video controls, a timeline view of compressed

observations , and additional information,

such as model outputs and sampled actions. Together these allow the

trajectory to be easily explored and understood.

### Attribution discussion

* **Sparsity.** Attribution tends to be concentrated in a

very small number of spatial positions and (post-NMF) channels. For

example, in the figure above, the top 10 position–channel pairs account

for more than 80% of the total absolute attribution. This might be

explained by our [earlier](#feature-visualization-discussion)

 hypothesis that the model identifies objects by picking out certain

small configurations of pixels. Because of this sparsity, we smooth out

attribution over nearby spatial positions for the full version of our

interface, so that the amount of visual space taken up can be used to

judge attribution strength. This trades off some spatial precision for

more precision with magnitudes.

* **Unexpected sign.** Value function attribution usually has

 the sign one would expect: positive for coins, negative for enemies,

and so on. However, this is sometimes not the case. For example, in the

figure above, the red channel that detects buzzsaw obstacles has both

positive and negative attribution in two neighboring spatial positions

towards the left. Our best guess is that this phenomenon is a result of

statistical [collinearity](https://en.wikipedia.org/wiki/Multicollinearity),

 caused by certain correlations in the procedural level generation

together with the agent's behavior. These could be visual, such as

correlations between nearby pixels, or more abstract, such as both coins

 and long walls appearing at the end of every level. As a toy example,

supposing the value function ought to increase by 2% when the end of the

 level becomes visible, the model could either increase the value

function by 1% for coins and 1% for long walls, or by 3% for coins and

−1% for long walls, and the effect would be similar.

* **Outlier frames.** When an unusual event causes the

network to output extreme values, attribution can behave especially

strangely. For example, in the [buzzsaw hallucination](#hallucinations)

 frame, most features have a significant amount of both positive and

negative attribution. We do not have a good explanation for this, but

perhaps features are interacting in more complicated ways than usual.

Moreover, in these cases there is often a significant component of the

attribution lying outside the space spanned by the NMF directions, which

 we display as an additional "residual" feature. This could be because

each frame is weighted equally when computing NMF, so outlier frames

have little influence over the NMF directions.

Questions for further research

------------------------------

### The [diversity hypothesis](#diversity-hypothesis)

2. **Relationship to generalization.** What is the three-way

relationship between diversity, interpretable features and

generalization? Do non-interpretable features indicate that a model will

 fail to generalize in certain ways? Generalization refers implicitly to

 an underlying distribution – how should this distribution be chosen?For

 example, to measure generalization for CoinRun models trained on a

limited number of levels, we used the distribution over all possible

procedurally-generated levels. However, to formalize the sense in which

CoinRun is not diverse in its visual patterns or dynamics rules, one

would need a distribution over levels from a wider class of games.

3. **Caveats.** How are interpretable features affected by

other factors, such as the choice of task or algorithm, and how do these

 interact with diversity? Speculatively, do big enough models obtain

4. **Quantification.** Can we quantitatively predict how much

diversity is needed for interpretable features, perhaps using

generalization metrics? Can we be precise about what is meant by an

"interpretable feature" and a "level of abstraction"?

### Interpretability in the absence of diversity

1. **Pervasiveness of non-diverse features.** Do "non-diverse

features", by which we mean the hard-to-interpret features that tend to

arise in the absence of diversity, remain when diversity is present? Is

there a connection between these non-diverse features and the

"non-robust features" that have been posited to explain adversarial

examples ?

2. **Coping with non-diverse levels of abstraction.** Are

there levels of abstraction at which even broad distributions like

ImageNet remain non-diverse, and how can we best interpret models at

these levels of abstraction?

 in the absence of diversity, and can it be made to work using

transformation robustness, regularization, data augmentation,

adversarial training, or other techniques? What property of the

optimization leads to the clouds of [extremal colors](#extremal-colors)?

 feature visualization may be untrustworthy by failing to separate the

things causing certain behavior from the things that merely correlate

### Interpretability in the RL framework

1. **Non-visual and abstract features.** What are the best

methods for interpreting models with non-visual inputs? Even vision

models may also have interpretable abstract features, such as

relationships between objects or anticipated events: will any method of

generating examples be enough to understand these, or do we need an

entirely new approach? For models with memory, how can we interpret

their hidden states ?

3. **Modifying training.** In what ways can we train RL models

 to make them more interpretable without a significant performance cost,

 such as by altering architectures or adding auxiliary predictive

losses?

4. **Leveraging the environment.** How can we enrich

interfaces using RL-specific data, such as trajectories of

agent–environment interaction, state distributions, and advantage

estimates? What are the benefits of incorporating user–environment

interaction, such as for exploring counterfactuals?

### What we would like to see from further research and why

 We are motivated to study interpretability for RL for two reasons.

* **To be able to interpret RL models.** RL can be applied to

 an enormous variety of tasks, and seems likely to be a part of

increasingly influential AI systems. It is therefore important to be

able to scrutinize RL models and to understand how they might fail. This

 may also benefit RL research through an improved understanding of the

pitfalls of different algorithms and environments.

* **As a testbed for interpretability techniques.** RL models

 pose a number of distinctive challenges for interpretability

techniques. In particular, environments like CoinRun straddle the

boundary between memorization and generalization, making them useful for

 studying the [diversity hypothesis](#diversity-hypothesis) and related ideas.

 We think that large neural networks are currently the most likely

type of model to be used in highly capable and influential AI systems in

 the future. Contrary to the traditional perception of neural networks

as black boxes, we think that there is a fighting chance that we will be

 able to clearly and thoroughly understand the behavior even of very

large networks. We are therefore most excited by neural network

interpretability research that scores highly according to the following

criteria.

* **Scalability.** The takeaways of the research should have

some chance of scaling to harder problems and larger networks. If the

techniques themselves do not scale, they should at least reveal some

relevant insight that might.

* **Trustworthiness.** Explanations should be faithful to the

 model. Even if they do not tell the full story, they should at least

not be biased in some fatal way (such as by using an approval-based

objective that leads to bad explanations that sound good, or by

depending on another model that badly distorts information).

* **Exhaustiveness.** This may turn out to be impossible at

scale, but we should strive for techniques that explain every essential

feature of our models. If there are theoretical limits to

exhaustiveness, we should try to understand these.

* **Low cost.** Our techniques should not be significantly

more computationally expensive than training the model. We hope that we

will not need to train models differently for them to be interpretable,

but if we do, we should try to minimize both the computational expense

and any performance cost, so that interpretable models are not

disincentivized from being used in practice.

 Our proposed questions reflect this perspective. One of the

reasons we emphasize diversity relates to exhaustiveness. If

"non-diverse features" remain when diversity is present, then our

current techniques are not exhaustive and could end up missing important

 features of more capable models. Developing tools to understand

non-diverse features may shed light on whether this is likely to be a

problem.

 We think there may be significant mileage in simply applying

existing interpretability techniques, with attention to detail, to more

models. Indeed, this was the mindset with which we initially approached

this project. If the diversity hypothesis is correct, then this may

become easier as we train our models to perform more complex tasks. Like

 early biologists encountering a new species, there may be a lot we can

glean from taking a magnifying glass to the creatures in front of us.

Supplementary material

----------------------

* **Model weights.** The weights of our model are available

for download, along with those of a number of other models, including

the models trained on different numbers of levels, the edited models,

Quantifying generalization in reinforcement learning

Visualizing and understanding convolutional networks

Striving for simplicity: The all convolutional net

Grad-CAM: Visual explanations from deep networks via gradient-based localization

Interpretable explanations of black boxes by meaningful perturbation

PatternNet and PatternLRP--Improving the interpretability of neural networks

The (un)reliability of saliency methods

Axiomatic attribution for deep networks

The Building Blocks of Interpretability

Leveraging Procedural Generation to Benchmark Reinforcement Learning

Proximal policy optimization algorithms

High-dimensional continuous control using generalized advantage estimation

Thread: Circuits

General Video Game AI: A multi-track framework for evaluating agents, games and content generation algorithms

Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning

Observational Overfitting in Reinforcement Learning

Feature Visualization

Visualizing higher-layer features of a deep network

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Imagenet: A large-scale hierarchical image database

Going deeper with convolutions

An Atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents

Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents

Caricatures

Towards deep learning models resistant to adversarial attacks

Intriguing properties of neural networks

Visualizing and understanding Atari agents

Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution

Video Interface: Assuming Multiple Perspectives on a Video Exposes Hidden Structure

Reconciling modern machine-learning practice and the classical bias--variance trade-off

Adversarial examples are not bugs, they are features

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'

Human-level performance in 3D multiplayer games with population-based reinforcement learning

Solving Rubik's Cube with a Robot Hand

Dota 2 with Large Scale Deep Reinforcement Learning

Does Attribution Make Sense?

IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Examples are Just Bugs, Too

Abstract: This article is part of a discussion of the Ilyas et al. paper                 “Adversarial examples are not bugs, they are features”.                 You can learn more in the                                      main discussion article                 .

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 color: hsl(0, 0%, 0.33);

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *"Adversarial examples are not bugs, they are features".*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[All Responses](https://distill.pub/2019/advex-bugs-discussion/#articles)

 We demonstrate that there exist adversarial examples which are just "bugs":

1. Do not transfer between models, and

2. Do not leak "non-robust features" which allow for learning, in the

 sense of Ilyas-Santurkar-Tsipras-Engstrom-Tran-Madry

 .

 We replicate the Ilyas et al.

 experiment of training on mislabeled adversarially-perturbed images

 (Section 3.2 of ),

 and show that it fails for our construction of adversarial perturbations.

 The message is, whether adversarial examples are features or bugs depends

 classifier.

 intrinsically have any vulnerable directions.

### Background

 Many have understood Ilyas et al.

 to claim that adversarial examples are not "bugs", but are "features".

 Specifically, Ilyas et al. postulate the following two worlds:

 As communicated to us by the original authors.

 In this world, adversarial examples occur because classifiers behave

 poorly off-distribution,

 when they are evaluated on inputs that are not natural images.

 Here, adversarial examples would occur in arbitrary directions,

 having nothing to do with the true data distribution.

 and which contain features of the target class.

 For example, consider the perturbation that

 makes an image of a dog to be classified as a cat.

 vs. dogs.

 are in both.

 Ilyas et al.

 show that there exist adversarial examples in World 2, and we show there exist

 examples in World 1.

 Constructing Non-transferrable Targeted Adversarial Examples

--------------------------------------------------------------

 fff,

 which do not transfer to other classifiers trained for the same problem.

 f(x′)=ytargf(x′)=ytargf(x') = y\_{targ}.

 PGD is described in the appendix.

 which starts at input xxx, and iteratively takes steps {xt}{xt}\{x\_t\}

 to minimize the loss L(f,xt,ytarg)L(f,xt,ytarg)L(f, x\_t, y\_{targ}).

 That is, we take steps in the direction

 −∇xL(f,xt,ytarg)−∇xL(f,xt,ytarg)-
abla\_x L(f, x\_t, y\_{targ})

 where L(f,x,y)L(f,x,y)L(f, x, y) is the loss of fff on input xxx, label yyy.

 Note that since PGD steps in the gradient direction towards the target class,

 background).

 since the "blue" direction is correlated with the plane class.

 In our construction below, we attempt to eliminate such feature leakage.

 and an off-manifold "random component".

 Our construction below attempts to step only in the off-manifold direction.

### Our Construction

 for the same classification problem as fff.

 different random initializations.

 For input example (x,y)(x,y)(x, y) and target class ytargytargy\_{targ},

 we perform iterative updates to find adversarial attacks — as in PGD.

 However, instead of stepping directly in the gradient direction, we

 step in the direction

 Formally, we replace the iterative step with

 where ΠεΠε\Pi\_\eps is the projection onto the εε\eps-ball around xxx.

 we minimize the "disentangled loss"

 We could also consider explicitly using the ensemble to decorrelate,

 by stepping in direction

 This works well for small ϵϵ\epsilon,

 but the given loss has better optimization properties for larger ϵϵ\epsilon.

 This loss encourages finding an xtxtx\_t which is adversarial for fff,

 but not for the ensemble {fi}{fi}\{f\_i\}.

 these examples are also not adversarial for

 *new* classifiers trained for the same problem.

### Experiments

 We train a ResNet18 on CIFAR10 as our target classifier fff.

 We then test the probability that

 a targeted attack for fff

 transfers to a new (freshly-trained) ResNet18, with the same targeted class.

 For L∞L∞L\_{\infty} attacks:

|  |  |  |

| --- | --- | --- |

|  | Attack Success | Transfer Success |

| PGD | 99.6% | 52.1% |

| Ours | 98.6% | 0.8% |

 For L2L2L\_2 attacks:

|  |  |  |

| --- | --- | --- |

|  | Attack Success | Transfer Success |

| PGD | 99.9% | 82.5% |

| Ours | 99.3% | 1.7% |

Adversarial Examples With No Features

-------------------------------------

 Using the above, we can construct adversarial examples

 which *do not suffice* for learning.

 Here, we replicate the Ilyas et al. experiment

 that "Non-robust features suffice for standard classification"

 (Section 3.2 of ),

 but show that it fails for our construction of adversarial examples.

 To review, the Ilyas et al. non-robust experiment was:

1. Train a standard classifier fff for CIFAR.

2. From the CIFAR10 training set S={(Xi,Yi)}S={(Xi,Yi)}S = \{(X\_i, Y\_i)\},

 Note that S′S′S' appears to humans as "mislabeled examples".

3. Train a new classifier f′f′f' on train set S′S′S'.

 Ilyas et al. use Step (3) to argue that

 adversarial examples have a meaningful "feature" component.

 However, for adversarial examples constructed using our method, Step (3) fails.

 (X,Y+1)(X,Y+1)(X, Y+1), which is intuitively what we trained on.

For L∞L∞L\_{\infty} attacks:

|  |  |  |

| --- | --- | --- |

| PGD | 23.7% | 40.4% |

| Ours | 2.5% | 75.9% |

 Table: Test Accuracies of f′f′f'

For L2L2L\_2 attacks:

|  |  |  |

| --- | --- | --- |

| PGD | 33.2% | 27.3% |

| Ours | 2.8% | 70.8% |

 Table: Test Accuracies of f′f′f'

Adversarial Squares: Adversarial Examples from Robust Features

--------------------------------------------------------------

 To further illustrate that adversarial examples can be "just bugs",

 reasonable

 *intrinsic* definition, this problem has no non-robust features.

 In particular, by the Ilyas et al. definition, **every** distribution

 distribution.

 family of classifiers being considered.

 overfitting, and

 label noise.

 with a small amount of random pixel noise and label noise.

 A sample of images from the distribution.

 Formally, let the distribution be as follows.

 Pick label Y∈{±1}Y∈{±1}Y \in \{\pm 1\} uniformly,

 egin{cases}

 (+ec{\mathbb{1}} + ec\eta\_\eps) \cdot \eta & 	ext{if $Y=1$}\

 (-ec{\mathbb{1}} + ec\eta\_\eps) \cdot \eta & 	ext{if $Y=-1$}\

 \end{cases}

 and

 A plot of samples from a 2D-version of this distribution is shown to the right.

 However, if we sample 10000 training images from this distribution, and train

 a ResNet18 to 99.9% train accuracy,

 the resulting classifier is highly non-robust:

 The input-noise and label noise are both essential for this construction.

 One intuition for what is happening is: in the initial stage of training

 classifier).

 to fit the label-noise, which hurts robustness.

 Figure adapted from .

Addendum: Data Poisoning via Adversarial Examples

-------------------------------------------------

 As an addendum, we observe that the "non-robust features"

 experiment of  (Section 3.2)

 directly implies data-poisoning attacks:

 to the classifier output labels (e.g. swapping cats and dogs).

 notation, and also using vanilla PGD to find adversarial examples.

 on distribution (X,Y)(X,Y)(X, Y).

 By permutation-symmetry of the labels, this implies that:

 on distribution (X,Y−1)(X,Y−1)(X, Y-1).

 but the classifier learns to predict the cyclically-shifted labels.

 Concretely, using the original numbers of

 Table 1 in , this reduction implies that

 and cause the learnt classifier to output shifted-labels

 43.7% of the time

 (cats classified as birds, dogs as deers, etc).**

 To cite Ilyas et al.'s response, please cite their

**Response Summary**: We note that as discussed in

 mere existence of adversarial

 examples

 that are "features" is sufficient to corroborate our main thesis. This comment

 illustrates, however, that we can indeed craft adversarial examples that are

 based on "bugs" in realistic settings. Interestingly, such examples don't

 transfer, which provides further support for the link between transferability

 and non-robust features.

 we did not intend to claim

 that adversarial examples arise *exclusively* from (useful) features but rather

 that useful non-robust features exist and are thus (at least

 partially) responsible for adversarial vulnerability. In fact,

 prior work already shows how in theory adversarial examples can arise from

 insufficient samples  or finite-sample overfitting

 , and the experiments

 presented here (particularly, the adversarial squares) constitute a neat

 real-world demonstration of these facts.

 Our main thesis that "adversarial examples will not just go away as we fix

 stemming from "bugs." As long as adversarial examples can stem from non-robust

 features (which the commenter seems to agree with), fixing these bugs will not

 solve the problem of adversarial examples.

Moreover, with regards to feature "leakage" from PGD, recall that in

 or D\_det dataset, the non-robust features are associated with the

 correct label whereas the robust features are associated with the wrong

 one. We wanted to emphasize that, as

 [shown in Appendix D.6](https://arxiv.org/abs/1905.02175) ,

 models trained on our DdetDdetD\_{det} dataset actually generalize *better* to

 the non-robust feature-label association that to the robust

 feature-label association. In contrast, if PGD introduced a small

 "leakage" of non-robust features, then we would expect the trained model

 would still predominantly use the robust feature-label association.

 That said, the experiments cleverly zoom in on some more fine-grained

 nuances in our understanding of adversarial examples. One particular thing that

 stood out to us is that by creating a set of adversarial examples that are

 *explicitly* non-transferable, one also prevents new classifiers from learning

 features from that dataset. This finding thus makes the connection between

 transferability of adversarial examples and their containing generalizing

 features even stronger! Indeed, we can add the constructed dataset into our

 "ˆDdetD^det\widehat{\mathcal{D}}\_{det} learnability vs transferability" plot

 (Figure 3 in the paper) — the point

 corresponding to this dataset fits neatly onto the trendline!

 attacks

SGD on Neural Networks Learns Functions of Increasing Complexity

Adversarially Robust Generalization Requires More Data

A boundary tilting persepective on the phenomenon of adversarial examples