The Paths Perspective on Value Learning

Abstract: In the last few years, reinforcement learning (RL) has made remarkable progress, including beating world-champion Go players, controlling robotic hands, and even painting pictures.

Introduction

------------

 One of the key sub-problems of RL is value estimation – learning the

 long-term consequences of being in a state.

 This can be tricky because future returns are generally noisy,

affected by many things other than the present state. The further we

look into the future, the more this becomes true.

 But while difficult, estimating value is also essential to many

approaches to RL.For many approaches

(policy-value iteration), estimating value essentially is the whole

problem, while in other approaches (actor-critic models), value

estimation is essential for reducing noise.

 The natural way to estimate the value of a state is as the average

return you observe from that state. We call this Monte Carlo value

estimation.

**Cliff World**

 is a classic RL example, where the agent learns to

 walk along a cliff to reach a goal.

 Sometimes the agent reaches its goal.

 Other times it falls off the cliff.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/cliffworld-mc.svg)

 Monte Carlo averages over trajectories where they intersect.

 If a state is visited by only one episode, Monte Carlo says its

value is the return of that episode. If multiple episodes visit a state,

 Monte Carlo estimates its value as the average over them.

 Let's write Monte Carlo a bit more formally.

 In tabular settings such as the Cliff World example, this "update

 and we could just as easily use the "+=" notation. But when using

parameteric function approximators such as neural networks, our "update

towards" operator may represent a gradient step, which cannot be written

 in "+=" notation. In order to keep our notation clean and general, we

chose to use the ↩\hookleftarrow↩ operator throughout.

V(st)  V(s\_t)~~V(st)

↩  \hookleftarrow~~↩

RtR\_tRt

 State value

 Return

 The term on the right is called the return and we use it to measure

the amount of long-term reward an agent earns. The return is just a

 is a discount factor which controls how much short term rewards are

worth relative to long-term rewards. Estimating value by updating

Beating Monte Carlo

-------------------

V(st)  V(s\_t)~~V(st)

↩  \hookleftarrow~~↩

rtr\_{t} rt

+++

γV(st+1)\gamma V(s\_{t+1})γV(st+1)

 State value

Reward

Next state value

 Intersections between two trajectories are handled differently

under this update. Unlike Monte Carlo, TD updates merge intersections so

 that the return flows backwards to all preceding states.

 Sometimes the agent reaches its goal.

 Other times it falls off the cliff.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/cliffworld-td.svg)

 TD learning merges paths where they intersect.

 What does it mean to "merge trajectories" in a more formal sense?

V(st+1)  V(s\_{t+1})~~V(st+1)

≃  \simeq~~≃

≃  \simeq~~≃

 Now we can use this equation to expand the TD update rule recursively:

V(st) V(s\_t)~V(st)

↩ \hookleftarrow~↩

rtr\_{t} rt

+++

γV(st+1)\gamma V(s\_{t+1})γV(st+1)

↩ \hookleftarrow~↩

rtr\_{t} rt

+++

γE[rt+1′]\gamma \mathop{\mathbb{E}} igr[ r'\_{t+1} igl]γE[rt+1′]

+++

↩ \hookleftarrow~↩

rtr\_{t} rt

+++

γE [rt+1′]\gamma \mathop{\mathbb{E}} ~ igr[ r'\_{t+1} igl]γE [rt+1′]

+++

+ +~+

...  …~~...

 This gives us a strange-looking sum of nested expectation values.

At first glance, it's not clear how to compare them with the more

simple-looking Monte Carlo update. More importantly, it's not clear that

 we *should* compare the two; the updates are so different that it

feels a bit like comparing apples to oranges. Indeed, it's easy to think

 of Monte Carlo and TD learning as two entirely different approaches.

 But they are not so different after all. Let's rewrite the Monte

Carlo update in terms of reward and place it beside the expanded TD

update.

**MC update**

V(st) V(s\_t)~V(st)

 ↩  ~\hookleftarrow~~ ↩

rtr\_{t}rt

+ +~+

γ rt+1\gamma ~ r\_{t+1}γ rt+1

+ +~+

γ2 rt+2\gamma^2 ~ r\_{t+2}γ2 rt+2

+ +~+

...…...

Reward from present path.

Reward from present path.

Reward from present path…

**TD update**

V(st) V(s\_t)~V(st)

 ↩  ~\hookleftarrow~~ ↩

rtr\_{t}rt

+ +~+

γE [rt+1′]\gamma \mathop{\mathbb{E}} ~ igr[ r'\_{t+1} igl]γE [rt+1′]

+ +~+

+ +~+

...…...

Reward from present path.

Expectation over paths intersecting present path.

Expectation over paths intersecting *paths intersecting* present path…

 A pleasant correspondence has emerged. The difference between

Monte Carlo and TD learning comes down to the nested expectation

operators. It turns out that there is a nice visual interpretation for

what they are doing. We call it the *paths perspective* on value learning.

The Paths Perspective

---------------------

 Trajectory 1

 Trajectory 2

 But this way of organizing experience de-emphasizes relationships *between*

 trajectories. Wherever two trajectories intersect, both outcomes are

valid futures for the agent. So even if the agent has followed

Trajectory 1 to the intersection, it could *in theory* follow

Trajectory 2 from that point onward. We can dramatically expand the

agent's experience using these simulated trajectories or "paths."

 #cliffworld-paths {

 display: grid;

 grid-template-columns: repeat(auto-fit, minmax(130px, 1fr));

 grid-gap: 20px;

 }

 Path 1

 Path 2

 Path 3

 Path 4

**Estimating value.** It turns out that Monte Carlo is

averaging over real trajectories whereas TD learning is averaging over

all possible paths. The nested expectation values we saw earlier

correspond to the agent averaging across *all possible future paths*.

 #compare-mctd {

 display: grid;

 grid-gap: 40px;

 grid-template-columns: repeat(auto-fit, minmax(320px, 1fr));

 }

 #compare-mctd .subfigure {

 display: grid;

 grid-gap: 20px;

 grid-template-columns: repeat(2, minmax(160px, 1fr));

 /\* grid-auto-rows: min-content; \*/

 /\* grid-template-rows: min-content auto; \*/

 }

 #compare-mctd .subfigure .column-heading {

 grid-column: 1 / -1;

 }

 #compare-mctd .figcaption {

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 padding-top: 5px;

 margin-top: 5px;

 /\* min-height: 179px; \*/

 /\* min-width: 179px; \*/

 /\* flex: 1; \*/

 }

#### Monte Carlo Estimation

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/traj-thumbnails.svg)

 Averages over **real trajectories**

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/cliffworld-mc.svg)

 Resulting MC estimate

#### Temporal Difference Estimation

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/path-thumbnails.svg)

 Averages over **possible paths**

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/cliffworld-td.svg)

 Resulting TD estimate

**Comparing the two.** Generally speaking, the best value

estimate is the one with the lowest variance. Since tabular TD and Monte

 Carlo are empirical averages, the method that gives the better estimate

 is the one that averages over more items. This raises a natural

question: Which estimator averages over more items?

Var[V(s)]  Var[V(s)]~~Var[V(s)]

∝  \propto~~∝

1Nrac{1}{N} N1

 Variance of estimate

Inverse of the number of items in the average

 First off, TD learning never averages over fewer trajectories than

 Monte Carlo because there are never fewer simulated trajectories than

real trajectories. On the other hand, when there are *more*

simulated trajectories, TD learning has the chance to average over more

of the agent's experience.

 This line of reasoning suggests that TD learning is the better

estimator and helps explain why TD tends to outperform Monte Carlo in

tabular environments.

Introducing Q-functions

-----------------------

 An alternative to the value function is the Q-function. Instead of

 estimating the value of a state, it estimates the value of a state and

an action. The most obvious reason to use Q-functions is that they allow

 us to compare different actions.

 #qlearning-intro {

 display: grid;

 grid-template-columns: repeat(auto-fit, minmax(130px, 1fr));

 grid-column-gap: 30px;

 }

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/policy.svg)

 Many times we'd like to compare the value of actions under a policy.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/value.svg)

 It's hard to do this with a value function.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/qvalue.svg)

 It's easier to use Q-functions, which estimate joint state-action values.

 There are some other nice properties of Q-functions. In order to

see them, let's write out the Monte Carlo and TD update rules.

Q(st,at)  Q(s\_t, a\_t)~~Q(st,at)

↩  \hookleftarrow~~↩

RtR\_tRt

 State-action value

 Return

 We still update towards the return. Instead of updating towards

the return of being in some state, though, we update towards the return

of being in some state *and* selecting some action.

 Now let's try doing the same thing with the TD update:

Q(st,at)  Q(s\_t, a\_t)~~Q(st,at)

↩  \hookleftarrow~~↩

rtr\_{t} rt

+++

γQ(st+1,at+1)\gamma Q(s\_{t+1}, a\_{t+1})γQ(st+1,at+1)

 State-action value

Reward

Next state value

 What we need is a better estimate of V(st+1)V(s\_{t+1})V(st+1).

Q(st,at)  Q(s\_t, a\_t)~~Q(st,at)

↩  \hookleftarrow~~↩

rtr\_{t} rt

+++

γV(st+1)\gamma V(s\_{t+1})γV(st+1)

 State-action value

Reward

Next state value

V(st+1)  V(s\_{t+1})~~V(st+1)

=  =~~=

?? ?

Learning Q-functions with reweighted paths

------------------------------------------

**Expected Sarsa.**

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/sarsa.svg)

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/expected-sarsa.svg)

 than a value estimate computed straight from the experience. This is

because the expectation value weights the Q-values by the true policy

distribution rather than the empirical policy distribution. In doing

**Off-policy value learning.** We can push this idea even

further. Instead of weighting Q-values by the true policy distribution,

we can weight them by an arbitrary policy, πoff\pi^{off}πoff:

**Off-policy value learning** weights Q-values by an arbitrary policy.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/off-policy.svg)

 This slight modification lets us estimate value under any policy

we like. It's interesting to think about Expected Sarsa as a special

case of off-policy learning that's used for on-policy estimation.

**Re-weighting path intersections.** What does the paths

perspective say about off-policy learning? To answer this question,

let's consider some state where multiple paths of experience intersect.

 #reweighting {

 display: grid;

 grid-template-columns: repeat(auto-fit, minmax(160px, 1fr));

 grid-gap: 30px;

 }

 /\* #reweighting .wrapper {

 min-width: 160px;

 } \*/

 #reweighting-full .cls-1 {

 fill: #2d307b;

 }

 #reweighting-full .cls-2 {

 fill: #e7ebe8;

 }

 #reweighting-full .cls-3 {

 fill: #cac9cc;

 }

 #reweighting-full .cls-4 {

 fill: #bd5f35;

 }

 #reweighting-full .cls-5, .cls-6, .cls-7 {

 fill: none;

 stroke-width: 10px;

 }

 #reweighting-full .cls-5 {

 stroke: #bd5f35;

 }

 #reweighting-full .cls-5, .cls-7 {

 stroke-miterlimit: 10;

 }

 #reweighting-full .cls-6 {

 stroke: #2d307b;

 stroke-linecap: round;

 stroke-linejoin: round;

 }

 #reweighting-full .cls-7 {

 stroke: #8191c9;

 }

 #reweighting-full .cls-8 {

 fill: #8191c9;

 }

 #reweighting-full .cls-9 {

 font-size: 24.25341px;

 fill: #d1d3d4;

 font-family: Arial-BoldMT, Arial;

 font-weight: 700;

 }

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/reweighting-1.svg)

 Multiple paths of experience intersect at this state.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/reweighting-2.svg)

+2

-1

Weight of upward path: 0.50

 Wherever intersecting paths are re-weighted, the paths that are

most representative of the off-policy distribution end up making larger

contributions to the value estimate. Meanwhile, paths that have low

probability make smaller contributions.

**Q-learning.** There are many cases where an agent needs to

collect experience under a sub-optimal policy (e.g. to improve

exploration) while estimating value under an optimal one. In these

cases, we use a version of off-policy learning called Q-learning.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/q-learning.svg)

 Q-learning prunes away all but the highest-valued paths. The paths

that remain are the paths that the agent will follow at test time; they

are the only ones it needs to pay attention to. This sort of value

**Double Q-Learning.** The problem with Q-learning is that it

gives biased value estimates. More specifically, it is over-optimistic

in the presence of noisy rewards. Here's an example where Q-learning

fails:

*You go to a casino and play a hundred slot machines. It's your

lucky day: you hit the jackpot on machine 43. Now, if you use Q-learning

 to estimate the value of being in the casino, you will choose the best

outcome over the actions of playing slot machines. You'll end up

thinking that the value of the casino is the value of the jackpot…and

decide that the casino is a great place to be!*

 Sometimes the largest Q-value of a state is large *just by chance*;

 choosing it over others makes the value estimate biased.

 One way to reduce this bias is to have a friend visit the casino

and play the same set of slot machines. Then, ask them what their

winnings were at machine 43 and use their response as your value

estimate. It's not likely that you both won the jackpot on the same

machine, so this time you won't end up with an over-optimistic estimate.

 We call this approach *Double Q-learning*.

**Putting it together.**  It's easy to think of Sarsa, Expected

 Sarsa, Q-learning, and Double Q-learning as different algorithms. But

#### On-policy methods

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/sarsa.svg)

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/expected-sarsa.svg)

#### Off-policy methods

**Off-policy value learning** weights Q-values by an arbitrary policy.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/off-policy.svg)

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/q-learning.svg)

 QBQ\_BQB.

**Re-weighting paths with Monte Carlo.** At this point, a

natural question is: Could we accomplish the same re-weighting effect

with Monte Carlo? We could, but it would be messier and involve

re-weighting all of the agent's experience. By working at intersections,

 TD learning re-weights individual transitions instead of episodes as a

whole. This makes TD methods much more convenient for off-policy

learning.

Merging Paths with Function Approximators

-----------------------------------------

 Up until now, we've learned one parameter — the value

estimate — for every state or every state-action pair. This works well

for the Cliff World example because it has a small number of states. But

 most interesting RL problems have a large or infinite number of states.

 This makes it hard to store value estimates for each state.

 #figure-fnapprox-intro {

 display: grid;

 grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));

 grid-column-gap: 20px;

 grid-row-gap: 10px;

 /\* grid-auto-flow: column; \*/

 }

 in these spaces often requires function approximation.

 memory and don't generalize.

 generalize to states they haven't visited yet.

 Instead, we must force our value estimator to have fewer

parameters than there are states. We can do this with machine learning

methods such as linear regression, decision trees, or neural networks.

All of these methods fall under the umbrella of function approximation.

**Merging nearby paths.** From the paths perspective, we can

interpret function approximation as a way of merging nearby paths. But

what do we mean by "nearby"? In the figure above, we made an implicit

decision to measure "nearby" with Euclidean distance. This was a good

idea because the Euclidean distance between two states is highly

correlated with the probability that the agent will transition between

them.

 However, it's easy to imagine cases where this implicit assumption

 breaks down. By adding a single long barrier, we can construct a case

where the Euclidean distance metric leads to bad generalization. The

problem is that we have merged the wrong paths.

 #fnapprox-barrier .wrapper {

 margin-left: auto;

 margin-right: auto;

 display: grid;

 grid-template-columns: 1fr 1.464fr;

 grid-template-rows: auto auto;

 grid-column-gap: 20px;

 grid-row-gap: 10px;

 grid-auto-flow: column;

 }

 Imagine changing the Cliff World setup by adding a long barrier.

 Now, using the Euclidean averager leads to bad value updates.

**Merging the wrong paths.** The diagram below shows the

effects of merging the wrong paths a bit more explicitly. Since the

Euclidean averager is to blame for poor generalization, both Monte Carlo

 and TD make bad value updates. However, TD learning amplifies these

errors dramatically whereas Monte Carlo does not.

 We've seen that TD learning makes more efficient value updates.

The price we pay is that these updates end up being much more sensitive

to bad generalization.

Implications for deep reinforcement learning

--------------------------------------------

**Neural networks.** Deep neural networks are perhaps the most

popular function approximators for reinforcement learning. These models

are exciting for many reasons, but one particularly nice property is

that they don't make implicit assumptions about which states are

"nearby."

 Early in training, neural networks, like averagers, tend to merge

the wrong paths of experience. In the Cliff Walking example, an

untrained neural network might make the same bad value updates as the

Euclidean averager.

 But as training progresses, neural networks can actually learn to

overcome these errors. They learn which states are "nearby" from

experience. In the Cliff World example, we might expect a fully-trained

 the barrier. This isn't something that most other function

approximators can do. It's one of the reasons deep RL is so interesting!

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/latent-distance.png)

 The agent, which was trained to grasp objects using the robotic arm,

takes into account obstacles and arm length when it measures the

distance between two states.

**TD or not TD?** So far, we've seen how TD learning can

outperform Monte Carlo by merging paths of experience where they

intersect. We've also seen that merging paths is a double-edged sword:

when function approximation causes bad value updates, TD can end up

doing worse.

 Over the last few decades, most work in RL has preferred TD

learning to Monte Carlo. Indeed, many approaches to RL use TD-style

value updates. With that being said, there are many other ways to use

Monte Carlo for reinforcement learning. Our discussion centers around

Monte Carlo for value estimation in this article, but it can also be

used for policy selection as in Silver et al.

 Since Monte Carlo and TD learning both have desirable properties,

why not try building a value estimator that is a mixture of the two?

 coefficient constant as they train a deep RL model. However, if we

think Monte Carlo learning is best early in training (before the agent

has learned a good state representation) and TD learning is best later

on (when it's easier to benefit from merging paths), maybe the best

approach is to anneal λ\lambdaλ over the course of training..

Conclusion

----------

 In this article we introduced a new way to think about TD

learning. It helps us see why TD learning can be beneficial, why it can

be effective for off-policy learning, and why there can be challenges in

 combining TD learning with function approximators.

#### Gridworld playground

##### Learning Algorithm

Monte CarloSarsaExpected SarsaQ-Learning##### Visualization

PolicyQ(s,a)V(s)##### Epsilon-greedy policy

exploreexploitAdd an agent+2-1

Reinforcement Learning: An Introduction

Double Q-learning

Universal Planning Networks

Mastering the game of go without human knowledge


The Building Blocks of Interpretability

Chris Olah
Arvind Satyanarayan
Ian Johnson
Shan Carter
Ludwig Schubert
Katherine Ye
Alexander Mordvintsev

Abstract: With the growing success of neural networks, there is a  corresponding need to be able to explain their decisions — including  building confidence about how they will behave in the real-world,  detecting model bias, and for scientific curiosity.      In order to do so, we need to both construct deep abstractions and  reify (or instantiate) them in rich interfaces .      With a few exceptions , existing work on interpretability fails to do these in concert.

 With the growing success of neural networks, there is a

corresponding need to be able to explain their decisions — including

building confidence about how they will behave in the real-world,

detecting model bias, and for scientific curiosity.

 In order to do so, we need to both construct deep abstractions and

reify (or instantiate) them in rich interfaces .

 for reasoning about neural networks.

 However, these techniques have been studied as isolated threads of

research, and the corresponding work of reifying them has been

neglected.

 On the other hand, the human-computer interaction community has

begun to explore rich user interfaces for neural networks ,

 but they have not yet engaged deeply with these abstractions.

 To the extent these abstractions have been used, it has been in

fairly standard ways.

 As a result, we have been left with impoverished interfaces (e.g.,

saliency maps or correlating abstract neurons) that leave a lot of value

 on the table.

 Worse, many interpretability techniques have not been fully

actualized into abstractions because there has not been pressure to make

 them generalizable or composable.

 In this article, we treat existing interpretability methods as

fundamental and composable building blocks for rich user interfaces.

 We find that these disparate techniques now come together in a

unified grammar, fulfilling complementary roles in the resulting

interfaces.

 Moreover, this grammar allows us to systematically explore the space

 of interpretability interfaces, enabling us to evaluate whether they

meet particular goals.

 For example, we will see how a network looking at a labrador

retriever detects floppy ears and how that influences its

classification.

 Our interfaces are speculative and one might wonder how reliable they are.

 actively investigating why this is, and hope to uncover principles for

designing interpretable models. In the meantime, while we demonstrate

our techniques on GoogLeNet, we provide code for you to try them on

other models.

 Although here we've made a specific choice of task and network, the

basic abstractions and patterns for combining them that we present can

be applied to neural networks in other domains.

Making Sense of Hidden Layers

-----------------------------

 Much of the recent work on interpretability is concerned with a

neural network's input and output layers.

 Arguably, this focus is due to the clear meaning these layers have:

in computer vision, the input layer represents values for the red,

green, and blue color channels for every pixel in the input image, while

 the output layer consists of class labels and their associated

probabilities.

 However, the power of neural networks lies in their hidden

layers — at every layer, the network discovers a new representation of

the input.

 In computer vision, we use neural networks that run the same feature

 detectors at every position in the image.

 We can think of each layer's learned representation as a

![dog_cat](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

a4,1 = [0, 0, 0, 25.2, 164.1, 0, 42.7, 4.51, 115.0, 51.3, 0, 0, ...]

{

:

886.

,

:

599.

,

:

328.

,

:

303.

,

... }

There seem to be detectors for

 floppy ears,

 dog snouts,

 cat heads,

 furry legs,

 and

 grass.

[Reproduce in a

 To make a semantic dictionary, we pair every neuron activation with a

 visualization of that neuron and sort them by the magnitude of the

activation.

 This marriage of activations and feature visualization changes our

relationship with the underlying mathematical object.

 Activations now map to iconic representations, instead of abstract

indices, with many appearing to be similar to salient human ideas, such

as "floppy ear," "dog snout," or "fur."

 Semantic dictionaries are powerful not just because they move away

from meaningless indices, but because they express a neural network's

learned abstractions with canonical examples.

 With image classification, the neural network learns a set of visual

 abstractions and thus images are the most natural symbols to represent

them.

 Were we working with audio, the more natural symbols would most

likely be audio clips.

 This is important because when neurons appear to correspond to human

 ideas, it is tempting to reduce them to words.

 Doing so, however, is a lossy operation — even for familiar

abstractions, the network may have learned a deeper nuance.

 For instance, GoogLeNet has multiple floppy ear detectors that

appear to detect slightly different levels of droopiness, length, and

surrounding context to the ears.

 There also may exist abstractions which are visually familiar, yet

that we lack good natural language descriptions for: for example, take

the particular column of shimmering light where sun hits rippling water.

 Moreover, the network may learn new abstractions that appear alien

to us — here, natural language would fail us entirely!

 In general, canonical examples are a more natural way to represent

the foreign abstractions that neural networks learn than native human

language.

 By bringing meaning to hidden layers, semantic dictionaries set the

stage for our existing interpretability techniques to be composable

building blocks.

 As we shall see, just like their underlying vectors, we can apply

dimensionality reduction to them.

 In other cases, semantic dictionaries allow us to push these

techniques further.

 For example, besides the one-way attribution that we currently

perform with the input and output layers, semantic dictionaries allow us

 to attribute to-and-from specific hidden layers.

 In principle, this work could have been done without semantic

dictionaries but it would have been unclear what the results meant.

 While we introduce semantic dictionaries in terms of neurons, they

can be used with any basis of activations. We will explore

 this more later.

What Does the Network See?

--------------------------

![dog_cat](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

Activation Vector

=

886.

+

599.

+

328.

+

303.

+

...

Channels

 Applying this technique to all the activation vectors allows us to

not only see what the network detects at each position, but also what

the network understands of the input image as a whole.

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d.jpeg)

[Reproduce in a

mixed4d

 And, by working across layers (eg. "mixed3a", "mixed4d"), we can

observe how the network's understanding evolves: from detecting edges in

 earlier layers, to more sophisticated shapes and object parts in the

latter.

Zoom In

Zoom Out

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed3a.jpeg)

#### mixed3a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4a.jpeg)

#### mixed4a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d.jpeg)

#### mixed4d

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed5a.jpeg)

#### mixed5a

 These visualizations, however, omit a crucial piece of information:

the magnitude of the activations.

 By scaling the area of each cell by the magnitude of the activation

vector, we can indicate how strongly the network detected features at

that position:

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed3a-magnitude.png)

#### mixed3a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4a-magnitude.png)

#### mixed4a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d-magnitude.png)

#### mixed4d

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed5a-magnitude.png)

#### mixed5a

How Are Concepts Assembled?

---------------------------

 We think there's a lot of important research to be done on

attribution methods, but for the purposes of this article the exact

approach taken to attribution doesn't matter.

 We use a fairly simple method, linearly approximating the

relationshipWe do attribution by linear

approximation in all of our interfaces. That is, we estimate the effect

of a neuron on the output is its activation times the rate at which

increasing its activation increases the output. When we talk about a

linear combination of activations, the attribution can be thought of as

the linear combination of the attributions of the units, or equivalently

 as the dot product between the activation of that combination and the

gradient.

 For spatial attribution, we do an additional trick.

GoogLeNet's strided max pooling introduces a lot of noise and

checkerboard patterns to it's gradients.

 To avoid our interface demonstrations being dominated by this noise, we

 (a) do a relaxation of the gradient of max pooling, distributing

gradient to inputs proportional to their activation instead of winner

takes all and (b) cancel out the checkerboard patterns.

### Spatial Attribution with Saliency Maps

 We see two weaknesses with this current approach.

 First, it is not clear that individual pixels should be the primary

unit of attribution.

 The meaning of each pixel is extremely entangled with other pixels,

is not robust to simple visual transforms (e.g., brightness, contrast,

etc.), and is far-removed from high-level concepts like the output

class.

 Second, traditional saliency maps are a very limited type of

interface — they only display the attribution for a single class at a

time, and do not allow you to probe into individual points more deeply.

 As they do not explicitly deal with hidden layers, it has been

difficult to fully explore their design space.

 We instead treat attribution as another user interface building

block, and apply it to the hidden layers of a neural network.

 In doing so, we change the questions we can pose.

 Rather than asking whether the color of a particular pixel was

important for the "labrador retriever" classification, we instead ask

 This approach is similar to what Class Activation Mapping (CAM) methods

 do but, because they interpret their results back onto the input image,

 they miss the opportunity to communicate in terms of the rich behavior

of a network's hidden layers.

#### Input Image

![](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

#### Output Classes

* Labrador retriever

* golden retriever

* tennis ball

* Rhodesian ridgeback

* Appenzeller

#### Output Factors

* Labrador retriever

* golden retriever

* beagle

* kuvasz

* redbone

* tiger

* tiger cat

* lynx

* collie

* Border collie

#### mixed3a

#### mixed4a

#### mixed4d

#### mixed5a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed3a-nmf.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed3a-magnitude.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4a-nmf.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4a-magnitude.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d-nmf.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d-magnitude.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed5a-nmf.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed5a-magnitude.png)

Attribution tends to be more meaningful in later layers.

 The

 floppy ear,

 dog snout,

 cat head,

 etc, do mostly what you expect.

 Surprisingly, the

 lower snout at mixed4d

 seems entangled with the idea of a tennis ball and

 supports "tennis ball" and "granny smith apple."

[Reproduce in a

 The above interface affords us a more flexible relationship with

attribution.

 To start, we perform attribution from each spatial position of each

hidden layer shown to all 1,000 output classes.

 In order to visualize this thousand-dimensional vector, we use

dimensionality reduction to produce a multi-directional saliency map.

 Overlaying these saliency maps on our magnitude-sized activation

grids provides an information scent

 over attribution space.

 The activation grids allow us to anchor attribution to the visual

vocabulary our semantic dictionaries first established.

 On hover, we update the legend to depict attribution to the output

classes (i.e., which classes does this spatial position most contribute

to?).

 On hover, additional saliency maps mask the hidden layers, in a

sense shining a light into their black boxes.

 This type of layer-to-layer attribution is a prime example of how

carefully considering interface design drives the generalization of our

existing abstractions for interpretability.

 With this diagram, we have begun to think of attribution in terms of

 higher-level concepts.

 However, at a particular position, many concepts are being detected

together and this interface makes it difficult to split them apart.

 By continuing to focus on spatial positions, these concepts remain

entangled.

### Channel Attribution

 An alternate way to slice the cube is by channels instead of spatial locations.

#### Input Image

![](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

#### Output Classes

* Labrador retriever

* golden retriever

* tennis ball

* Rhodesian ridgeback

* Appenzeller

#### Top Channels Supporting

 Labrador retriever

mixed3b

...

Showing 3 of **480**mixed4a

...

Showing 3 of **508**mixed4b

...

Showing 3 of **512**mixed4c

...

Showing 3 of **512**mixed4d

...

Showing 3 of **528**

 This diagram is analogous to the previous one we saw: we conduct

layer-to-layer attribution but this time over channels rather than

spatial positions.

 Once again, we use the icons from our semantic dictionary to

represent the channels that most contribute to the final output

classification.

 Hovering over an individual channel displays a heatmap of its

activations overlaid on the input image.

 The legend also updates to show its attribution to the output

classes (i.e., what are the top classes this channel supports?).

 Clicking a channel allows us to drill into the layer-to-layer

attributions, identifying the channels at lower layers that most

contributed as well as the channels at higher layers that are most

supported.

 Attribution to spatial locations and channels can reveal powerful

things about a model, especially when we combine them together.

 Unfortunately, this family of approaches is burdened by two

significant problems.

 On the one hand, it is very easy to end up with an overwhelming

amount of information: it would take hours of human auditing to

understand the long-tail of channels that slightly impact the output.

 On the other hand, both the aggregations we have explored are

extremely lossy and can miss important parts of the story.

 And, while we could avoid lossy aggregation by working with

individual neurons, and not aggregating at all, this explodes the first

problem combinatorially.

Making Things Human-Scale

-------------------------

 In previous sections, we've considered three ways of slicing the

cube of activations: into spatial activations, channels, and individual

neurons.

 Each of these has major downsides.

 If one only uses spatial activations or channels, they miss out on

very important parts of the story.

 For example it's interesting that the floppy ear detector helped us

classify an image as a Labrador retriever, but it's much more

interesting when that's combined with the locations that fired to do so.

 One can try to drill down to the level of neurons to tell the whole

story, but the tens of thousands of neurons are simply too much

information.

 Even the hundreds of channels, before being split into individual

neurons, can be overwhelming to show users!

 If we want to make useful interfaces into neural networks, it isn't

enough to make things meaningful.

 We need to make them human scale, rather than overwhelming dumps of

information.

 The key to doing so is finding more meaningful ways of breaking up

our activations.

 There is good reason to believe that such decompositions exist.

 Often, many channels or spatial positions will work together in a

highly correlated way and are most useful to think of as one unit.

 Other channels or positions will have very little activity, and can

be ignored for a high-level overview.

 So, it seems like we ought to be able to find better decompositions

if we had the right tools.

 There is an entire field of research, called matrix factorization,

that studies optimal strategies for breaking up matrices.

 By flattening our cube into a matrix of spatial locations and

channels, we can apply these techniques to get more meaningful groups of

 neurons.

 These groups will not align as naturally with the cube as the

groupings we previously looked at.

 Instead, they will be combinations of spatial locations and

channels.

 Moreover, these groups are constructed to explain the behavior of a

network on a particular image.

 It would not be effective to reuse the same groupings on another

image; each image requires calculating a unique set of groups.

 In addition to naturally slicing a hidden layer's cube of

activations into neurons, spatial locations, or channels, we can also

consider more arbitrary groupings of locations and channels.

 The groups that come out of this factorization will be the atoms of

the interface a user works with. Unfortunately, any grouping is

inherently a tradeoff between reducing things to human scale and,

because any aggregation is lossy, preserving information. Matrix

factorization lets us pick what our groupings are optimized for, giving

us a better tradeoff than the natural groupings we saw earlier.

 The goals of our user interface should influence what we optimize

our matrix factorization to prioritize. For example, if we want to

prioritize what the network detected, we would want the factorization to

 fully describe the activations. If we instead wanted to prioritize what

 would change the network's behavior, we would want the factorization to

 fully describe the gradient. Finally, if we want to prioritize what

caused the present behavior, we would want the factorization to fully

describe the attributions. Of course, we can strike a balance between

these three objectives rather than optimizing one to the exclusion of

the others.

 Most matrix factorization

algorithms and libraries are set up to minimize the mean squared error

of the reconstruction of a matrix you give them. There are ways to hack

such libraries to achieve more general objectives through clever

manipulations of the provided matrix, as we will see below. More

broadly, matrix factorization is an optimization problem, and with

custom tools you can achieve all sorts of custom factorizations.

 with non-negative matrix factorization

 As the name suggests, non-negative

 matrix factorization (NMF) constrains its factors to be positive. This

is fine for the activations of a ReLU network, which must be positive as

 well. Our experience is that the groups we get from NMF seem more

independent and semantically meaningful than those without this

constraint. Because of this constraints, groups from NMF are a less

efficient at representing the activations than they would be without,

but our experience is that they seem more independent and semantically

meaningful.

  .

 Notice how the overwhelmingly large number of neurons has been

reduced to a small set of groups, concisely summarizing the story of the

 neural network.

By

 using non-negative matrix factorization we can reduce the large number

of neurons to a small set of groups that concisely summarize the story

of the network.

[Reproduce in a

#### Input image

#### Activations

 of neuron groups

#### neuron groups based on matrix factorization of mixed4d layer

6 groups

color key

 hover to isolate

#### Effect of neuron groups on output classes

Labrador retrieverbeagletiger catlynxtennis ball

2.249

3.298

-0.350

0.111

0.920

3.755

0.599

-0.994

-0.642

1.336

-1.193

-0.110

-1.607

-0.057

0.152

-1.141

-0.356

0.116

0.117

-0.885

1.117

-0.133

0.248

1.120

1.227

-1.892

-2.618

0.205

0.152

-0.480

 This figure only focuses at a single layer but, as we saw earlier,

it can be useful to look across multiple layers to understand how a

neural network assembles together lower-level detectors into

higher-level concepts.

 The groups we constructed before were optimized to understand a

single layer independent of the others. To understand multiple layers

together, we would like each layer's factorization to be

"compatible" — to have the groups of earlier layers naturally compose

into the groups of later layers. This is also something we can optimize

the factorization for

 We formalize this "compatibility" in a manner described below,

although we're not confident it's the best formalization and won't be

surprised if it is superseded in future work.

 The basic idea is to split each entry in the activation matrix into *N*

 entries on the channel dimension, spreading the values proportional to

the absolute value of its attribution to the corresponding group.

 Any factorization of this matrix induces a factorization of the

original matrix by collapsing the duplicated entries in the column

factors.

 However, the resulting factorization tries to create separate

factors when the activation of the same channel has different

attributions in different places.

  .

#### Input Image

![](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

To understand multiple layers together, we would like

each layer's factorization to be "compatible"—to have the groups of

 earlier layers naturally compose into the groups of later

layers. This is also something we can optimize the factorization

 for.

  positive influence

 negative influence

#### Attribution by Factorized Groups

##### Mixed4a

8 groups

##### Mixed4d

6 groups

##### Output class

 Align layer factors

tiger catbeagleLabrador retrieverlynxtennis ball

 In this section, we recognize that the way in which we break apart

the cube of activations is an important interface decision. Rather than

resigning ourselves to the natural slices of the cube of activations, we

 construct more optimal groupings of neurons. These improved groupings

are both more meaningful and more human-scale, making it less tedious

for users to understand the behavior of the network.

 Our visualizations have only begun to explore the potential of

alternate bases in providing better atoms for understanding neural

networks.

 For example, while we focus on creating smaller numbers of

directions to explain individual examples, there's recently been

The Space of Interpretability Interfaces

----------------------------------------

We can think of an interface as a union of individual elements.

#### Layers

* output

* hidden

* input

#### Atoms

* group

* spatial

* channel

* neuron

#### Content

* activations

* attribution

#### Presentation

* information visualization

* feature visualization

 We think this is a powerful technique and present this grammar as an

initial exploration of how it might apply to interpretability

interfaces.

 `Id = Int

Atom = Neuron | Spatial | Channel | Group | Whole

Layer = Input | Hidden Int | Out

Substrate = Network Id Atom Layer

| Dataset Id

 | Parameters Id

 Content = Substrate

| Activation Substrate

| Attribution Substrate Substrate

| Transform Content *Content?*

Element = InfoVis Content | FeatureVis Content

Interface = [ Element ]`,

 but we find it helpful to think about the space visually.

 We can represent the network's substrate (which layers we display,

and how we break them apart) as a grid, with the content and style of

presentation plotted on this grid as points and connections.

![](The%20Building%20Blocks%20of%20Interpretability_files/empty.svg)

 For instance, let us consider our teaser figure again.

![](The%20Building%20Blocks%20of%20Interpretability_files/teaser.svg)

**1. Feature visualization**

![](The%20Building%20Blocks%20of%20Interpretability_files/teaser-1.png)

**2. Filter by output attribution**

![](The%20Building%20Blocks%20of%20Interpretability_files/teaser-2.png)

 Next, we filter for specific classes by calculating the output attribution.

**3. Drill down on hover**

![](The%20Building%20Blocks%20of%20Interpretability_files/teaser-3.png)

 Hovering over channels, we get a heatmap of spatial activations.

 figure .teaser-thumb {

 width: 100%;

 border: 30px solid rgb(248, 248, 248);

 border-radius: 10px;

 box-sizing: border-box;

 margin: 12px 0 10px;

 }

 In this article, we have only scratched the surface of

possibilities.

 There are lots of combinations of our building blocks left to

explore, and the design space gives us a way to do so systematically.

 Moreover, each building block represents a broad class of

techniques.

 Our interfaces take only one approach but, as we saw in each

section, there are a number of alternatives for feature visualization,

attribution, and matrix factorization.

 An immediate next step would be to try using these alternate

techniques, and research ways to improve them.

 We can think of dataset examples as another substrate in our design

space, thus becoming another building block that fully composes with the

 others.

 In doing so, we can now imagine interfaces that not only allow us to

 inspect the influence of dataset examples on the final output

classification (as Koh & Liang proposed), but also how examples

influence the features of hidden layers, and how they influence the

relationship between these features and the output.

 For example, if we consider our "Labrador retriever" image, we can

not only see which dataset examples most influenced the model to arrive

at this classification, but also which dataset examples most caused the

"floppy ear" detectors to fire, and which dataset examples most caused

these detectors to increase the "Labrador retriever" classification.

![](The%20Building%20Blocks%20of%20Interpretability_files/dataset.svg)

 A new substrate.

 An interface showing how examples influence the channels of hidden layers.

 An interface for identifying which dataset examples most caused

particular detectors to increase the output classification.

 While most models today are trained to optimize simple objective

functions that one can easily describe, many of the things we'd like

models to do in the real world are subtle, nuanced, and hard to describe

 mathematically.

 An extreme example of the subtle

objective problem is something like "creating interesting art", but much

 more mundane examples arise more or less whenever humans are involved.

 However, even with human feedback, it may still be hard to train

models to behave the way we want if the problematic aspect of the model

doesn't surface strongly in the training regime where humans are giving

feedback.

 There are lots of reasons why problematic behavior may not surface

 or may be hard for an evaluator to give feedback on.

 For example, discrimination and bias may be subtly present

throughout the model's behavior, such that it's hard for a human

evaluator to critique.

 Or the model may be making a decision in a way that has

problematic consequences, but those consequences never play out in the

problems we're training it on.

 Human feedback on the model's decision making process, facilitated

by interpretability interfaces, could be a powerful solution to these

problems.

 (There is however a danger here: we are optimizing our model to look

 the way we want in our interface — if we aren't careful, this may lead

 Another exciting possibility is interfaces for comparing multiple

models.

 For instance, we might want to see how a model evolves during

training, or how it changes when you transfer it to a new task.

 Or, we might want to understand how a whole family of models

compares to each other.

 Existing work has primarily focused on comparing the output behavior

 One of the unique challenges of this work is that we may want to

align the atoms of each model; if we have completely different models,

can we find the most analogous neurons between them?

 Zooming out, can we develop interfaces that allow us to evaluate

large spaces of models at once?

How Trustworthy Are These Interfaces?

-------------------------------------

 In order for interpretability interfaces to be effective, we must

trust the story they are telling us.

 We perceive two concerns with the set of building blocks we

currently use.

 First, do neurons have a relatively consistent meaning across

different input images, and is that meaning accurately reified by

feature visualization?

 Semantic dictionaries, and the interfaces that build on top of them,

 are premised off this question being true.

 Second, does attribution make sense and do we trust any of the

attribution methods we presently have?

 validated this in a number of ways: we visualized them without a

generative model prior, so that the content of the visualizations

 was causally linked to the neuron firing; we inspected the spectrum

of examples that cause the neuron to fire; and used diversity

 visualizations to try to create different inputs that cause the

neuron to fire.

For more details, see

 Besides these neurons, however, we also found many neurons that do

not have as clean a meaning including "poly-semantic" neurons that

respond to a mixture of salient ideas (e.g., "cat" and "car").

 There are natural ways that interfaces could respond to this: we

could use diversity visualizations to reveal the variety of meanings the

 neuron can take, or rotate our semantic dictionaries so their

components are more disentangled.

 Of course, just like our models can be fooled, the features that

make them up can be too — including with adversarial examples .

 In fact, it can be interesting to identify when a detector misfires.

 One might even wonder if the idea is fundamentally flawed, since a

function's output could be the result of non-linear interactions between

 its inputs.

 One way these interactions can pan out is as attribution being

"path-dependent".

 A natural response to this would be for interfaces to explicitly

surface this information: how path-dependent is the attribution?

 A deeper concern, however, would be whether this path-dependency

dominates the attribution.

 Clearly, this is not a concern for attribution between adjacent

layers because of the simple (essentially linear) mapping between them.

 While there may be technicalities about correlated inputs, we

believe that attribution is on firm grounding here.

 And even with layers further apart, our experience has been that

attribution between high-level features at the output is much more

consistent than attribution to the input — we believe that

path-dependence is not a dominating concern here.

 Model behavior is extremely complex, and our current building blocks

 force us to show only specific aspects of it.

 An important direction for future interpretability research will be

developing techniques that achieve broader coverage of model behavior.

 But, even with such improvements, we anticipate that a key marker of

 trustworthiness will be interfaces that do not mislead.

 Interacting with the explicit information displayed should not cause

 users to implicitly draw incorrect assessments about the model (we see a

 similar principle articulated by Mackinlay for data visualization).

 Undoubtedly, the interfaces we present in this article have room to

improve in this regard.

 Fundamental research, at the intersection of machine learning and

human-computer interaction, is necessary to resolve these issues.

 Trusting our interfaces is essential for many of the ways we want to

 use interpretability.

 This is both because the stakes can be high (as in safety and

fairness) and also because ideas like training models with

interpretability feedback put our interpretability techniques in the

middle of an adversarial setting.

Conclusion & Future Work

------------------------

 There is a rich design space for interacting with enumerative

algorithms, and we believe an equally rich space exists for interacting

with neural networks.

 We have a lot of work left ahead of us to build powerful and

trustworthy interfaces for interpretability.

 But, if we succeed, interpretability promises to be a powerful tool

in enabling meaningful human oversight and in building fair, safe, and

aligned AI systems.

Thought as a Technology

Visualizing Representations: Deep Learning and Human Beings

Understanding neural networks through deep visualization

Using Artificial Intelligence to Augment Human Intelligence

Visualizing higher-layer features of a deep network

Feature Visualization

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Visualizing and understanding convolutional networks

Striving for simplicity: The all convolutional net

Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization

Interpretable Explanations of Black Boxes by Meaningful Perturbation

PatternNet and PatternLRP--Improving the interpretability of neural networks

The (Un)reliability of saliency methods

Axiomatic attribution for deep networks

Visualizing data using t-SNE

LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent neural networks

ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models

Do convolutional neural networks learn class hierarchy?

Going deeper with convolutions

Deconvolution and Checkerboard Artifacts

Learning deep features for discriminative localization

Information foraging

TCAV: Relative concept importance testing with Linear Concept Activation Vectors

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks

Understanding Black-box Predictions via Influence Functions

Deep reinforcement learning from human preferences

Interactive optimization for steering machine classification

Interacting with predictions: Visual inspection of black-box machine learning models

Modeltracker: Redesigning performance analysis tools for machine learning

Network dissection: Quantifying interpretability of deep visual representations

Intriguing properties of neural networks

Efficient estimation of word representations in vector space

Unsupervised representation learning with deep convolutional generative adversarial networks

Adversarial manipulation of deep representations

Automating the design of graphical presentations of relational information


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Examples are Just Bugs, Too

Abstract: This article is part of a discussion of the Ilyas et al. paper                 “Adversarial examples are not bugs, they are features”.                 You can learn more in the                                      main discussion article                 .

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 color: hsl(0, 0%, 0.33);

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *"Adversarial examples are not bugs, they are features".*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[All Responses](https://distill.pub/2019/advex-bugs-discussion/#articles)

 We demonstrate that there exist adversarial examples which are just "bugs":

1. Do not transfer between models, and

2. Do not leak "non-robust features" which allow for learning, in the

 sense of Ilyas-Santurkar-Tsipras-Engstrom-Tran-Madry

 .

 We replicate the Ilyas et al.

 experiment of training on mislabeled adversarially-perturbed images

 (Section 3.2 of ),

 and show that it fails for our construction of adversarial perturbations.

 The message is, whether adversarial examples are features or bugs depends

 classifier.

 intrinsically have any vulnerable directions.

### Background

 Many have understood Ilyas et al.

 to claim that adversarial examples are not "bugs", but are "features".

 Specifically, Ilyas et al. postulate the following two worlds:

 As communicated to us by the original authors.

 In this world, adversarial examples occur because classifiers behave

 poorly off-distribution,

 when they are evaluated on inputs that are not natural images.

 Here, adversarial examples would occur in arbitrary directions,

 having nothing to do with the true data distribution.

 and which contain features of the target class.

 For example, consider the perturbation that

 makes an image of a dog to be classified as a cat.

 vs. dogs.

 are in both.

 Ilyas et al.

 show that there exist adversarial examples in World 2, and we show there exist

 examples in World 1.

 Constructing Non-transferrable Targeted Adversarial Examples

--------------------------------------------------------------

 fff,

 which do not transfer to other classifiers trained for the same problem.

 f(x′)=ytargf(x′)=ytargf(x') = y\_{targ}.

 PGD is described in the appendix.

 which starts at input xxx, and iteratively takes steps {xt}{xt}\{x\_t\}

 to minimize the loss L(f,xt,ytarg)L(f,xt,ytarg)L(f, x\_t, y\_{targ}).

 That is, we take steps in the direction

 −∇xL(f,xt,ytarg)−∇xL(f,xt,ytarg)-
abla\_x L(f, x\_t, y\_{targ})

 where L(f,x,y)L(f,x,y)L(f, x, y) is the loss of fff on input xxx, label yyy.

 Note that since PGD steps in the gradient direction towards the target class,

 background).

 since the "blue" direction is correlated with the plane class.

 In our construction below, we attempt to eliminate such feature leakage.

 and an off-manifold "random component".

 Our construction below attempts to step only in the off-manifold direction.

### Our Construction

 for the same classification problem as fff.

 different random initializations.

 For input example (x,y)(x,y)(x, y) and target class ytargytargy\_{targ},

 we perform iterative updates to find adversarial attacks — as in PGD.

 However, instead of stepping directly in the gradient direction, we

 step in the direction

 Formally, we replace the iterative step with

 where ΠεΠε\Pi\_\eps is the projection onto the εε\eps-ball around xxx.

 we minimize the "disentangled loss"

 We could also consider explicitly using the ensemble to decorrelate,

 by stepping in direction

 This works well for small ϵϵ\epsilon,

 but the given loss has better optimization properties for larger ϵϵ\epsilon.

 This loss encourages finding an xtxtx\_t which is adversarial for fff,

 but not for the ensemble {fi}{fi}\{f\_i\}.

 these examples are also not adversarial for

 *new* classifiers trained for the same problem.

### Experiments

 We train a ResNet18 on CIFAR10 as our target classifier fff.

 We then test the probability that

 a targeted attack for fff

 transfers to a new (freshly-trained) ResNet18, with the same targeted class.

 For L∞L∞L\_{\infty} attacks:

|  |  |  |

| --- | --- | --- |

|  | Attack Success | Transfer Success |

| PGD | 99.6% | 52.1% |

| Ours | 98.6% | 0.8% |

 For L2L2L\_2 attacks:

|  |  |  |

| --- | --- | --- |

|  | Attack Success | Transfer Success |

| PGD | 99.9% | 82.5% |

| Ours | 99.3% | 1.7% |

Adversarial Examples With No Features

-------------------------------------

 Using the above, we can construct adversarial examples

 which *do not suffice* for learning.

 Here, we replicate the Ilyas et al. experiment

 that "Non-robust features suffice for standard classification"

 (Section 3.2 of ),

 but show that it fails for our construction of adversarial examples.

 To review, the Ilyas et al. non-robust experiment was:

1. Train a standard classifier fff for CIFAR.

2. From the CIFAR10 training set S={(Xi,Yi)}S={(Xi,Yi)}S = \{(X\_i, Y\_i)\},

 Note that S′S′S' appears to humans as "mislabeled examples".

3. Train a new classifier f′f′f' on train set S′S′S'.

 Ilyas et al. use Step (3) to argue that

 adversarial examples have a meaningful "feature" component.

 However, for adversarial examples constructed using our method, Step (3) fails.

 (X,Y+1)(X,Y+1)(X, Y+1), which is intuitively what we trained on.

For L∞L∞L\_{\infty} attacks:

|  |  |  |

| --- | --- | --- |

| PGD | 23.7% | 40.4% |

| Ours | 2.5% | 75.9% |

 Table: Test Accuracies of f′f′f'

For L2L2L\_2 attacks:

|  |  |  |

| --- | --- | --- |

| PGD | 33.2% | 27.3% |

| Ours | 2.8% | 70.8% |

 Table: Test Accuracies of f′f′f'

Adversarial Squares: Adversarial Examples from Robust Features

--------------------------------------------------------------

 To further illustrate that adversarial examples can be "just bugs",

 reasonable

 *intrinsic* definition, this problem has no non-robust features.

 In particular, by the Ilyas et al. definition, **every** distribution

 distribution.

 family of classifiers being considered.

 overfitting, and

 label noise.

 with a small amount of random pixel noise and label noise.

 A sample of images from the distribution.

 Formally, let the distribution be as follows.

 Pick label Y∈{±1}Y∈{±1}Y \in \{\pm 1\} uniformly,

 egin{cases}

 (+ec{\mathbb{1}} + ec\eta\_\eps) \cdot \eta & 	ext{if $Y=1$}\

 (-ec{\mathbb{1}} + ec\eta\_\eps) \cdot \eta & 	ext{if $Y=-1$}\

 \end{cases}

 and

 A plot of samples from a 2D-version of this distribution is shown to the right.

 However, if we sample 10000 training images from this distribution, and train

 a ResNet18 to 99.9% train accuracy,

 the resulting classifier is highly non-robust:

 The input-noise and label noise are both essential for this construction.

 One intuition for what is happening is: in the initial stage of training

 classifier).

 to fit the label-noise, which hurts robustness.

 Figure adapted from .

Addendum: Data Poisoning via Adversarial Examples

-------------------------------------------------

 As an addendum, we observe that the "non-robust features"

 experiment of  (Section 3.2)

 directly implies data-poisoning attacks:

 to the classifier output labels (e.g. swapping cats and dogs).

 notation, and also using vanilla PGD to find adversarial examples.

 on distribution (X,Y)(X,Y)(X, Y).

 By permutation-symmetry of the labels, this implies that:

 on distribution (X,Y−1)(X,Y−1)(X, Y-1).

 but the classifier learns to predict the cyclically-shifted labels.

 Concretely, using the original numbers of

 Table 1 in , this reduction implies that

 and cause the learnt classifier to output shifted-labels

 43.7% of the time

 (cats classified as birds, dogs as deers, etc).**

 To cite Ilyas et al.'s response, please cite their

**Response Summary**: We note that as discussed in

 mere existence of adversarial

 examples

 that are "features" is sufficient to corroborate our main thesis. This comment

 illustrates, however, that we can indeed craft adversarial examples that are

 based on "bugs" in realistic settings. Interestingly, such examples don't

 transfer, which provides further support for the link between transferability

 and non-robust features.

 we did not intend to claim

 that adversarial examples arise *exclusively* from (useful) features but rather

 that useful non-robust features exist and are thus (at least

 partially) responsible for adversarial vulnerability. In fact,

 prior work already shows how in theory adversarial examples can arise from

 insufficient samples  or finite-sample overfitting

 , and the experiments

 presented here (particularly, the adversarial squares) constitute a neat

 real-world demonstration of these facts.

 Our main thesis that "adversarial examples will not just go away as we fix

 stemming from "bugs." As long as adversarial examples can stem from non-robust

 features (which the commenter seems to agree with), fixing these bugs will not

 solve the problem of adversarial examples.

Moreover, with regards to feature "leakage" from PGD, recall that in

 or D\_det dataset, the non-robust features are associated with the

 correct label whereas the robust features are associated with the wrong

 one. We wanted to emphasize that, as

 [shown in Appendix D.6](https://arxiv.org/abs/1905.02175) ,

 models trained on our DdetDdetD\_{det} dataset actually generalize *better* to

 the non-robust feature-label association that to the robust

 feature-label association. In contrast, if PGD introduced a small

 "leakage" of non-robust features, then we would expect the trained model

 would still predominantly use the robust feature-label association.

 That said, the experiments cleverly zoom in on some more fine-grained

 nuances in our understanding of adversarial examples. One particular thing that

 stood out to us is that by creating a set of adversarial examples that are

 *explicitly* non-transferable, one also prevents new classifiers from learning

 features from that dataset. This finding thus makes the connection between

 transferability of adversarial examples and their containing generalizing

 features even stronger! Indeed, we can add the constructed dataset into our

 "ˆDdetD^det\widehat{\mathcal{D}}\_{det} learnability vs transferability" plot

 (Figure 3 in the paper) — the point

 corresponding to this dataset fits neatly onto the trendline!

 attacks

SGD on Neural Networks Learns Functions of Increasing Complexity

Adversarially Robust Generalization Requires More Data

A boundary tilting persepective on the phenomenon of adversarial examples