A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Examples are Just Bugs, Too

Abstract: This article is part of a discussion of the Ilyas et al. paper                 “Adversarial examples are not bugs, they are features”.                 You can learn more in the                                      main discussion article                 .

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 color: hsl(0, 0%, 0.33);

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[All Responses](https://distill.pub/2019/advex-bugs-discussion/#articles)

 We demonstrate that there exist adversarial examples which are just “bugs”:

1. Do not transfer between models, and

2. Do not leak “non-robust features” which allow for learning, in the

 sense of Ilyas-Santurkar-Tsipras-Engstrom-Tran-Madry

 .

 We replicate the Ilyas et al.

 experiment of training on mislabeled adversarially-perturbed images

 (Section 3.2 of ),

 and show that it fails for our construction of adversarial perturbations.

 The message is, whether adversarial examples are features or bugs depends

 classifier.

 intrinsically have any vulnerable directions.

### Background

 Many have understood Ilyas et al.

 to claim that adversarial examples are not “bugs”, but are “features”.

 Specifically, Ilyas et al. postulate the following two worlds:

 As communicated to us by the original authors.

 In this world, adversarial examples occur because classifiers behave

 poorly off-distribution,

 when they are evaluated on inputs that are not natural images.

 Here, adversarial examples would occur in arbitrary directions,

 having nothing to do with the true data distribution.

 and which contain features of the target class.

 For example, consider the perturbation that

 makes an image of a dog to be classified as a cat.

 vs. dogs.

 are in both.

 Ilyas et al.

 show that there exist adversarial examples in World 2, and we show there exist

 examples in World 1.

 Constructing Non-transferrable Targeted Adversarial Examples

--------------------------------------------------------------

 fff,

 which do not transfer to other classifiers trained for the same problem.

 f(x′)=ytargf(x′)=ytargf(x’) = y\_{targ}.

 PGD is described in the appendix.

 which starts at input xxx, and iteratively takes steps {xt}{xt}\{x\_t\}

 to minimize the loss L(f,xt,ytarg)L(f,xt,ytarg)L(f, x\_t, y\_{targ}).

 That is, we take steps in the direction

 −∇xL(f,xt,ytarg)−∇xL(f,xt,ytarg)-\nabla\_x L(f, x\_t, y\_{targ})

 where L(f,x,y)L(f,x,y)L(f, x, y) is the loss of fff on input xxx, label yyy.

 Note that since PGD steps in the gradient direction towards the target class,

 background).

 since the “blue” direction is correlated with the plane class.

 In our construction below, we attempt to eliminate such feature leakage.

 and an off-manifold “random component”.

 Our construction below attempts to step only in the off-manifold direction.

### Our Construction

 for the same classification problem as fff.

 different random initializations.

 For input example (x,y)(x,y)(x, y) and target class ytargytargy\_{targ},

 we perform iterative updates to find adversarial attacks — as in PGD.

 However, instead of stepping directly in the gradient direction, we

 step in the direction

 Formally, we replace the iterative step with

 where ΠεΠε\Pi\_\eps is the projection onto the εε\eps-ball around xxx.

 we minimize the “disentangled loss”

 We could also consider explicitly using the ensemble to decorrelate,

 by stepping in direction

 This works well for small ϵϵ\epsilon,

 but the given loss has better optimization properties for larger ϵϵ\epsilon.

 This loss encourages finding an xtxtx\_t which is adversarial for fff,

 but not for the ensemble {fi}{fi}\{f\_i\}.

 these examples are also not adversarial for

 *new* classifiers trained for the same problem.

### Experiments

 We train a ResNet18 on CIFAR10 as our target classifier fff.

 We then test the probability that

 a targeted attack for fff

 transfers to a new (freshly-trained) ResNet18, with the same targeted class.

 For L∞L∞L\_{\infty} attacks:

|  |  |  |

| --- | --- | --- |

|  | Attack Success | Transfer Success |

| PGD | 99.6% | 52.1% |

| Ours | 98.6% | 0.8% |

 For L2L2L\_2 attacks:

|  |  |  |

| --- | --- | --- |

|  | Attack Success | Transfer Success |

| PGD | 99.9% | 82.5% |

| Ours | 99.3% | 1.7% |

Adversarial Examples With No Features

-------------------------------------

 Using the above, we can construct adversarial examples

 which *do not suffice* for learning.

 Here, we replicate the Ilyas et al. experiment

 that “Non-robust features suffice for standard classification”

 (Section 3.2 of ),

 but show that it fails for our construction of adversarial examples.

 To review, the Ilyas et al. non-robust experiment was:

1. Train a standard classifier fff for CIFAR.

2. From the CIFAR10 training set S={(Xi,Yi)}S={(Xi,Yi)}S = \{(X\_i, Y\_i)\},

 Note that S′S′S’ appears to humans as “mislabeled examples”.

3. Train a new classifier f′f′f’ on train set S′S′S’.

 Ilyas et al. use Step (3) to argue that

 adversarial examples have a meaningful “feature” component.

 However, for adversarial examples constructed using our method, Step (3) fails.

 (X,Y+1)(X,Y+1)(X, Y+1), which is intuitively what we trained on.

For L∞L∞L\_{\infty} attacks:

|  |  |  |

| --- | --- | --- |

| PGD | 23.7% | 40.4% |

| Ours | 2.5% | 75.9% |

 Table: Test Accuracies of f′f′f’

For L2L2L\_2 attacks:

|  |  |  |

| --- | --- | --- |

| PGD | 33.2% | 27.3% |

| Ours | 2.8% | 70.8% |

 Table: Test Accuracies of f′f′f’

Adversarial Squares: Adversarial Examples from Robust Features

--------------------------------------------------------------

 To further illustrate that adversarial examples can be “just bugs”,

 reasonable

 *intrinsic* definition, this problem has no non-robust features.

 In particular, by the Ilyas et al. definition, **every** distribution

 distribution.

 family of classifiers being considered.

 overfitting, and

 label noise.

 with a small amount of random pixel noise and label noise.

 A sample of images from the distribution.

 Formally, let the distribution be as follows.

 Pick label Y∈{±1}Y∈{±1}Y \in \{\pm 1\} uniformly,

 \begin{cases}

 (+\vec{\mathbb{1}} + \vec\eta\_\eps) \cdot \eta & \text{if $Y=1$}\\

 (-\vec{\mathbb{1}} + \vec\eta\_\eps) \cdot \eta & \text{if $Y=-1$}\\

 \end{cases}

 and

 A plot of samples from a 2D-version of this distribution is shown to the right.

 However, if we sample 10000 training images from this distribution, and train

 a ResNet18 to 99.9% train accuracy,

 the resulting classifier is highly non-robust:

 The input-noise and label noise are both essential for this construction.

 One intuition for what is happening is: in the initial stage of training

 classifier).

 to fit the label-noise, which hurts robustness.

 Figure adapted from .

Addendum: Data Poisoning via Adversarial Examples

-------------------------------------------------

 As an addendum, we observe that the “non-robust features”

 experiment of  (Section 3.2)

 directly implies data-poisoning attacks:

 to the classifier output labels (e.g. swapping cats and dogs).

 notation, and also using vanilla PGD to find adversarial examples.

 on distribution (X,Y)(X,Y)(X, Y).

 By permutation-symmetry of the labels, this implies that:

 on distribution (X,Y−1)(X,Y−1)(X, Y-1).

 but the classifier learns to predict the cyclically-shifted labels.

 Concretely, using the original numbers of

 Table 1 in , this reduction implies that

 and cause the learnt classifier to output shifted-labels

 43.7% of the time

 (cats classified as birds, dogs as deers, etc).**

 To cite Ilyas et al.’s response, please cite their

**Response Summary**: We note that as discussed in

 mere existence of adversarial

 examples

 that are “features” is sufficient to corroborate our main thesis. This comment

 illustrates, however, that we can indeed craft adversarial examples that are

 based on “bugs” in realistic settings. Interestingly, such examples don’t

 transfer, which provides further support for the link between transferability

 and non-robust features.

 we did not intend to claim

 that adversarial examples arise *exclusively* from (useful) features but rather

 that useful non-robust features exist and are thus (at least

 partially) responsible for adversarial vulnerability. In fact,

 prior work already shows how in theory adversarial examples can arise from

 insufficient samples  or finite-sample overfitting

 , and the experiments

 presented here (particularly, the adversarial squares) constitute a neat

 real-world demonstration of these facts.

 Our main thesis that “adversarial examples will not just go away as we fix

 stemming from “bugs.” As long as adversarial examples can stem from non-robust

 features (which the commenter seems to agree with), fixing these bugs will not

 solve the problem of adversarial examples.

Moreover, with regards to feature “leakage” from PGD, recall that in

 or D\_det dataset, the non-robust features are associated with the

 correct label whereas the robust features are associated with the wrong

 one. We wanted to emphasize that, as

 [shown in Appendix D.6](https://arxiv.org/abs/1905.02175) ,

 models trained on our DdetDdetD\_{det} dataset actually generalize *better* to

 the non-robust feature-label association that to the robust

 feature-label association. In contrast, if PGD introduced a small

 “leakage” of non-robust features, then we would expect the trained model

 would still predominantly use the robust feature-label association.

 That said, the experiments cleverly zoom in on some more fine-grained

 nuances in our understanding of adversarial examples. One particular thing that

 stood out to us is that by creating a set of adversarial examples that are

 *explicitly* non-transferable, one also prevents new classifiers from learning

 features from that dataset. This finding thus makes the connection between

 transferability of adversarial examples and their containing generalizing

 features even stronger! Indeed, we can add the constructed dataset into our

 “ˆDdetD^det\widehat{\mathcal{D}}\_{det} learnability vs transferability” plot

 (Figure 3 in the paper) — the point

 corresponding to this dataset fits neatly onto the trendline!

 attacks

SGD on Neural Networks Learns Functions of Increasing Complexity

Adversarially Robust Generalization Requires More Data

A boundary tilting persepective on the phenomenon of adversarial examples

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'_ Adversarial Examples are Just Bugs, Too.html


The Paths Perspective on Value Learning

Abstract: In the last few years, reinforcement learning (RL) has made remarkable progress, including beating world-champion Go players, controlling robotic hands, and even painting pictures.

Introduction

------------

 One of the key sub-problems of RL is value estimation – learning the

 long-term consequences of being in a state.

 This can be tricky because future returns are generally noisy,

affected by many things other than the present state. The further we

look into the future, the more this becomes true.

 But while difficult, estimating value is also essential to many

approaches to RL.For many approaches

(policy-value iteration), estimating value essentially is the whole

problem, while in other approaches (actor-critic models), value

estimation is essential for reducing noise.

 The natural way to estimate the value of a state is as the average

return you observe from that state. We call this Monte Carlo value

estimation.

**Cliff World**

 is a classic RL example, where the agent learns to

 walk along a cliff to reach a goal.

 Sometimes the agent reaches its goal.

 Other times it falls off the cliff.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/cliffworld-mc.svg)

 Monte Carlo averages over trajectories where they intersect.

 If a state is visited by only one episode, Monte Carlo says its

value is the return of that episode. If multiple episodes visit a state,

 Monte Carlo estimates its value as the average over them.

 Let’s write Monte Carlo a bit more formally.

 In tabular settings such as the Cliff World example, this “update

 and we could just as easily use the “+=” notation. But when using

parameteric function approximators such as neural networks, our “update

towards” operator may represent a gradient step, which cannot be written

 in “+=” notation. In order to keep our notation clean and general, we

chose to use the ↩\hookleftarrow↩ operator throughout.

V(st)  V(s\_t)~~V(st)

↩  \hookleftarrow~~↩

RtR\_tRt

 State value

 Return

 The term on the right is called the return and we use it to measure

the amount of long-term reward an agent earns. The return is just a

 is a discount factor which controls how much short term rewards are

worth relative to long-term rewards. Estimating value by updating

Beating Monte Carlo

-------------------

V(st)  V(s\_t)~~V(st)

↩  \hookleftarrow~~↩

rtr\_{t} rt

+++

γV(st+1)\gamma V(s\_{t+1})γV(st+1)

 State value

Reward

Next state value

 Intersections between two trajectories are handled differently

under this update. Unlike Monte Carlo, TD updates merge intersections so

 that the return flows backwards to all preceding states.

 Sometimes the agent reaches its goal.

 Other times it falls off the cliff.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/cliffworld-td.svg)

 TD learning merges paths where they intersect.

 What does it mean to “merge trajectories” in a more formal sense?

V(st+1)  V(s\_{t+1})~~V(st+1)

≃  \simeq~~≃

≃  \simeq~~≃

 Now we can use this equation to expand the TD update rule recursively:

V(st) V(s\_t)~V(st)

↩ \hookleftarrow~↩

rtr\_{t} rt

+++

γV(st+1)\gamma V(s\_{t+1})γV(st+1)

↩ \hookleftarrow~↩

rtr\_{t} rt

+++

γE[rt+1′]\gamma \mathop{\mathbb{E}} \bigr[ r’\_{t+1} \bigl]γE[rt+1′]

+++

↩ \hookleftarrow~↩

rtr\_{t} rt

+++

γE [rt+1′]\gamma \mathop{\mathbb{E}} ~ \bigr[ r’\_{t+1} \bigl]γE [rt+1′]

+++

+ +~+

...  …~~...

 This gives us a strange-looking sum of nested expectation values.

At first glance, it’s not clear how to compare them with the more

simple-looking Monte Carlo update. More importantly, it’s not clear that

 we *should* compare the two; the updates are so different that it

feels a bit like comparing apples to oranges. Indeed, it’s easy to think

 of Monte Carlo and TD learning as two entirely different approaches.

 But they are not so different after all. Let’s rewrite the Monte

Carlo update in terms of reward and place it beside the expanded TD

update.

**MC update**

V(st) V(s\_t)~V(st)

 ↩  ~\hookleftarrow~~ ↩

rtr\_{t}rt

+ +~+

γ rt+1\gamma ~ r\_{t+1}γ rt+1

+ +~+

γ2 rt+2\gamma^2 ~ r\_{t+2}γ2 rt+2

+ +~+

...…...

Reward from present path.

Reward from present path.

Reward from present path…

**TD update**

V(st) V(s\_t)~V(st)

 ↩  ~\hookleftarrow~~ ↩

rtr\_{t}rt

+ +~+

γE [rt+1′]\gamma \mathop{\mathbb{E}} ~ \bigr[ r’\_{t+1} \bigl]γE [rt+1′]

+ +~+

+ +~+

...…...

Reward from present path.

Expectation over paths intersecting present path.

Expectation over paths intersecting *paths intersecting* present path…

 A pleasant correspondence has emerged. The difference between

Monte Carlo and TD learning comes down to the nested expectation

operators. It turns out that there is a nice visual interpretation for

what they are doing. We call it the *paths perspective* on value learning.

The Paths Perspective

---------------------

 Trajectory 1

 Trajectory 2

 But this way of organizing experience de-emphasizes relationships *between*

 trajectories. Wherever two trajectories intersect, both outcomes are

valid futures for the agent. So even if the agent has followed

Trajectory 1 to the intersection, it could *in theory* follow

Trajectory 2 from that point onward. We can dramatically expand the

agent’s experience using these simulated trajectories or “paths.”

 #cliffworld-paths {

 display: grid;

 grid-template-columns: repeat(auto-fit, minmax(130px, 1fr));

 grid-gap: 20px;

 }

 Path 1

 Path 2

 Path 3

 Path 4

**Estimating value.** It turns out that Monte Carlo is

averaging over real trajectories whereas TD learning is averaging over

all possible paths. The nested expectation values we saw earlier

correspond to the agent averaging across *all possible future paths*.

 #compare-mctd {

 display: grid;

 grid-gap: 40px;

 grid-template-columns: repeat(auto-fit, minmax(320px, 1fr));

 }

 #compare-mctd .subfigure {

 display: grid;

 grid-gap: 20px;

 grid-template-columns: repeat(2, minmax(160px, 1fr));

 /\* grid-auto-rows: min-content; \*/

 /\* grid-template-rows: min-content auto; \*/

 }

 #compare-mctd .subfigure .column-heading {

 grid-column: 1 / -1;

 }

 #compare-mctd .figcaption {

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 padding-top: 5px;

 margin-top: 5px;

 /\* min-height: 179px; \*/

 /\* min-width: 179px; \*/

 /\* flex: 1; \*/

 }

#### Monte Carlo Estimation

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/traj-thumbnails.svg)

 Averages over **real trajectories**

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/cliffworld-mc.svg)

 Resulting MC estimate

#### Temporal Difference Estimation

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/path-thumbnails.svg)

 Averages over **possible paths**

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/cliffworld-td.svg)

 Resulting TD estimate

**Comparing the two.** Generally speaking, the best value

estimate is the one with the lowest variance. Since tabular TD and Monte

 Carlo are empirical averages, the method that gives the better estimate

 is the one that averages over more items. This raises a natural

question: Which estimator averages over more items?

Var[V(s)]  Var[V(s)]~~Var[V(s)]

∝  \propto~~∝

1N\frac{1}{N} N1

 Variance of estimate

Inverse of the number of items in the average

 First off, TD learning never averages over fewer trajectories than

 Monte Carlo because there are never fewer simulated trajectories than

real trajectories. On the other hand, when there are *more*

simulated trajectories, TD learning has the chance to average over more

of the agent’s experience.

 This line of reasoning suggests that TD learning is the better

estimator and helps explain why TD tends to outperform Monte Carlo in

tabular environments.

Introducing Q-functions

-----------------------

 An alternative to the value function is the Q-function. Instead of

 estimating the value of a state, it estimates the value of a state and

an action. The most obvious reason to use Q-functions is that they allow

 us to compare different actions.

 #qlearning-intro {

 display: grid;

 grid-template-columns: repeat(auto-fit, minmax(130px, 1fr));

 grid-column-gap: 30px;

 }

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/policy.svg)

 Many times we’d like to compare the value of actions under a policy.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/value.svg)

 It’s hard to do this with a value function.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/qvalue.svg)

 It’s easier to use Q-functions, which estimate joint state-action values.

 There are some other nice properties of Q-functions. In order to

see them, let’s write out the Monte Carlo and TD update rules.

Q(st,at)  Q(s\_t, a\_t)~~Q(st,at)

↩  \hookleftarrow~~↩

RtR\_tRt

 State-action value

 Return

 We still update towards the return. Instead of updating towards

the return of being in some state, though, we update towards the return

of being in some state *and* selecting some action.

 Now let’s try doing the same thing with the TD update:

Q(st,at)  Q(s\_t, a\_t)~~Q(st,at)

↩  \hookleftarrow~~↩

rtr\_{t} rt

+++

γQ(st+1,at+1)\gamma Q(s\_{t+1}, a\_{t+1})γQ(st+1,at+1)

 State-action value

Reward

Next state value

 What we need is a better estimate of V(st+1)V(s\_{t+1})V(st+1).

Q(st,at)  Q(s\_t, a\_t)~~Q(st,at)

↩  \hookleftarrow~~↩

rtr\_{t} rt

+++

γV(st+1)\gamma V(s\_{t+1})γV(st+1)

 State-action value

Reward

Next state value

V(st+1)  V(s\_{t+1})~~V(st+1)

=  =~~=

?? ?

Learning Q-functions with reweighted paths

------------------------------------------

**Expected Sarsa.**

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/sarsa.svg)

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/expected-sarsa.svg)

 than a value estimate computed straight from the experience. This is

because the expectation value weights the Q-values by the true policy

distribution rather than the empirical policy distribution. In doing

**Off-policy value learning.** We can push this idea even

further. Instead of weighting Q-values by the true policy distribution,

we can weight them by an arbitrary policy, πoff\pi^{off}πoff:

**Off-policy value learning** weights Q-values by an arbitrary policy.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/off-policy.svg)

 This slight modification lets us estimate value under any policy

we like. It’s interesting to think about Expected Sarsa as a special

case of off-policy learning that’s used for on-policy estimation.

**Re-weighting path intersections.** What does the paths

perspective say about off-policy learning? To answer this question,

let’s consider some state where multiple paths of experience intersect.

 #reweighting {

 display: grid;

 grid-template-columns: repeat(auto-fit, minmax(160px, 1fr));

 grid-gap: 30px;

 }

 /\* #reweighting .wrapper {

 min-width: 160px;

 } \*/

 #reweighting-full .cls-1 {

 fill: #2d307b;

 }

 #reweighting-full .cls-2 {

 fill: #e7ebe8;

 }

 #reweighting-full .cls-3 {

 fill: #cac9cc;

 }

 #reweighting-full .cls-4 {

 fill: #bd5f35;

 }

 #reweighting-full .cls-5, .cls-6, .cls-7 {

 fill: none;

 stroke-width: 10px;

 }

 #reweighting-full .cls-5 {

 stroke: #bd5f35;

 }

 #reweighting-full .cls-5, .cls-7 {

 stroke-miterlimit: 10;

 }

 #reweighting-full .cls-6 {

 stroke: #2d307b;

 stroke-linecap: round;

 stroke-linejoin: round;

 }

 #reweighting-full .cls-7 {

 stroke: #8191c9;

 }

 #reweighting-full .cls-8 {

 fill: #8191c9;

 }

 #reweighting-full .cls-9 {

 font-size: 24.25341px;

 fill: #d1d3d4;

 font-family: Arial-BoldMT, Arial;

 font-weight: 700;

 }

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/reweighting-1.svg)

 Multiple paths of experience intersect at this state.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/reweighting-2.svg)

+2

-1

Weight of upward path: 0.50

 Wherever intersecting paths are re-weighted, the paths that are

most representative of the off-policy distribution end up making larger

contributions to the value estimate. Meanwhile, paths that have low

probability make smaller contributions.

**Q-learning.** There are many cases where an agent needs to

collect experience under a sub-optimal policy (e.g. to improve

exploration) while estimating value under an optimal one. In these

cases, we use a version of off-policy learning called Q-learning.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/q-learning.svg)

 Q-learning prunes away all but the highest-valued paths. The paths

that remain are the paths that the agent will follow at test time; they

are the only ones it needs to pay attention to. This sort of value

**Double Q-Learning.** The problem with Q-learning is that it

gives biased value estimates. More specifically, it is over-optimistic

in the presence of noisy rewards. Here’s an example where Q-learning

fails:

*You go to a casino and play a hundred slot machines. It’s your

lucky day: you hit the jackpot on machine 43. Now, if you use Q-learning

 to estimate the value of being in the casino, you will choose the best

outcome over the actions of playing slot machines. You’ll end up

thinking that the value of the casino is the value of the jackpot…and

decide that the casino is a great place to be!*

 Sometimes the largest Q-value of a state is large *just by chance*;

 choosing it over others makes the value estimate biased.

 One way to reduce this bias is to have a friend visit the casino

and play the same set of slot machines. Then, ask them what their

winnings were at machine 43 and use their response as your value

estimate. It’s not likely that you both won the jackpot on the same

machine, so this time you won’t end up with an over-optimistic estimate.

 We call this approach *Double Q-learning*.

**Putting it together.**  It’s easy to think of Sarsa, Expected

 Sarsa, Q-learning, and Double Q-learning as different algorithms. But

#### On-policy methods

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/sarsa.svg)

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/expected-sarsa.svg)

#### Off-policy methods

**Off-policy value learning** weights Q-values by an arbitrary policy.

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/off-policy.svg)

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/q-learning.svg)

 QBQ\_BQB.

**Re-weighting paths with Monte Carlo.** At this point, a

natural question is: Could we accomplish the same re-weighting effect

with Monte Carlo? We could, but it would be messier and involve

re-weighting all of the agent’s experience. By working at intersections,

 TD learning re-weights individual transitions instead of episodes as a

whole. This makes TD methods much more convenient for off-policy

learning.

Merging Paths with Function Approximators

-----------------------------------------

 Up until now, we’ve learned one parameter — the value

estimate — for every state or every state-action pair. This works well

for the Cliff World example because it has a small number of states. But

 most interesting RL problems have a large or infinite number of states.

 This makes it hard to store value estimates for each state.

 #figure-fnapprox-intro {

 display: grid;

 grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));

 grid-column-gap: 20px;

 grid-row-gap: 10px;

 /\* grid-auto-flow: column; \*/

 }

 in these spaces often requires function approximation.

 memory and don’t generalize.

 generalize to states they haven’t visited yet.

 Instead, we must force our value estimator to have fewer

parameters than there are states. We can do this with machine learning

methods such as linear regression, decision trees, or neural networks.

All of these methods fall under the umbrella of function approximation.

**Merging nearby paths.** From the paths perspective, we can

interpret function approximation as a way of merging nearby paths. But

what do we mean by “nearby”? In the figure above, we made an implicit

decision to measure “nearby” with Euclidean distance. This was a good

idea because the Euclidean distance between two states is highly

correlated with the probability that the agent will transition between

them.

 However, it’s easy to imagine cases where this implicit assumption

 breaks down. By adding a single long barrier, we can construct a case

where the Euclidean distance metric leads to bad generalization. The

problem is that we have merged the wrong paths.

 #fnapprox-barrier .wrapper {

 margin-left: auto;

 margin-right: auto;

 display: grid;

 grid-template-columns: 1fr 1.464fr;

 grid-template-rows: auto auto;

 grid-column-gap: 20px;

 grid-row-gap: 10px;

 grid-auto-flow: column;

 }

 Imagine changing the Cliff World setup by adding a long barrier.

 Now, using the Euclidean averager leads to bad value updates.

**Merging the wrong paths.** The diagram below shows the

effects of merging the wrong paths a bit more explicitly. Since the

Euclidean averager is to blame for poor generalization, both Monte Carlo

 and TD make bad value updates. However, TD learning amplifies these

errors dramatically whereas Monte Carlo does not.

 We’ve seen that TD learning makes more efficient value updates.

The price we pay is that these updates end up being much more sensitive

to bad generalization.

Implications for deep reinforcement learning

--------------------------------------------

**Neural networks.** Deep neural networks are perhaps the most

popular function approximators for reinforcement learning. These models

are exciting for many reasons, but one particularly nice property is

that they don’t make implicit assumptions about which states are

“nearby.”

 Early in training, neural networks, like averagers, tend to merge

the wrong paths of experience. In the Cliff Walking example, an

untrained neural network might make the same bad value updates as the

Euclidean averager.

 But as training progresses, neural networks can actually learn to

overcome these errors. They learn which states are “nearby” from

experience. In the Cliff World example, we might expect a fully-trained

 the barrier. This isn’t something that most other function

approximators can do. It’s one of the reasons deep RL is so interesting!

![](The%20Paths%20Perspective%20on%20Value%20Learning_files/latent-distance.png)

 The agent, which was trained to grasp objects using the robotic arm,

takes into account obstacles and arm length when it measures the

distance between two states.

**TD or not TD?** So far, we’ve seen how TD learning can

outperform Monte Carlo by merging paths of experience where they

intersect. We’ve also seen that merging paths is a double-edged sword:

when function approximation causes bad value updates, TD can end up

doing worse.

 Over the last few decades, most work in RL has preferred TD

learning to Monte Carlo. Indeed, many approaches to RL use TD-style

value updates. With that being said, there are many other ways to use

Monte Carlo for reinforcement learning. Our discussion centers around

Monte Carlo for value estimation in this article, but it can also be

used for policy selection as in Silver et al.

 Since Monte Carlo and TD learning both have desirable properties,

why not try building a value estimator that is a mixture of the two?

 coefficient constant as they train a deep RL model. However, if we

think Monte Carlo learning is best early in training (before the agent

has learned a good state representation) and TD learning is best later

on (when it’s easier to benefit from merging paths), maybe the best

approach is to anneal λ\lambdaλ over the course of training..

Conclusion

----------

 In this article we introduced a new way to think about TD

learning. It helps us see why TD learning can be beneficial, why it can

be effective for off-policy learning, and why there can be challenges in

 combining TD learning with function approximators.

#### Gridworld playground

##### Learning Algorithm

Monte CarloSarsaExpected SarsaQ-Learning##### Visualization

PolicyQ(s,a)V(s)##### Epsilon-greedy policy

exploreexploitAdd an agent+2-1

Reinforcement Learning: An Introduction

Double Q-learning

Universal Planning Networks

Mastering the game of go without human knowledge

The Paths Perspective on Value Learning.html


Curve Circuits

Nick Cammarata
Gabriel Goh
Shan Carter
Chelsea Voss
Ludwig Schubert
Chris Olah

Abstract: This article is part of the Circuits thread,  an experimental format collecting invited short articles and critical  commentary delving into the inner workings of neural networks.

### Contents

* [3a Line Neuron Family](#line-families)

* [Cosmetic Neurons](#cosmetic-neurons)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

 takes the opposite approach: building up from individual neurons and

individual weights to easily verifiable explanations of tiny slices of

neural networks. But it faces a different question: can we ever hope to

understand a full neural network this way?

 We find that although curve detection involves more than 50,000

parameters, those parameters actually implement a simple algorithm that

can be read off the weights and described in just a few English

sentences. Based on this understanding, we re-implement curve detectors

 Rotational equivariance reduces the complexity by a factor of 10-20x

while scale equivariance reduces it by an additional 2-3x, for a total

reduction of ~50xA small portion involving

 the interactions of color-contrast detectors with line detectors is

further reduced by a factor of ~8x due to hue equivariance, but doesn't

significantly reduce the complexity of the overall circuit..

 We find this an exciting example of how motifs can dramatically

simplify circuits. In this circuit the main motif is equivariance, but

in others different motifs may be more important.

While the curve

circuit in InceptionV1 is quaint next to the 175b weights in GPT-3, it's

 We think the surprising simplicity of the curve circuit is a glimmer of

 hope that the Circuits approach may scale to letting us

reverse-engineer big neural networks into small verifiable explanations.

---

The Curve Detector Algorithm

----------------------------

 Decomposed feature visualization renders a grid of feature

visualization that shows the expanded weights of an upstream layer to a

downstream neuron of interest (in this case 3b:379)Specifically,

 we calculate the gradient from each layer to the 3b curve family using a

 linear approximation of InceptionV1 where ReLUs are replaced with the

identity. Then we use feature visualization with the objective of

maximizing the dot product between each image and each spatial position

of the gradient.. Intuitively, this is like decomposing the

 curve into simpler shapes, with simpler shapes the further back we go,

from early curves, to lines, and eventually Gabor filters. Each shape is

 oriented along the tangent of the curve in 3b:379.

conv2d0Gabor

 Filters line the tangent with secondary influence from the Color

 tangent is lined primarily by Complex Gabor neurons, which resemble

Gabor Filters but are invariant to the exact positions of which side is

 tangent of 3a is primarily lined by the Early Curve family as well as a

 second Line family. We’ll explore this layer in depth throughout this

 use decomposed feature visualization to show how the first four layers

of InceptionV1 incrementally build towards curve detectors in 3b. To

increase legibility, we render each feature visualization with alpha

transparency and grayscale. Since curve detectors are invariant to

color, the vibrant colors feature visualization produces don't give us

 To give us a sense of the strength of each position we set the opacity

relative to its magnitude. We see the highest magnitude weights follow

the tangent of 3b:379.We can take the

highest magnitude position from each layer above and see which neuron

 doing this decomposition, it really matters which metric you choose to

measure how neurons contribute to the vector, such as average or total

weight, since there is a long tail of neurons with low magnitude that

collectively make up a large fraction of the total weight. Training

models with weight sparsity would reduce this long tail..

 decompose the highest magnitude spatial position in each layer into

neuron families to see which families contribute most to curve

 each family we can also visualize how they connect to each other,

showing us a birds-eye view of the curve detection algorithm. Again, we

see the layers progressively building more complex shapes, with shapes

that closely resemble curve detectors like early curves and well-defined

 lines being built in 3a, only one layer before curve detectors.

![](Curve%20Circuits_files/figure_003.png)We

 can visualize how neuron families connect to see a high-level view of

the curve detection algorithm of InceptionV1. In addition to giving us a

 sense of how these families are built, it lets us see how information

flows to 3b curves. For instance, we see that conv2d2 lines contribute

to 3b curves through both 3a lines and 3a early curves. While it may

seem like this throws away lots of information, in a sense this is a

natural way to understand a circuit. It’s much easier to think about a

line family in conv2d2 than 33 individual line neurons that vary in

orientation.Now that we know which neuron families

 are most important for curve detection, we can invest in understanding

the weights connecting them. Luckily this is sometimes easy, since

 Many families in this diagram implement the rotational equivariance

motif, meaning that each neuron in a family is approximately a rotated

version of another neuron in that family. We can see this in the weights

 connecting 3a early curves and 3b curves.

 weights connecting 3a early curves with 3b:379 implement rotational

equivariance. It turns out there are curves of curves literally

inscribed in the weights of neural networks. We’ll discuss equivariance

later in this paper, and likely again in the Circuits thread in a

dedicated paper.When neuron families implement

rotational equivariance we learn a lot looking at the strongest positive

 and negative neuron connections, because the others are just rotated

versions of them. In the weights we see a general pattern, as each layer

 builds a contrast detector by tiling a simpler contrast detector along

its tangent. When the two shapes are aligned the weight is most

positive, and most negative when they are perpendicular.

![](Curve%20Circuits_files/figure_002.png)Some

 of the strongest positive and negative weights between neuron families

in the curve circuit. We see two patterns. First, the most positive

weights tend to be where shapes are aligned, and the most negative are

where shapes are in opposing orientations. Secondly, we see the same

transitions can occur multiple times across different layers, such as

the transition from lines to curves.So far we’ve

looked at the neuron families that are most important to curve

detection, and also at the circuit motifs connecting them. This

high-level “circuit schematic” is useful for seeing a complex algorithm

at a glance, and it tells us the main components we'd need to build if

we wanted to reimplement the circuit from scratch.

The circuit

schematic also makes it easy to describe a few sentence English story of

 how curve detection works. Gabor filters turn into proto-lines which

build lines and early curves. Finally, lines and early curves are

composed into curves. In each case, each shape family (eg conv2d2 line)

has positive weight across the tangent of the shape family it builds

(eg. 3a early curve). Each shape family implements the rotational

equivariance motif, containing multiple rotated copies of approximately

the same neuron.

In the next few sections we'll zoom in from this

high level description to a weight-level analysis of how each of the 3a

families we've looked at so far contribute to curve detectors.To

 read more about the idea and benefits of fluidly navigating different

---

The Components of Curve Detectors

---------------------------------

### 3a Early Curve Family

Early

 curves in 3a contribute more to the construction of curve detectors

than any other neuron family. At every position curve neurons are

excited by early curves in a similar orientation and inhibited by ones

in the opposing orientation, closely following the general pattern we

saw earlier.

 every position curve neurons are excited by early curves in a similar

orientation and inhibited by ones in the opposing orientation. If you

look closely you can see the weights shift slightly over the course of

the curve to track the change in local curvature.

 are a couple other subtle things you can find if you look closely.

Firstly, you can see that the weights shift slightly over the course of

the curve, to track the change in local curvature. Secondly, curve

detectors in the same orientation are slightly inhibitory when they’re

located off the curve. This makes sense, as seeing an early curve

located anywhere but the tangent indicates a different shape than a pure

 curve.The weight matrix connecting early curves

and curves shows a striking set of positive weights lining the diagonal

where the two shapes have similar orientations. This band of positive

weights is surrounded by negative weights where the early curve and

curve orientations differ. The transition is smooth — if a curve

strongly wants to see an early curve it wants to see its neighbor a bit

too.

 do we see that curve detectors which are rotated 180 degrees from each

other are inhibitory, but not ones rotated 90 degrees? Recall from our

previous article that early curve detectors respond slightly to curves

that are 180 degrees rotated from their prefered orientation, which we

called an echo. This makes sense: a curve in the opposite orientation

runs tangent to the curve we’re trying to detect in the middle of the

receptive field, causing similar edge detectors to fire.

These

negative weights are the reason curve neurons in 3b have no echoes,

which we can validate by using circuit editing to remove them and

confirming that echoes appear.

 curve family of 3b uses negative weights to avoid the problem of

echoes. By removing these negative weights with circuit editing we see

echoes reappear.Using negative weights in this way

 follows a more general trend: our experience is that negative weights

are used for things that are in some way similar enough that they could

### 3a Line Families

 Since they have similar roles in the context of curve detectors we’ll

discuss them together, while pointing out their subtle differences.

Like

 early curves, lines align to the tangent of curve detectors, with more

positive weights when the neurons have the same orientation. However,

this pattern is more nuanced and discontinuous with lines than early

curves. A line with a similar orientation to a curve will excite it, but

 a line that's rotated a little may inhibit it. This makes it hard to

see a general pattern by looking directly at the weight matrix.

Instead,

 we can study which line orientations excite curve detectors using

synthetic stimuli. We can take a similar approach to decomposed feature

visualization to see which lines different spatial positions respond to.

 This shows us that each 3b curve neuron is excited by edges along its

tangent line with a tolerance of between about 10° to 45°.

 tolerance isn’t always symmetric, which we can see in 3b:406 below. On

its left side it is most excited by lines about 10° upwards. If the line

 is oriented above 10° it is still excited, but if it is less than 10°

it switches to being inhibited.

 view tells us how each curve detector responds to different

orientations of lines in general. We can connect this back to individual

 line neurons by studying which orientations those line neurons respond

#### Lines

There

 are 11 simple line neurons in 3a that mostly fire to one orientation,

although some activate more weakly to an "echo" 90 degrees away.

Five

 neurons in 3a respond to lines that are perpendicular to the

orientation where they are longest. These neurons mostly detect fur, but

 they also contribute to 3b curves.

Finally,

 there are five line neurons with curiously sharp transitions. These

lines want an orientation facing a particular direction, and tolerate

error in that direction, but definitely don't want it going the other

way, even if slightly. In curve detection, this is useful for handling

imperfections, like bumps.

 find cliff-like line neurons an interesting example of non-linear

behavior. We usually think of neurons as measuring distance from some

ideal. For instance, we may expect car neurons to prefer Or

 we could imagine a more sophisticated metric, such as style. Since

ImageNet contains classes for sports cars and non-sports cars, there may

 be neurons that measure the “sportiness" of different cars. We think

that studying neurons that correspond to cars is an interesting research

 direction, since it’s easy to accumulate datasets of cars with labelled

 properties such as year, country of origin, and price.

cars of a certain size, and have weaker activations to cars that are

bigger or smaller than its ideal. But this line family provides a

counter-example, accepting error on one side while not firing at all on

the other.

#### Lines in conv2d2

The

 different types of line neurons we looked at above each have different

behaviors, which is part of why the weight matrix between 3a lines and

3b curves is indecipherable. However, if we go back one more layer and

look at how conv2d2 lines connect to 3b curves, we see structure.

 think this points to an interesting property of both curves and lines

in InceptionV1. The line family in conv2d2 are roughly "pure" line

detectors, detecting lines in an elegant pattern. The next layer (3a)

builds lines too, but they're more applied and nuanced, with

seemingly-awkward behavior like cliff-lines where the network finds them

 useful. Similarly, the 3b curve detectors are surprisingly elegant

detectors of curves behaviorally, and mostly follow clean patterns in

their construction. In contrast, the curves in the next layer (4a) are

 and seemingly specialized for detecting real-world objects like the

tops of cups and pans. Perhaps this points to a yet unnamed motif of

pure shape detectors directly followed by applied ones.

### Cosmetic Neurons

In

 Curve Detectors we saw how curve neurons seem to be robust to several

cosmetic properties, with similar behavior across textures like metal

and wood in a variety of lighting. How do they do it?

![](Curve%20Circuits_files/datasetExamples.png)We

 believe this reflects a more widespread phenomenon in early vision. As

progressively sophisticated shapes are built in each layer, new shapes

incorporate cosmetic neuron families like colors and texture. For

instance, 3b curve neurons are built primarily from the line and early

curve neuron families, but they also incorporate a family of 65 texture

neurons. This means they both inherit the cosmetic robustness of the

line and early curve neurons, as well as strengthen it by including more

 textures.

While we won't do a detailed weight-level analysis of

how cosmetic robustness propagates through the shapes of early vision in

 this article, which is a broader topic than curve detection, we think

this is an exciting direction for future research.

---

An Artificial Artificial Neural Network

---------------------------------------

How

 do we know this story about the mechanics of curve detectors is true?

One way is to use it to reimplement curve detectors from scratch. We

manually set the weights of a blank neural network to implement the

 This was initially a few hour process for one person (Chris Olah), and

they did not look at the original neural network’s weights when

constructing it, which would go against the spirit of the exercise.

Later, before publishing they tweaked the weights and in particular

added negative weights to remove echoes in the activations.

To

compare our artificial curve detectors against InceptionV1's naturally

 available to us. We'll choose three: feature visualization, dataset

examples, and synthetic stimuli. From there we'll run two additional

comparisons by leveraging circuits and model editing.

First we'll

look at feature visualization and responses to synthetic curve stimuli

together. We see the feature visualizations indeed render curves, except

 they are grayscale since we never include cosmetic features such as

color-contrast detectors in our artificial curves. We see their response

 to curve stimuli approximates the natural curve detectors across a

range of radii and all orientations. One difference is our artificial

 can compare the curve detectors in a neural network we hand-crafted

with the curve detectors in InceptionV1 by measuring how they activate

to synthetic curve stimuli. We see that across a range of radii and

orientations, our artificial curve neurons approximate the natural ones.We

 can also get a qualitative sense for the differences by looking at a

saliency map of dataset examples that cause artificial curve detectors

to fire stronglySince our artificial curve

 detectors don't respond to color, there are likely many false-negatives

 where natural curve detectors correctly catch that our artificial ones

don't. Our goal in building artificial curve detectors isn't perfection,

 but to show that we can use the circuit patterns described in this

article to roughly approximate the curve detection algorithm in

InceptionV1..

 we can compare the weights for the circuits connecting neuron families

in the two models, alongside feature visualizations for each of those

families. We see they follow approximately the same circuit structure.

 comparison of the most important family in each layer across our

artificial artificial neural network with InceptionV1 trained on

ImageNet.We can also zoom into specific weight

matrices that we've already studied in this article. We see the raw

weights between early curves and curves as well as lines to curves look

approximately like the ones in InceptionV1, but cleaner since we set

them programmatically.

 weights of the artificial artificial neural network closely resemble

the equivariant weight in the naturally trained network, except their

pattern is cleaner because the weights are programmetically generated.

equivariance.Finally, a preliminary experiment

suggests that adding artificial curve detectors helps recover some of

the loss in classification accuracy across the dataset of removing them

from the model entirelySpecifically, when

we remove InceptionV1's 3b curve detectors entirely, the model's top-5

accuracy across 2000 validation set images drops from 88.6% to 86.3% (46

 fewer correct classifications). When we replace 3b curve detectors with

 our artificial curve detectors more than half of this drop is

recovered, with an accuracy of 87.6% (reducing the gap from 46 incorrect

 classifications to 20). We suspect the remaining gap is due to three

factors. First, our artificial neurons are grayscale, since we are not

implementing color contrast neurons or texture neurons. Secondly, the

receptive fields of artificial neurons may not be exactly centered the

same as natural curve detectors. Thirdly, we may not have scaled the

activations of curve detectors optimally.

Additionally, there are

 two caveats worth mentioning about our experimental setup. First, our

ImageNet evaluation likely doesn't mimic the exact conditions the model

was trained under (eg. data preprocessing), since the original model was

 trained at Google using a precursor Tensorflow. Secondly, the reason we

 ran it on less than the full validation set was operational, not a

result of cherry-picking. We initially ran it on a small set in a

prototype experiment to validate our hypothesis. We planned to run it on

 the full set before our publication date, but due OpenAI infrastructure

 changes our setup broke and we were unable to reimplement it in time.

For this reason we emphasize that the experiment is preliminary,

although we suspect it's likely to work on the full validation set as

well..

Overall, we believe these five experiments

show our artificial curve detectors are roughly analogous to the

naturally trained ones. Since they are nearly a direct translation from

the neuron families and circuit motifs we've described in this article

into Python code for setting weights, we think this is strong evidence

these patterns accurately reflect the underlying circuits that construct

 curve detectors.

---

Downstream

----------

While

this article focused mostly on how curve detection works upstream of the

 curve detector, it's also worth briefly considering how curves are used

 downstream in the model. It's easiest to see their mark on the next

layer, 4a, where they're used to construct more sophisticated shapes.

Many

 of these shapes look for individual curves at different spatial

positions, such as circles and evolutes. These shapes often reappear

across different branches of 4a, such as the 3x3 and 5x5 branch.

![](Curve%20Circuits_files/figure.png)Layer

 4a also constructs a series of curve detectors, mostly in the 5x5

branch that specializes in 3d geometry. However, we believe they should

be thought of less as pure abstract shapes and more as corresponding to

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Two Examples of Useful, Non-Robust Features

Abstract: This article is part of a discussion of the Ilyas et al. paper         “Adversarial examples are not bugs, they are features”.         You can learn more in the                      main discussion article         .

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

[Comment by Ilyas et al.](#rebuttal)

 Ilyas et al.  define a *feature* as a function fff that

 label. But in the presence of an adversary Ilyas et al.  argues

 the metric that truly matters is a feature’s *robust usefulness*,

E[inf∥δ∥≤ϵyf(x+δ)],

 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)\right],

 E[∥δ∥≤ϵinfyf(x+δ)],

 its correlation with the label while under attack. Ilyas et al.

 like?

### Non-Robust Features in Linear Models

 nonlinear models encountered in deep learning. As Ilyas et al

 to linear features of the form:

 \text{and} \quad \mathbf{E}[x] = 0.

 f(x)=∥a∥ΣaTxwhereΣ=E[xxT]andE[x]=0.

 The robust usefulness of a linear feature admits an elegant decomposition

  This

 \begin{aligned}

\mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)\right] &

=\mathbf{E}\left[yf(x)+\inf\_{\|\delta\|\leq\epsilon}yf(\delta)\right]\\

 &

 &

 \end{aligned}

 into two terms:

 .undomargin {

 position: relative;

 left: -1em;

 top: 0.2em;

 }

E[inf∥δ∥≤ϵyf(x+δ)]

 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)\right]

 E[∥δ∥≤ϵinfyf(x+δ)]

===

E[yf(x)]\mathop{\mathbf{E}[yf(x)]}E[yf(x)]

−-−

ϵ∥a∥∗∥a∥Σ\epsilon\frac{\|a\|\_{\*}}{\|a\|\_{\Sigma}}ϵ∥a∥Σ∥a∥∗

 The robust usefulness of a feature

 the correlation of the feature with the label

 the feature’s non-robustness

 dimensional plot.

 Subject to an

 L\_2

 adversery, observe that high frequency features are both less useful and

 less robust.

  Useful Non-Useful A  B  C  D  E  F

 Pareto frontier of points in the non-robustness and usefulness space.

 positive label.

 \log \left( \frac{\|a\_i\|\_\Sigma}{\|a\_i\|} \right) =

 \log(\lambda\_i) Feature’s log robustness. When

 a\_i's

 are the

 i^{th}

 eigenvalues of

 \Sigma

 , the robustness reduces to the

 i^{th}

 singular value of

 \lambda\_i   ABCDEFf-12-11-10-9-8-7-6-5-4-3-4-3-2-10

 We demonstrate two constructions:

**Ensembles** The work of Tsipras et al

 suggests a collection of non-robust and non-useful features, if

sufficiently uncorrelated, can be ensembled into a single useful,

non-robust useful feature f.

 process is illustrated above numerically. We choose a set of non-robust

 features by excluding all features above a threshold, and naively

ensembling them according to:

 (1-\alpha) \cdot a\_{\text{non-robust}} + \alpha \cdot a\_{\text{robust}},

 It is surprising, thus, that the experiments of Madry et al.

 To cite Ilyas et al.’s response, please cite their

**Response Summary**: The construction of explicit non-robust features is

 the useful non-robust features detected by our experiments. We also agree that

 non-robust features arising as “distractors” is indeed not precluded by our

 theoretical framework, even if it is precluded by our experiments.

 This simple theoretical framework sufficed for reasoning about and

 predicting the outcomes of our experiments

 We also presented a theoretical setting where we can

 analyze things fully rigorously in Section 4 of our paper..

 However, this comment rightly identifies finding a more comprehensive

 definition of feature as an important future research direction.

**Response**: These experiments (visualizing the robustness and

 corroborate the existence of useful, non-robust features and make progress

 towards visualizing what these non-robust features actually look like.

We also appreciate the point made by the provided construction of non-robust

 features (as defined in our theoretical framework) that are combinations of

 useful+robust and useless+non-robust features. Our theoretical framework indeed

 enables such a scenario, even if — as the commenter already notes — our

 framework technically captures.) Specifically, in such a scenario, during the

 construction of the D^det\widehat{\mathcal{D}}\_{det}D

det dataset, only the non-robust

 and useless term of the feature would be flipped. Thus, a classifier trained on

 such a dataset would associate the predictive robust feature with the

 *wrong* label and would thus not generalize on the test set. In contrast,

det

 do generalize.

Overall, our focus while developing our theoretical framework was on

 the comment points out, putting forth a theoretical framework that captures

 non-robust features in a very precise way is an important future research

 direction in itself.

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features' Two Examples of Useful, Non-Robust Features.html