Ettore Randazzo
Alexander Mordvintsev
Eyvind Niklasson
Michael Levin

Abstract: This article is part of the     Differentiable Self-organizing Systems Thread,     an experimental format collecting invited short articles delving into     differentiable self-organizing systems, interspersed with critical     commentary from several experts in adjacent fields.

### Contents

[Adversarial MNIST CAs](#adversarial-mnist-cas) |

[Perturbing the states of Growing CAs](#perturbing-the-states-of-growing-cas) |

[Related Work](#related-work)

[Discussion](#discussion)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Self-Organising Textures](https://distill.pub/selforg/2021/textures/)

In a complex system, whether biological, technological, or social,

how can we discover signaling events that will alter system-level

behavior in desired ways? Even when the rules governing the individual

components of these complex systems are known, the inverse problem -

going from desired behaviour to system design - is at the heart of many

barriers for the advance of biomedicine, robotics, and other fields of

importance to society.

Biology, specifically, is transitioning from a focus on mechanism

(what is required for the system to work) to a focus on information

(what algorithm is sufficient to implement adaptive behavior). Advances

in machine learning represent an exciting and largely untapped source of

 inspiration and tooling to assist the biological sciences. Growing

Neural Cellular Automata  and Self-classifying MNIST Digits

 introduced the Neural Cellular Automata (Neural CA) model and

demonstrated how tasks requiring self-organisation, such as pattern

growth and self-classification of digits, can be trained in an

end-to-end, differentiable fashion. The resulting models were robust to

various kinds of perturbations: the growing CA expressed regenerative

capabilities when damaged; the MNIST CA were responsive to changes in

the underlying digits, triggering reclassification whenever necessary.

These computational frameworks represent quantitative models with which

to understand important biological phenomena, such as scaling of single

cell behavior rules into reliable organ-level anatomies. The latter is a

 kind of anatomical homeostasis, achieved by feedback loops that must

recognize deviations from a correct target morphology and progressively

reduce anatomical error.

In this work, we *train adversaries* whose goal is to reprogram

CA into doing something other than what they were trained to do. In

order to understand what kinds of lower-level signals alter

system-level behavior of our CA, it is important to understand how these

 CA are constructed and where local versus global information resides.

The system-level behavior of Neural CA is affected by:

* **Individual cell states.** States store

information which is used for both diversification among cell behaviours

 and for communication with neighbouring cells.

* **The model parameters.** These describe the

input/output behavior of a cell and are shared by every cell of the same

 family. The model parameters can be seen as *the way the system works*.

* **The perceptive field.** This is how cells perceive

their environment. In Neural CA, we always restrict the perceptive field

 to be the eight nearest neighbors and the cell itself. The way cells

are perceived by each other is different between the Growing CA and

MNIST CA. The Growing CA perceptive field is a set of weights fixed both

 during training and inference, while the MNIST CA perceptive field is

learned as part of the model parameters.

We will explore two kinds of adversarial attacks: 1) injecting a few

adversarial cells into an existing grid running a pretrained model; and

2) perturbing the global state of all cells on a grid.

For the first type of adversarial attacks we train a new CA model

that, when placed in an environment running one of the original models

described in the previous articles, is able to hijack the behavior of

the collective mix of adversarial and non-adversarial CA. This is an

example of injecting CA with differing *model parameters* into the

system. In biology, numerous forms of hijacking are known, including

 Especially fascinating are the many cases of non-cell-autonomous

signaling developmental biology and cancer, showing that some cell

behaviors can significantly alter host properties both locally and at

long range. For example, bioelectrically-abnormal cells can trigger

metastatic conversion in an otherwise normal body (with no genetic

 All of these phenomena underlie the importance of understanding how

cell groups make collective decisions, and how those tissue-level

decisions can be subverted by the activity of a small number of cells.

It is essential to develop quantitative models of such dynamics, in

order to drive meaningful progress in regenerative medicine that

controls system-level outcomes top-down, where cell- or molecular-level

micromanagement is infeasible .

 We apply a global state perturbation to all living cells. This can be

seen as inhibiting or enhancing combinations of state values, in turn

hijacking proper communications among cells and within the cell’s own

states. Models like this represent not only ways of thinking about

adversarial relationships in nature (such as parasitism and evolutionary

 arms races of genetic and physiological mechanisms), but also a roadmap

 for the development of regenerative medicine strategies.

Next-generation biomedicine will need computational tools for inferring

minimal, least-effort interventions that can be applied to biological

systems to predictively change their large-scale anatomical and

behavioral properties.

Recall how the Self-classifying MNIST digits task consisted of

placing CA cells on a plane forming the shape of an MNIST digit. The

cells then had to communicate among themselves in order to come to a

complete consensus as to which digit they formed.

 (a) Local information neighbourhood - each cell can only observe itself

 and its neighbors’ states, or the absence of neighbours.

Your browser does not support the video tag.

 and freeze its parameters. We then train a new CA whose model

architecture is identical to the frozen model but is randomly

initialized. The training regime also closely approximates that of

self-classifying MNIST digits CA. There are three important differences:

* For each batch and each pixel, the CA is randomly chosen to be

either the pretrained model or the new adversarial one. The adversarial

CA is used 10% of the time, and the pre-trained, frozen, model the rest

of the time.

The adversarial attack as defined here only modifies a small

percentage of the overall system, but the goal is to propagate signals

that affect all the living cells. Therefore, these adversaries have to

somehow learn to communicate deceiving information that causes wrong

classifications in their neighbours and further cascades in the

propagation of deceiving information by ‘unaware’ cells. The unaware

cells’ parameters cannot be changed so the only means of attack by the

adversaries is to cause a change in the cells’ states. Cells’ states are

 responsible for communication and diversification.

The task is remarkably simple to optimize, reaching convergence in as

 little as 2000 training steps (as opposed to the two orders of

magnitude more steps needed to construct the original MNIST CA). By

visualising what happens when we remove the adversaries, we observe that

 the adversaries must be constantly communicating with their

non-adversarial neighbours to keep them convinced of the malicious

classification. While some digits don’t recover after the removal of

adversaries, most of them self-correct to the right classification.

Below we show examples where we introduce the adversaries at 200 steps

and remove them after a further 200 steps.

Your browser does not support the video tag.

We introduce the adversaries (red pixels) after 200 steps and remove

them after 200 more steps. Most digits recover, but not all. We

highlight mistakes in classification with a red background.

While we trained the adversaries with a 10-to-90% split of

adversarial vs. non-adversarial cells, we observe that often

significantly fewer adversaries are needed to succeed in the deception.

Below we evaluate the experiment with just one percent of cells being

adversaries.

Your browser does not support the video tag.

Adversaries constituting up 1% of the cell collective (red pixels). We

highlight mistakes in classification with a red background.

We created a demo playground where the reader can draw digits and

place adversaries with surgical precision. We encourage the reader to

play with the demo to get a sense of how easily non-adversarial cells

are swayed towards the wrong classification.

The natural follow up question is whether these adversarial attacks

work on Growing CA, too. The Growing CA goal is to be able to grow a

complex image from a single cell, and having its result be persistent

over time and robust to perturbations. In this article, we focus on the

lizard pattern model from Growing CA.

Your browser does not support the video tag.

The target CA to hijack.

The goal is to have some adversarial cells change the global

configuration of all the cells. We choose two new targets we would like

the adversarial cells to try and morph the lizard into: a tailless

lizard and a red lizard.

The desired mutations we want to apply.

These targets have different properties:

* **Red lizard:** converting a lizard from green to

red would show a global change in the behaviour of the cell collective.

This behavior is not present in the dynamics observed by the original

model. The adversaries are thus tasked with fooling other cells into

doing things they have never done before (create the lizard shape as

before, but now colored in red).

* **Tailless lizard:** having a severed tail is a more

localized change that only requires some cells to be fooled into

behaving in the wrong way: the cells at the base of the tail need to be

convinced they constitute the edge or silhouette of the lizard, instead

of proceeding to grow a tail as before.

We first train adversaries for the tailless target with a 10% chance

for any given cell to be an adversary. We prohibit cells to be

adversaries if they are outside the target pattern; i.e. the tail

contains no adversaries.

Your browser does not support the video tag.

10% of the cells are adversarial.

The video above shows six different instances of the same model with

differing stochastic placement of the adversaries. The results vary

considerably: sometimes the adversaries succeed in removing the tail,

sometimes the tail is only shrunk but not completely removed, and other

times the pattern becomes unstable. Training these adversaries required

many more gradient steps to achieve convergence, and the pattern

converged to is qualitatively worse than what was achieved for the

adversarial MNIST CA experiment.

The red lizard pattern fares even worse. Using only 10% adversarial

cells results in a complete failure: the original cells are unaffected

by the adversaries. Some readers may wonder whether the original

pretrained CA has the requisite skill, or ‘subroutine’ of producing a

red output at all, since there are no red regions in the original

target, and may suspect this was an impossible task to begin with.

Therefore, we increased the proportion of adversarial cells until we

managed to find a successful adversarial CA, if any were possible.

Your browser does not support the video tag.

In the video above we can see how, at least in the first stages of

morphogenesis, 60% of adversaries are capable of coloring the lizard

 where we hide the adversarial cells and show only the original cells.

There, we see how a handful of original cells are colored in red. This

is proof that the adversaries successfully managed to steer neighboring

cells to color themselves red, where needed.

However, the model is very unstable when iterated for periods of time

 longer than seen during training. Moreover, the learned adversarial

attack is dependent on a majority of cells being adversaries. For

instance, when using fewer adversaries on the order of 20-30%, the

configuration is unstable.

In comparison to the results of the previous experiment, the Growing

CA model shows a greater resistance to adversarial perturbation than

those of the MNIST CA. A notable difference between the two models is

that the MNIST CA cells have to always be ready and able to change an

opinion (a classification) based on information propagated through

several neighbors. This is a necessary requirement for that model

because at any time the underlying digit may change, but most of the

cells would not observe any change in their neighbors’ placements. For

instance, imagine the case of a one turning into a seven where the lower

 stroke of each overlap perfectly. From the point of view of the cells

in the lower stroke of the digit, there is no change, yet the digit

formed is now a seven. We therefore hypothesise MNIST CA are more

reliant and ‘trusting’ of continuous long-distance communication than

Growing CA, where cells never have to reconfigure themselves to generate

 something different to before.

We suspect that more general-purpose Growing CA that have learned a

variety of target patterns during training are more likely to be

susceptible to adversarial attacks.

We observed that it is hard to fool Growing CA into changing their

morphology by placing adversarial cells inside the cell collective.

These adversaries had to devise complex local behaviors that would cause

 the non-adversarial cells nearby, and ultimately globally throughout

the image, to change their overall morphology.

In this section, we explore an alternative approach: perturbing the

global state of all cells without changing the model parameters of any

cell.

As before, we base our experiments on the Growing CA model trained to

 produce a lizard. Every cell of a Growing CA has an internal state

vector with 16 elements. Some of them are phenotypical elements (the

RGBA states) and the remaining 12 serve arbitrary purposes, used for

storing and communicating information. We can perturb the states of

these cells to hijack the overall system in certain ways (the discovery

of such perturbation strategies is a key goal of biomedicine and

synthetic morphology). There are a variety of ways we can perform state

perturbations. We will focus on *global state perturbations*,

defined as perturbations that are applied on every living cell at every

time step (analogous to “systemic” biomedical interventions, that are

given to the whole organism (e.g., a chemical taken internally), as

opposed to highly localized delivery systems). The new goal is to

discover a certain type of global state perturbation that results in a

stable new pattern.

Diagram showing some possible stages for perturbing a lizard

 pattern. (a) We start from a seed that grows into a lizard (b) Fully

converged lizard. (c) We apply a global state perturbation at every

step. As a result, the lizard loses its tail. (d) We stop perturbing the

 state. We observe the lizard immediately grows back its tail.

We show 6 target patterns: the tailless and red lizard from the

previous experiment, plus a blue lizard and lizards with various severed

 limbs and severed head.

Mosaic of the desired mutations we want to apply.

 To give insight on why we chose this, an even simpler “state addition”

mutation (a mutation consisting only of the addition of a vector to

every state) would be insufficient because the value of the states of

our models are unbounded, and often we would want to suppress something

by setting it to zero. The latter is generally impossible with constant

state additions, as a constant addition or subtraction of a value would

generally lead to infinity, except for some fortunate cases where the

natural residual updates of the cells would cancel out with the constant

 addition at precisely state value zero. However, matrix multiplications

 have the possibility of amplifying/suppressing combinations of elements

 in the states: multiplying a state value repeatedly for a constant

value less than one can easily suppress a state value to zero. We

constrain the matrix to be symmetric for reasons that will become clear

in the following section.

* The underlying CA parameters are frozen and we only train AAA.

* We consider the set of initial image configurations to be both the

seed state and the state with a fully grown lizard (as opposed to the

Growing CA article, where initial configurations consisted of the seed

state only).

Your browser does not support the video tag.

Effect of applying the trained perturbations.

The video above shows the model successfully discovering global state

 perturbations able to change a target pattern to a desired variation.

We show what happens when we stop perturbing the states (an

out-of-training situation) at step 500 through step 1000, then

reapplying the mutation. This demonstrates the ability of our

perturbations to achieve the desired result both when starting from a

seed, and when starting from a fully grown pattern. Furthermore it

demonstrates that the original CA easily recover from these state

perturbations once it goes away. This last result is perhaps not

surprising given how robust growing CA models are in general.

Not all perturbations are equally effective. In particular, the

headless perturbation is the least successful as it results in a loss of

 other details across the whole lizard pattern such as the white

coloring on its back. We hypothesize that the best perturbation our

training regime managed to find, due to the simplicity of the

perturbation, was suppressing a “structure” that contained both the

morphology of the head and the white colouring. This may be related to

the concept of differentiation and distinction of biological organs.

Predicting what kinds of perturbations would be harder or impossible to

be done, before trying them out empirically, is still an open research

question in biology. On the other hand, a variant of this kind of

synthetic analysis might help with defining higher order structures

within biological and synthetic systems.

### Directions and compositionality of perturbations

Our choice of using a symmetric matrix for representing global state

perturbations is justified by a desire to have compositionality. Every

complex symmetric matrix AAA can be diagonalized as follows:

A=QΛQ⊺A = Q \Lambda Q^\intercalA=QΛQ⊺

where Λ\LambdaΛ is the diagonal eigenvalues matrix and QQQ

 is the unitary matrix of its eigenvectors. Another way of seeing this

is applying a change of basis transformation, scaling each component

proportional to the eigenvalues, and then changing back to the original

basis. This should also give a clearer intuition on the ease of

suppressing or amplifying combinations of states. Moreover, we can now

infer what would happen if all the eigenvalues were to be one. In that

Let us then take the tailless perturbation and see what happens as we vary kkk:

Your browser does not support the video tag.

 negative, the lizard grows a longer tail. Unfortunately, the further

away we go, the more unstable the system becomes and eventually the

lizard pattern grows in an unbounded fashion. This behaviour likely

stems from that perturbations applied on the states also affect the

homeostatic regulation of the system, making some cells die out or grow

in different ways than before, resulting in a behavior akin to “cancer”

in biological systems.

In that case,

 this could result in a stable perturbation. An intuitive understanding

of this is interpolating stable perturbations using the direction

coefficients.

In practice, however, the eigenvectors are also different, so the

results of the combination will likely be worse the more different the

respective eigenvector bases are.

Below, we interpolate the direction coefficients, while keeping their

 sum to be one, of two types of perturbations: tailless and no-leg

lizards.

Your browser does not support the video tag.

While it largely achieves what we expect, we observe some unintended

effects such as the whole pattern starting to traverse vertically in the

 grid. Similar results happen with other combinations of perturbations.

What happens if we remove the restriction of the sum of kkks

 being equal to one, and instead add both perturbations in their

entirety? We know that if the two perturbations were the same, we would

end twice as far away from the identity perturbation, and in general we

expect the variance of these perturbations to increase. Effectively,

this means going further and further away from the stable perturbations

discovered during training. We would expect more unintended effects that

 may disrupt the CA as the sum of kkks increases.

Below, we demonstrate what happens when we combine the tailless and

the no-leg lizard perturbations at their fullest. Note that when we set

Your browser does not support the video tag.

Effect of composing two perturbations.

Surprisingly, the resulting pattern is almost as desired. However, it

 also suffers from the vertical movement of the pattern observed while

interpolating kkks.

This framework can be generalized to any arbitrary number of

perturbations. Below, we have created a small playground that allows the

 reader to input their desired combinations. Empirically, we were

surprised by how many of these combinations result in the intended

Related work

------------

This work is inspired by Generative Adversarial Networks (GANs) .

 While with GANs it is typical to cotrain pairs of models, in this work

we froze the original CA and trained the adversaries only. This setup is

### Influence maximization

Adversarial cellular automata have parallels to the field of

influence maximization. Influence maximization involves determining the

optimal nodes to influence in order to maximize influence over an entire

 graph, commonly a social graph, with the property that nodes can in

turn influence their neighbours. Such models are used to model a wide

variety of real-world applications involving information spread in a

graph.

 A common setting is that each vertex in a graph has a binary state,

which will change if and only if a sufficient fraction of its

neighbours’ states switch. Examples of such models are social influence

maximization (maximally spreading an idea in a network of people),

 (when small perturbations to a system bring about a larger ‘phase

change’). At the time of writing this article, for instance, contagion

minimization is a model of particular interest. NCA are a graph - each

cell is a vertex and has edges to its eight neighbours, through which it

 can pass information. This graph and message structure is significantly

 more complex than the typical graph underlying much of the research in

influence maximization, because NCA cells pass vector-valued messages

and have a complex update rules for their internal states, whereas

graphs in influence maximization research typically consist of more

simple binary cells states and threshold functions on edges determining

whether a node has switched states. Many concepts from the field could

be applied and are of interest, however.

For example, in this work, we have made an assumption that our

adversaries can be positioned anywhere in a structure to achieve a

desired behaviour. A common focus of investigation in influence

maximization problems is deciding which nodes in a graph will result in

maximal influence on the graph, referred to as target set selection .

 This problem isn’t always tractable, often NP-hard, and solutions

frequently involve simulations. Future work on adversarial NCA may

involve applying techniques from influence maximization in order to find

 the optimal placement of adversarial cells.

Discussion

----------

This article showed two different kinds of adversarial attacks on Neural CA.

Injections of adversarial CA in a pretrained Self-classifying MNIST

CA showed how an existing system of cells that are heavily reliant on

the passing of information among each other is easily swayed by

deceitful signaling. This problem is routinely faced by biological

systems, which face hijacking of behavioral, physiological, and

morphological regulatory mechanisms by parasites and other agents in the

 biosphere with which they compete. Future work in this field of

computer technology can benefit from research on biological

communication mechanisms to understand how cells maximize reliability

and fidelity of inter- and intra-cellular messages required to implement

 adaptive outcomes.

The adversarial injection attack was much less effective against

Growing CA and resulted in overall unstable CA. This dynamic is also of

importance to the scaling of control mechanisms (swarm robotics and

nested architectures): a key step in “multicellularity” (joining

together to form larger systems from sub-agents )

 is informational fusion, which makes it difficult to identify the

source of signals and memory engrams. An optimal architecture would need

 to balance the need for validating control messages with a possibility

of flexible merging of subunits, which wipes out metadata about the

specific source of informational signals. Likewise, the ability to

respond successfully to novel environmental challenges is an important

goal for autonomous artificial systems, which may import from biology

strategies that optimize tradeoff between maintaining a specific set of

signals and being flexible enough to establish novel signaling regimes

when needed.

The global state perturbation experiment on Growing CA shows how it

is still possible to hijack these CA towards stable out-of-training

configurations and how these kinds of attacks are somewhat composable in

 a similar way to how embedding spaces are manipulable in the natural

 We hypothesize that this is partially due to the regenerative

capabilities of the pretrained CA, and that other models may be less

capable of recovery from arbitrary perturbations.

Growing Neural Cellular Automata

Self-classifying MNIST Digits

Herpes Simplex Virus: The Hostile Guest That Takes Over Your Home

The
 role of gut microbiota (commensal bacteria) and the mucosal barrier in
the pathogenesis of inflammatory and autoimmune diseases and cancer:
contribution of germ-free and gnotobiotic animal models of human
diseases

Regulation of axial and head patterning during planarian regeneration by a commensal bacterium

Toxoplasma gondii infection and behavioral outcomes in humans: a systematic review

Resting potential, oncogene-induced tumorigenesis, and metastasis: the bioelectric basis of cancer in vivo

Transmembrane voltage potential of somatic cells controls oncogene-mediated tumorigenesis at long-range

Cross-limb communication during Xenopus hindlimb regenerative response: non-local bioelectric injury signals

Local
 and long-range endogenous resting potential gradients antagonistically
regulate apoptosis and proliferation in the embryonic CNS

Top-down models in biology: explanation and control of complex living systems above the molecular level

Generative Adversarial Networks

Adversarial Reprogramming of Neural Networks

Efficient Estimation of Word Representations in Vector Space

Fader Networks: Manipulating Images by Sliding Attributes

Maximizing the spread of influence through a social network

The Independent Cascade and Linear Threshold Models

A Survey on Influence Maximization in a Social Network

Simplicial models of social contagion

Cascading Behavior in Networks: Algorithmic and Economic Issues

On the Approximability of Influence in Social Networks

The Computational Boundary of a “Self”: Developmental Bioelectricity Drives Multicellularity and Scale-Free Cognition

Adversarial Reprogramming of Neural Cellular Automata.html


Augustus Odena
Vincent Dumoulin
Chris Olah

Deconvolution and Checkerboard Artifacts

========================================

 dt-byline {

 font-size: 12px;

 line-height: 18px;

 display: block;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 color: rgba(0, 0, 0, 0.5);

 padding-top: 12px;

 padding-bottom: 12px;

 }

 dt-article.centered dt-byline {

 text-align: center;

 }

 dt-byline a,

 dt-article dt-byline a {

 text-decoration: none;

 border-bottom: none;

 }

 dt-article dt-byline a:hover {

 text-decoration: underline;

 border-bottom: none;

 }

 dt-byline .authors {

 text-align: left;

 }

 dt-byline .name {

 display: inline;

 text-transform: uppercase;

 }

 dt-byline .affiliation {

 display: inline;

 }

 dt-byline .date {

 display: block;

 text-align: left;

 }

 dt-byline .year, dt-byline .month {

 display: inline;

 }

 dt-byline .citation {

 display: block;

 text-align: left;

 }

 dt-byline .citation div {

 display: inline;

 }

 @media(min-width: 768px) {

 dt-byline {

 }

 }

 @media(min-width: 1080px) {

 dt-byline {

 border-bottom: none;

 margin-bottom: 70px;

 }

 dt-byline a:hover {

 color: rgba(0, 0, 0, 0.9);

 }

 dt-byline .authors {

 display: inline-block;

 }

 dt-byline .author {

 display: inline-block;

 margin-right: 12px;

 /\*padding-left: 20px;\*/

 /\*border-left: 1px solid #ddd;\*/

 }

 dt-byline .affiliation {

 display: block;

 }

 dt-byline .author:last-child {

 margin-right: 0;

 }

 dt-byline .name {

 display: block;

 }

 dt-byline .date {

 border-left: 1px solid rgba(0, 0, 0, 0.1);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .year, dt-byline .month {

 display: block;

 }

 dt-byline .citation {

 border-left: 1px solid rgba(0, 0, 0, 0.15);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .citation div {

 display: block;

 }

 }

Augustus Odena

[Google Brain](http://g.co/brain)

[Vincent Dumoulin](http://vdumoulin.github.io/)

[Université de Montréal](https://mila.umontreal.ca/en/)

[Chris Olah](http://colah.github.io/)

[Google Brain](http://g.co/brain)

Oct. 17

2016

[Citation:

Odena, et al., 2016](#citation)

When we look very closely at images generated by neural networks,

we often see a strange checkerboard pattern of artifacts. It’s more

obvious in some cases than others, but a large fraction of recent models

 exhibit this behavior.

.pattern-examples {

 width: 100%;

}

.pattern-examples > figure {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 position: relative;

}

.pattern-examples > figure:after {

 overflow: visible;

 content: "";

 background-image: url("assets/pointer.svg");

 width: 27px;

 height: 27px;

 position: absolute;

 right: -51px;

}

.pattern-examples .example {

 width: 100%;

 margin-right: 2.5%;

}

.pattern-examples .example:last-of-type {

 margin-right: 0;

}

.pattern-examples .original {

 position: relative;

}

.pattern-examples .original:after {

 content: "";

 display: block;

 width: 100%;

 height: 150%;

 position: absolute;

 top: -20%;

 z-index: 1;

}

.pattern-examples .example .reticle {

 position: absolute;

 pointer-events: none;

}

.pattern-examples .example .reticle:after {

 box-sizing: border-box;

 position: relative;

 width: 100%;

 height: 100%;

 left: -50%;

 top: -50%;

 border-radius: 50%;

 border: 1px solid white;

 display: block;

 content: '';

 background-color: rgba(255, 255, 255, 0.2);

 box-shadow: 0 0 4px rgba(0, 0, 0, 0.5);

}

.pattern-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

.pattern-examples .closeup {

 display: block;

 width: 90%;

 border-radius: 50%;

 margin-bottom: 12px;

 position: relative;

 left: 5%;

 top: -5px;

 border: 2px solid white;

 overflow: hidden;

 box-shadow: 0 2px 8px rgba(0, 0, 0, 0.3);

}

.pattern-examples figcaption {

 width: 90%;

 margin-left: 5%;

 text-align: center;

}

.pattern-examples .closeup img {

 position: absolute;

 width: 800%;

}

.pattern-examples .closeup:after {

 padding-top: 100%;

 display: block;

 content: '';

}

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_dcgan.png)

Radford, et al., 2015 [1]

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_salimans.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_salimans.png)

Salimans et al., 2016 [2]

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_donahue.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_donahue.png)

Donahue, et al., 2016 [3]

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_dumoulin.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_dumoulin.png)

Dumoulin, et al., 2016 [4]

(function() {

var html = d3.select(".pattern-examples");

var original = html.selectAll(".example .original");

html

 .on("mouseleave", () => {

 console.log("out")

 original.each(resetReticle);

 });

original

 .datum(function() {

 var focus = this.getAttribute("data-focus").split(",")

 return {

 zoom: focus[0],

 x: focus[1],

 y: focus[2]

 };

 })

 .on("mouseleave", resetReticle)

 .on("mousemove", updateReticle)

 .each(resetReticle);

function updateReticle(d) {

 original.each(resetReticle);

 var x = d3.event.offsetX / this.getBoundingClientRect().width;

 var y = d3.event.offsetY / this.getBoundingClientRect().height;

 setPosition(this, x, y, d.zoom, 100);

}

function resetReticle(d) {

 setPosition(this, d.x, d.y, d.zoom, 300);

}

function setPosition(element, x, y, zoom, duration) {

 var margin = 1 / zoom / 2;

 x = Math.min(Math.max(margin, x), 1 - margin)

 y = Math.min(Math.max(margin, y), 1 - margin)

 var parent = d3.select(element.parentElement);

 var reticle = parent.select(".reticle");

 var closeup = parent.select(".closeup img");

 console.log(x, y)

 reticle

 .style("width", 100 / zoom + "%")

 .style("height", 100 / zoom + "%")

 .transition()

 .ease(d3.ease("cubic-out"))

 .duration(duration)

 .styleTween("left", function (d, i, a) {

 var from = this.style.left,

 to = x \* 100 + "%";

 return d3.interpolateString(from, to);

 })

 .styleTween("top", function (d, i, a) {

 var from = this.style.top,

 to = y \* 100 + "%";

 return d3.interpolateString(from, to);

 })

 closeup

 .style("width", zoom \* 100 + "%")

 .style("height", zoom \* 100 + "%")

 .transition()

 .ease(d3.ease("cubic-out"))

 .duration(duration)

 .styleTween("left", function (d, i, a) {

 var from = this.style.left,

 to = -x \* 100 \* zoom + 50 + "%";

 return d3.interpolateString(from, to);

 })

 .styleTween("top", function (d, i, a) {

 var from = this.style.top,

 to = -y \* 100 \* zoom + 50 + "%";

 return d3.interpolateString(from, to);

 })

}

})();

Mysteriously, the checkerboard pattern tends to be most prominent

in images with strong colors.

 What’s going on? Do neural networks hate bright colors? The actual

cause of these artifacts is actually remarkably simple, as is a method

for avoiding them.

---

Deconvolution & Overlap

-----------------------

When we have neural networks generate images, we often have them build them up

 from low resolution, high-level descriptions.

 We generally do this with the *deconvolution* operation.

 Roughly, deconvolution layers allow the model to use every point

 in the small image to “paint” a square in the larger one.

 We use the name “deconvolution” in this article for brevity.

 For excellent discussion of deconvolution, see [5, 6].)

Unfortunately, deconvolution can easily have “uneven overlap,”

 putting more of the metaphorical paint in some places than others [7].

 In particular, deconvolution has uneven overlap when the kernel size

(the output window size) is not divisible by the stride (the spacing

between points on the top).

 While the network could, in principle, carefully learn weights to

avoid this

  — as we’ll discuss in more detail later —

 in practice neural networks struggle to avoid it completely.

stride = 1size = 8stride = 2size = 3

deconv1d();

The overlap pattern also forms in two dimensions.

 The uneven overlaps on the two axes multiply together,

 creating a characteristic checkerboard-like pattern of varying magnitudes.

deconv2d();

In fact, the uneven overlap tends to be more extreme in two dimensions!

 Because the two patterns are multiplied together, the unevenness gets squared.

 but in two dimensions this becomes a factor of four.

 While it’s possible for these stacked deconvolutions to cancel out artifacts,

 they often compound, creating artifacts on a variety of scales.

stride = 1size = 5stride = 2size = 3stride = 2size = 3

deconv1d\_multi();

Stride 1 deconvolutions —

 which we often see as the last layer in successful models (eg. [2])

  — are quite effective at dampening artifacts.

 They can remove artifacts of frequencies

 size. However, artifacts can still leak through, as seen in many recent models.

In addition to the high frequency checkerboard-like artifacts we observed above,

 early deconvolutions can create lower-frequency artifacts,

 which we’ll explore in more detail later.

These artifacts tend to be most prominent when outputting unusual colors.

 Since neural network layers typically have a bias

 (a learned value added to the output) it’s easy to output the average color.

 The further a color — like bright red — is away from the average color,

 the more deconvolution needs to contribute.

---

Overlap & Learning

------------------

Thinking about things in terms of uneven overlap is — while a useful framing —

 is evenly balanced.

In fact, not only do models with uneven overlap not learn to avoid this,

 but models with even overlap often learn kernels that cause similar artifacts!

 While it isn’t their default behavior the way it is for uneven overlap,

 it’s still very easy for even overlap deconvolution to cause artifacts.

Completely avoiding artifacts is still a significant restriction on filters,

 (See [4],

 which uses stride 2 size 4 deconvolutions, as an example.)

There are probably a lot of factors at play here.

 For example, in the case of Generative Adversarial Networks (GANs),

one issue may be the discriminator and its gradients

 (we’ll discuss this more later).

 But a big part of the problem seems to be deconvolution.

 At best, deconvolution is fragile because it very easily represents

artifact creating functions, even when the size is carefully chosen.

 At worst, creating artifacts is the default behavior of deconvolution.

Is there a different way to upsample that is more resistant to artifacts?

---

Better Upsampling

-----------------

 Ideally, it would go further, and be biased against such artifacts.

 avoiding the overlap issue.

 This is equivalent to “sub-pixel convolution,” a technique which has recently

 had success in image super-resolution [8].

Both deconvolution and the different resize-convolution approaches

are linear operations, and can be interpreted as matrices.

 This a helpful way to see the differences between them.

 Where deconvolution has a unique entry for each output window,

resize-convolution is implicitly weight-tying in a way that discourages

high frequency artifacts.

We’ve had our best results with nearest-neighbor interpolation, and

 had difficulty making bilinear resize work.

 This may simply mean that, for our models, the nearest-neighbor

happened to work well with hyper-parameters optimized for deconvolution.

 It might also point at trickier issues with naively using bilinear

interpolation, where it resists high-frequency image features too

strongly.

 We don’t necessarily think that either approach is the final solution

to upsampling, but they do fix the checkerboard artifacts.

### Code

---

Image Generation Results

------------------------

One case where we’ve found this approach to help is Generative

Adversarial Networks. Simply switching out the standard deconvolutional

layers for nearest-neighbor resize followed by convolution causes

artifacts of different frequencies to disappear.

.fix-examples > .row {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 /\*align-items: center;\*/

 margin-bottom: 10px;

}

.fix-examples > .row > .example {

 flex-grow: 1;

 width: 100%;

 margin-right: 10px;

}

.fix-examples > .row > .explanation {

 flex-grow: 2;

 width: 200%;

 margin-left: 10px;

}

.fix-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

 Deconv in last two layers.

 Other layers use resize-convolution.

*Artifacts of frequency 2 and 4.*

 Deconv only in last layer.

 Other layers use resize-convolution.

*Artifacts of frequency 2.*

 All layers use resize-convolution.

*No artifacts.*

In fact, the difference in artifacts can be seen before any training occurs.

 we can already see the artifacts:

.fix-step0-examples > .row {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 /\*align-items: center;\*/

 margin-bottom: 10px;

}

.fix-step0-examples > .row > .exampleA {

 flex-grow: 1;

 width: 100%;

 margin-right: 10px;

}

.fix-step0-examples > .row > .exampleB {

 flex-grow: 1;

 width: 100%;

 margin-right: 20px;

}

.fix-step0-examples > .row > .explanation {

 flex-grow: 2;

 width: 200%;

}

.fix-step0-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_two_0.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_two_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_one_0.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_one_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_zero_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_zero_2.png)

 Deconvolution in last two layers.

 *Artifacts prior to any training.*

 Deconvolution only in last layer.

*Artifacts prior to any training.*

 All layers use resize-convolution.

*No artifacts before or after training.*

This suggests that the artifacts are due to this method of

generating images, rather than adversarial training.

 (It also suggests that we might be able to learn a lot about good

generator design without the slow feedback cycle of training models.)

Another reason to believe these artifacts aren’t GAN specific is

that we see them in other kinds of models, and have found that they also

 go away when we switch to resize-convolution upsampling.

.style-examples > .row {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 /\*align-items: center;\*/

 margin-bottom: 10px;

}

.style-examples > .row > .example {

 flex-grow: 4;

 width: 400%;

 margin-right: 10px;

}

.style-examples > .row > .explanation {

 flex-grow: 2;

 width: 200%;

 margin-left: 10px;

}

.style-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/style_artifacts.png)

 Using deconvolution.

*Heavy checkerboard artifacts.*

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/style_clean.png)

 Using resize-convolution.

*No checkerboard artifacts.*

Forthcoming papers from the Google Brain team will demonstrate the

benefits of this technique

 in more thorough experiments and state-of-the-art results.

 (We’ve chosen to present this technique separately because we felt it

merited more detailed discussion, and because it cut across multiple

papers.)

---

Artifacts in Gradients

----------------------

Whenever we compute the gradients of a convolutional layer,

 we do deconvolution (transposed convolution) on the backward pass.

 This can cause checkerboard patterns in the gradient,

 just like when we use deconvolution to generate images.

The presence of high-frequency “noise” in image model gradients is

 Somehow, feature visualization methods must compensate for this noise.

For example, DeepDream [11]

 seems to cause destructive interference between artifacts in a number of ways,

.deepdream-examples {

 position: relative;

}

.deepdream-examples::after {

 overflow: visible;

 content: "";

 background-image: url("assets/pointer.svg");

 width: 27px;

 height: 27px;

 position: absolute;

 right: -51px;

 top: 0;

}

.deepdream-examples .example {

 position: relative;

}

.deepdream-examples .original {

 position: relative;

 width: 45%;

}

.deepdream-examples .original:after {

 content: "";

 display: block;

 width: 100%;

 height: 70%;

 position: absolute;

 top: -20%;

 z-index: 1;

}

.deepdream-examples .example .reticle {

 position: absolute;

 pointer-events: none;

}

.deepdream-examples .example .reticle:after {

 box-sizing: border-box;

 position: relative;

 width: 100%;

 height: 100%;

 left: -50%;

 top: -50%;

 border-radius: 50%;

 border: 1px solid white;

 display: block;

 content: '';

 background-color: rgba(255, 255, 255, 0.2);

 box-shadow: 0 0 4px rgba(0, 0, 0, 0.5);

}

.deepdream-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

.deepdream-examples .closeup img {

 -webkit-filter: contrast(150%);

 filter: contrast(150%);

}

.deepdream-examples .closeup {

 position: absolute;

 right: 34%;

 top: 0;

 width: 22.5%;

 border-radius: 50%;

 margin-bottom: 12px;

 border: 2px solid white;

 overflow: hidden;

 box-shadow: 0 2px 8px rgba(0, 0, 0, 0.3);

}

.deepdream-examples figcaption {

 position: absolute;

 right: 0%;

 top: 0;

 width: 30%;

}

.deepdream-examples .closeup img {

 position: absolute;

 width: 800%;

}

.deepdream-examples .closeup:after {

 padding-top: 100%;

 display: block;

 content: '';

}

 DeepDream only applying the neural network to a fixed position.

*Severe artifacts.*

 DeepDream applying the network to a different position each step.

*Reduced artifacts.*

(function() {

var html = d3.select(".deepdream-examples");

var original = html.selectAll(".example .original");

html

 .on("mouseleave", () => {

 console.log("out")

 original.each(resetReticle);

 });

original

 .datum(function() {

 var focus = this.getAttribute("data-focus").split(",")

 return {

 zoom: focus[0],

 x: focus[1],

 y: focus[2]

 };

 })

 .on("mouseleave", resetReticle)

 .on("mousemove", updateReticle)

 .each(resetReticle);

function updateReticle(d) {

 original.each(resetReticle);

 var x = d3.event.offsetX / this.getBoundingClientRect().width;

 var y = d3.event.offsetY / this.getBoundingClientRect().height;

 setPosition(this, x, y, d.zoom, 100);

}

function resetReticle(d) {

 setPosition(this, d.x, d.y, d.zoom, 300);

}

function setPosition(element, x, y, zoom, duration) {

 var marginX = 1 / zoom / 4;

 var marginY = 1 / zoom / 2;

 x = Math.min(Math.max(marginX, x), 1 - marginX)

 y = Math.min(Math.max(marginY, y), 1 - marginY)

 var parent = d3.select(element.parentElement);

 var reticle = parent.select(".reticle");

 var closeup = parent.select(".closeup img");

 reticle

 .style("width", 50 / zoom + "%")

 .style("height", 100 / zoom + "%")

 .transition()

 .ease(d3.ease("cubic-out"))

 .duration(duration)

 .styleTween("left", function (d, i, a) {

 var from = this.style.left,

 to = x \* 100 + "%";

 return d3.interpolateString(from, to);

 })

 .styleTween("top", function (d, i, a) {

 var from = this.style.top,

 to = y \* 100 + "%";

 return d3.interpolateString(from, to);

 })

 closeup

 .style("width", zoom \* 200 + "%")

 .style("height", zoom \* 100 + "%")

 .transition()

 .ease(d3.ease("cubic-out"))

 .duration(duration)

 .styleTween("left", function (d, i, a) {

 var from = this.style.left,

 to = -x \* 200 \* zoom + 50 + "%";

 return d3.interpolateString(from, to);

 })

 .styleTween("top", function (d, i, a) {

 var from = this.style.top,

 to = -y \* 100 \* zoom + 50 + "%";

 return d3.interpolateString(from, to);

 })

}

})();

(While some of the artifacts are our standard checkerboard pattern,

 others are a less organized high-frequency pattern.

 We believe these to be caused by max pooling.

 Max pooling was previously linked to high-frequency artifacts in [12].)

More recent work in feature visualization (eg. [13]),

Do these gradient artifacts affect GANs?

 If gradient artifacts can affect an image being optimized based on a

neural networks gradients in feature visualization,

 we might also expect it to affect the family of images parameterized

by the generator as they’re optimized by the discriminator in GANs.

We’ve found that this does happen in some cases.

 When the generator is neither biased for or against checkerboard patterns,

 strided convolutions in the discriminator can cause them.

.discrim-examples > .row {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 /\*align-items: center;\*/

 margin-bottom: 10px;

}

.discrim-examples > .row > .exampleA {

 flex-grow: 1;

 width: 100%;

 margin-right: 10px;

}

.discrim-examples > .row > .exampleB {

 flex-grow: 1;

 width: 100%;

 margin-right: 20px;

}

.discrim-examples > .row > .explanationA {

 flex-grow: 3;

 width: 300%;

 padding-right: 20px;

}

.discrim-examples > .row > .explanationB {

 flex-grow: 3;

 width: 300%;

 padding-right: 10px;

}

.discrim-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride2_0.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride2_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride2_2.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride1_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride1_2.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride1_0.png)

 Discriminator has stride 2 convolution in first layer.

 *Strong frequency 2 artifacts.*

 Discriminator has regular convolution in first layer.

 *Very mild artifacts.*

It’s unclear what the broader implications of these gradient artifacts are.

 Neither of those sounds ideal.

It seems possible that having some pixels affect the network output

 much more than others may exaggerate adversarial counter-examples.

 Because the derivative is concentrated on small number of pixels,

 small perturbations of those pixels may have outsized effects.

 We have not investigated this.

---

Conclusion

----------

The standard approach of producing images with

deconvolution — despite its successes! — has some conceptually simple

issues that lead to artifacts in produced images.

 Using a natural alternative without these issues causes the artifacts

to go away

 (Analogous arguments suggest that standard strided convolutional

layers may also have issues).

This seems like an exciting opportunity to us!

 It suggests that there is low-hanging fruit to be found in carefully

thinking through neural network architectures, even ones where we seem

to have clean working solutions.

In the meantime, we’ve provided an easy to use solution that

improves the quality of many approaches to generating images with neural

 networks. We look forward to seeing what people do with it, and whether

 it helps in domains like audio, where high frequency artifacts would be

 particularly problematic.

Deconvolution and Checkerboard Artifacts.html


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Robust Feature Leakage

Abstract: This article is part of a discussion of the Ilyas et al. paper         “Adversarial examples are not bugs, they are features”.         You can learn more in the                      main discussion article         .

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

 Ilyas et al.  report a surprising result: a model trained on

### Lower Bounding Leakage

 Our technique for quantifying leakage consisting of two steps:

 specify.

2. Next, we train a linear classifier as per ,

 Equation 3 on the datasets D^det\hat{\mathcal{D}}\_{\text{det}}D^det and

 D^rand\hat{\mathcal{D}}\_{\text{rand}}D^rand (Defined , Table 1) on

 these robust features *only*.

 Since Ilyas et al.  only specify robustness in the two class

 setting:

 **Specification 1**

For at least one of the

 classes, the feature is γ\gammaγ-robustly useful   with

 **Specification 2**

 that remain static in a neighborhood of radius 0.25 on the L2L\_2L2 norm ball.

 CIFAR test set incurs an accuracy of **23.5%** (out of 88%). Doing the same on

 features, e.g. from a robust deep neural network.

 The results of D^det\hat{\mathcal{D}}\_{\text{det}}D^det in Table 1 of

 non-robust features, exactly the thesis of .

 To cite Ilyas et al.’s response, please cite their

**Response Summary**: This

 is a valid concern that was actually one of our motivations for creating the

 D^det\widehat{\mathcal{D}}\_{det}D

det dataset (which, as the comment notes, actually

 has *misleading* robust features). The provided experiment further

 improves our understanding of the underlying phenomenon.

**Response**: This comment raises a valid concern which was in fact one of

 the primary reasons for designing the D^det\widehat{\mathcal{D}}\_{det}D

det dataset.

rand

 dataset: assign each input a random target label and do PGD towards that label.

 Note that unlike the D^det\widehat{\mathcal{D}}\_{det}D

det dataset (in which the

rand

 dataset allows for robust features to actually have a (small) positive

 correlation with the label.

To see how this can happen, consider the following simple setting: we have a

 (as in the dataset D^rand\widehat{\mathcal{D}}\_{rand}D

rand) would make this feature

 targeted attack might in this case induce some correlation with the

 to correctly classify new inputs.

In other words, starting from a dataset with no features, one can encode

 robust features within small perturbations. In contrast, in the

 D^det\widehat{\mathcal{D}}\_{det}D

det dataset, the robust features are *correlated

 with the original label* (since the labels are permuted) and since they are

 robust, they cannot be flipped to correlate with the newly assigned (wrong)

 label. Still, the D^rand\widehat{\mathcal{D}}\_{rand}D

rand dataset enables us to show

 that (a) PGD-based adversarial examples actually alter features in the data and

 (b) models can learn from human-meaningless/mislabeled training data. The

 D^det\widehat{\mathcal{D}}\_{det}D

det dataset, on the other hand, illustrates that the

 non-robust features are actually sufficient for generalization and can be

 preferred over robust ones in natural settings.

The experiment put forth in the comment is a clever way of showing that such

 leakage is indeed possible. However, we want to stress (as the comment itself

 does) that robust feature leakage does *not* have an impact on our main

 thesis — the D^det\widehat{\mathcal{D}}\_{det}D

det dataset explicitly controls

 for robust

 feature leakage (and in fact, allows us to quantify the models’ preference for

 robust features vs non-robust features — see Appendix D.6 in the

 [paper](https://arxiv.org/abs/1905.02175)).


The Building Blocks of Interpretability

Chris Olah
Arvind Satyanarayan
Ian Johnson
Shan Carter
Ludwig Schubert
Katherine Ye
Alexander Mordvintsev

Abstract: With the growing success of neural networks, there is a  corresponding need to be able to explain their decisions — including  building confidence about how they will behave in the real-world,  detecting model bias, and for scientific curiosity.      In order to do so, we need to both construct deep abstractions and  reify (or instantiate) them in rich interfaces .      With a few exceptions , existing work on interpretability fails to do these in concert.

 With the growing success of neural networks, there is a

corresponding need to be able to explain their decisions — including

building confidence about how they will behave in the real-world,

detecting model bias, and for scientific curiosity.

 In order to do so, we need to both construct deep abstractions and

reify (or instantiate) them in rich interfaces .

 for reasoning about neural networks.

 However, these techniques have been studied as isolated threads of

research, and the corresponding work of reifying them has been

neglected.

 On the other hand, the human-computer interaction community has

begun to explore rich user interfaces for neural networks ,

 but they have not yet engaged deeply with these abstractions.

 To the extent these abstractions have been used, it has been in

fairly standard ways.

 As a result, we have been left with impoverished interfaces (e.g.,

saliency maps or correlating abstract neurons) that leave a lot of value

 on the table.

 Worse, many interpretability techniques have not been fully

actualized into abstractions because there has not been pressure to make

 them generalizable or composable.

 In this article, we treat existing interpretability methods as

fundamental and composable building blocks for rich user interfaces.

 We find that these disparate techniques now come together in a

unified grammar, fulfilling complementary roles in the resulting

interfaces.

 Moreover, this grammar allows us to systematically explore the space

 of interpretability interfaces, enabling us to evaluate whether they

meet particular goals.

 For example, we will see how a network looking at a labrador

retriever detects floppy ears and how that influences its

classification.

 Our interfaces are speculative and one might wonder how reliable they are.

 actively investigating why this is, and hope to uncover principles for

designing interpretable models. In the meantime, while we demonstrate

our techniques on GoogLeNet, we provide code for you to try them on

other models.

 Although here we’ve made a specific choice of task and network, the

basic abstractions and patterns for combining them that we present can

be applied to neural networks in other domains.

Making Sense of Hidden Layers

-----------------------------

 Much of the recent work on interpretability is concerned with a

neural network’s input and output layers.

 Arguably, this focus is due to the clear meaning these layers have:

in computer vision, the input layer represents values for the red,

green, and blue color channels for every pixel in the input image, while

 the output layer consists of class labels and their associated

probabilities.

 However, the power of neural networks lies in their hidden

layers — at every layer, the network discovers a new representation of

the input.

 In computer vision, we use neural networks that run the same feature

 detectors at every position in the image.

 We can think of each layer’s learned representation as a

![dog_cat](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

a4,1 = [0, 0, 0, 25.2, 164.1, 0, 42.7, 4.51, 115.0, 51.3, 0, 0, ...]

{

:

886.

,

:

599.

,

:

328.

,

:

303.

,

... }

There seem to be detectors for

 floppy ears,

 dog snouts,

 cat heads,

 furry legs,

 and

 grass.

[Reproduce in a

 To make a semantic dictionary, we pair every neuron activation with a

 visualization of that neuron and sort them by the magnitude of the

activation.

 This marriage of activations and feature visualization changes our

relationship with the underlying mathematical object.

 Activations now map to iconic representations, instead of abstract

indices, with many appearing to be similar to salient human ideas, such

as “floppy ear,” “dog snout,” or “fur.”

 Semantic dictionaries are powerful not just because they move away

from meaningless indices, but because they express a neural network’s

learned abstractions with canonical examples.

 With image classification, the neural network learns a set of visual

 abstractions and thus images are the most natural symbols to represent

them.

 Were we working with audio, the more natural symbols would most

likely be audio clips.

 This is important because when neurons appear to correspond to human

 ideas, it is tempting to reduce them to words.

 Doing so, however, is a lossy operation — even for familiar

abstractions, the network may have learned a deeper nuance.

 For instance, GoogLeNet has multiple floppy ear detectors that

appear to detect slightly different levels of droopiness, length, and

surrounding context to the ears.

 There also may exist abstractions which are visually familiar, yet

that we lack good natural language descriptions for: for example, take

the particular column of shimmering light where sun hits rippling water.

 Moreover, the network may learn new abstractions that appear alien

to us — here, natural language would fail us entirely!

 In general, canonical examples are a more natural way to represent

the foreign abstractions that neural networks learn than native human

language.

 By bringing meaning to hidden layers, semantic dictionaries set the

stage for our existing interpretability techniques to be composable

building blocks.

 As we shall see, just like their underlying vectors, we can apply

dimensionality reduction to them.

 In other cases, semantic dictionaries allow us to push these

techniques further.

 For example, besides the one-way attribution that we currently

perform with the input and output layers, semantic dictionaries allow us

 to attribute to-and-from specific hidden layers.

 In principle, this work could have been done without semantic

dictionaries but it would have been unclear what the results meant.

 While we introduce semantic dictionaries in terms of neurons, they

can be used with any basis of activations. We will explore

 this more later.

What Does the Network See?

--------------------------

![dog_cat](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

Activation Vector

=

886.

+

599.

+

328.

+

303.

+

...

Channels

 Applying this technique to all the activation vectors allows us to

not only see what the network detects at each position, but also what

the network understands of the input image as a whole.

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d.jpeg)

[Reproduce in a

mixed4d

 And, by working across layers (eg. “mixed3a”, “mixed4d”), we can

observe how the network’s understanding evolves: from detecting edges in

 earlier layers, to more sophisticated shapes and object parts in the

latter.

Zoom In

Zoom Out

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed3a.jpeg)

#### mixed3a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4a.jpeg)

#### mixed4a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d.jpeg)

#### mixed4d

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed5a.jpeg)

#### mixed5a

 These visualizations, however, omit a crucial piece of information:

the magnitude of the activations.

 By scaling the area of each cell by the magnitude of the activation

vector, we can indicate how strongly the network detected features at

that position:

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed3a-magnitude.png)

#### mixed3a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4a-magnitude.png)

#### mixed4a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d-magnitude.png)

#### mixed4d

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed5a-magnitude.png)

#### mixed5a

How Are Concepts Assembled?

---------------------------

 We think there’s a lot of important research to be done on

attribution methods, but for the purposes of this article the exact

approach taken to attribution doesn’t matter.

 We use a fairly simple method, linearly approximating the

relationshipWe do attribution by linear

approximation in all of our interfaces. That is, we estimate the effect

of a neuron on the output is its activation times the rate at which

increasing its activation increases the output. When we talk about a

linear combination of activations, the attribution can be thought of as

the linear combination of the attributions of the units, or equivalently

 as the dot product between the activation of that combination and the

gradient.

 For spatial attribution, we do an additional trick.

GoogLeNet’s strided max pooling introduces a lot of noise and

checkerboard patterns to it’s gradients.

 To avoid our interface demonstrations being dominated by this noise, we

 (a) do a relaxation of the gradient of max pooling, distributing

gradient to inputs proportional to their activation instead of winner

takes all and (b) cancel out the checkerboard patterns.

### Spatial Attribution with Saliency Maps

 We see two weaknesses with this current approach.

 First, it is not clear that individual pixels should be the primary

unit of attribution.

 The meaning of each pixel is extremely entangled with other pixels,

is not robust to simple visual transforms (e.g., brightness, contrast,

etc.), and is far-removed from high-level concepts like the output

class.

 Second, traditional saliency maps are a very limited type of

interface — they only display the attribution for a single class at a

time, and do not allow you to probe into individual points more deeply.

 As they do not explicitly deal with hidden layers, it has been

difficult to fully explore their design space.

 We instead treat attribution as another user interface building

block, and apply it to the hidden layers of a neural network.

 In doing so, we change the questions we can pose.

 Rather than asking whether the color of a particular pixel was

important for the “labrador retriever” classification, we instead ask

 This approach is similar to what Class Activation Mapping (CAM) methods

 do but, because they interpret their results back onto the input image,

 they miss the opportunity to communicate in terms of the rich behavior

of a network’s hidden layers.

#### Input Image

![](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

#### Output Classes

* Labrador retriever

* golden retriever

* tennis ball

* Rhodesian ridgeback

* Appenzeller

#### Output Factors

* Labrador retriever

* golden retriever

* beagle

* kuvasz

* redbone

* tiger

* tiger cat

* lynx

* collie

* Border collie

#### mixed3a

#### mixed4a

#### mixed4d

#### mixed5a

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed3a-nmf.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed3a-magnitude.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4a-nmf.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4a-magnitude.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d-nmf.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed4d-magnitude.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed5a-nmf.png)

![](The%20Building%20Blocks%20of%20Interpretability_files/mixed5a-magnitude.png)

Attribution tends to be more meaningful in later layers.

 The

 floppy ear,

 dog snout,

 cat head,

 etc, do mostly what you expect.

 Surprisingly, the

 lower snout at mixed4d

 seems entangled with the idea of a tennis ball and

 supports "tennis ball" and "granny smith apple."

[Reproduce in a

 The above interface affords us a more flexible relationship with

attribution.

 To start, we perform attribution from each spatial position of each

hidden layer shown to all 1,000 output classes.

 In order to visualize this thousand-dimensional vector, we use

dimensionality reduction to produce a multi-directional saliency map.

 Overlaying these saliency maps on our magnitude-sized activation

grids provides an information scent

 over attribution space.

 The activation grids allow us to anchor attribution to the visual

vocabulary our semantic dictionaries first established.

 On hover, we update the legend to depict attribution to the output

classes (i.e., which classes does this spatial position most contribute

to?).

 On hover, additional saliency maps mask the hidden layers, in a

sense shining a light into their black boxes.

 This type of layer-to-layer attribution is a prime example of how

carefully considering interface design drives the generalization of our

existing abstractions for interpretability.

 With this diagram, we have begun to think of attribution in terms of

 higher-level concepts.

 However, at a particular position, many concepts are being detected

together and this interface makes it difﬁcult to split them apart.

 By continuing to focus on spatial positions, these concepts remain

entangled.

### Channel Attribution

 An alternate way to slice the cube is by channels instead of spatial locations.

#### Input Image

![](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

#### Output Classes

* Labrador retriever

* golden retriever

* tennis ball

* Rhodesian ridgeback

* Appenzeller

#### Top Channels Supporting

 Labrador retriever

mixed3b

...

Showing 3 of **480**mixed4a

...

Showing 3 of **508**mixed4b

...

Showing 3 of **512**mixed4c

...

Showing 3 of **512**mixed4d

...

Showing 3 of **528**

 This diagram is analogous to the previous one we saw: we conduct

layer-to-layer attribution but this time over channels rather than

spatial positions.

 Once again, we use the icons from our semantic dictionary to

represent the channels that most contribute to the final output

classification.

 Hovering over an individual channel displays a heatmap of its

activations overlaid on the input image.

 The legend also updates to show its attribution to the output

classes (i.e., what are the top classes this channel supports?).

 Clicking a channel allows us to drill into the layer-to-layer

attributions, identifying the channels at lower layers that most

contributed as well as the channels at higher layers that are most

supported.

 Attribution to spatial locations and channels can reveal powerful

things about a model, especially when we combine them together.

 Unfortunately, this family of approaches is burdened by two

significant problems.

 On the one hand, it is very easy to end up with an overwhelming

amount of information: it would take hours of human auditing to

understand the long-tail of channels that slightly impact the output.

 On the other hand, both the aggregations we have explored are

extremely lossy and can miss important parts of the story.

 And, while we could avoid lossy aggregation by working with

individual neurons, and not aggregating at all, this explodes the first

problem combinatorially.

Making Things Human-Scale

-------------------------

 In previous sections, we’ve considered three ways of slicing the

cube of activations: into spatial activations, channels, and individual

neurons.

 Each of these has major downsides.

 If one only uses spatial activations or channels, they miss out on

very important parts of the story.

 For example it’s interesting that the floppy ear detector helped us

classify an image as a Labrador retriever, but it’s much more

interesting when that’s combined with the locations that fired to do so.

 One can try to drill down to the level of neurons to tell the whole

story, but the tens of thousands of neurons are simply too much

information.

 Even the hundreds of channels, before being split into individual

neurons, can be overwhelming to show users!

 If we want to make useful interfaces into neural networks, it isn’t

enough to make things meaningful.

 We need to make them human scale, rather than overwhelming dumps of

information.

 The key to doing so is finding more meaningful ways of breaking up

our activations.

 There is good reason to believe that such decompositions exist.

 Often, many channels or spatial positions will work together in a

highly correlated way and are most useful to think of as one unit.

 Other channels or positions will have very little activity, and can

be ignored for a high-level overview.

 So, it seems like we ought to be able to find better decompositions

if we had the right tools.

 There is an entire field of research, called matrix factorization,

that studies optimal strategies for breaking up matrices.

 By flattening our cube into a matrix of spatial locations and

channels, we can apply these techniques to get more meaningful groups of

 neurons.

 These groups will not align as naturally with the cube as the

groupings we previously looked at.

 Instead, they will be combinations of spatial locations and

channels.

 Moreover, these groups are constructed to explain the behavior of a

network on a particular image.

 It would not be effective to reuse the same groupings on another

image; each image requires calculating a unique set of groups.

 In addition to naturally slicing a hidden layer’s cube of

activations into neurons, spatial locations, or channels, we can also

consider more arbitrary groupings of locations and channels.

 The groups that come out of this factorization will be the atoms of

the interface a user works with. Unfortunately, any grouping is

inherently a tradeoff between reducing things to human scale and,

because any aggregation is lossy, preserving information. Matrix

factorization lets us pick what our groupings are optimized for, giving

us a better tradeoff than the natural groupings we saw earlier.

 The goals of our user interface should influence what we optimize

our matrix factorization to prioritize. For example, if we want to

prioritize what the network detected, we would want the factorization to

 fully describe the activations. If we instead wanted to prioritize what

 would change the network’s behavior, we would want the factorization to

 fully describe the gradient. Finally, if we want to prioritize what

caused the present behavior, we would want the factorization to fully

describe the attributions. Of course, we can strike a balance between

these three objectives rather than optimizing one to the exclusion of

the others.

 Most matrix factorization

algorithms and libraries are set up to minimize the mean squared error

of the reconstruction of a matrix you give them. There are ways to hack

such libraries to achieve more general objectives through clever

manipulations of the provided matrix, as we will see below. More

broadly, matrix factorization is an optimization problem, and with

custom tools you can achieve all sorts of custom factorizations.

 with non-negative matrix factorization

 As the name suggests, non-negative

 matrix factorization (NMF) constrains its factors to be positive. This

is fine for the activations of a ReLU network, which must be positive as

 well. Our experience is that the groups we get from NMF seem more

independent and semantically meaningful than those without this

constraint. Because of this constraints, groups from NMF are a less

efficient at representing the activations than they would be without,

but our experience is that they seem more independent and semantically

meaningful.

  .

 Notice how the overwhelmingly large number of neurons has been

reduced to a small set of groups, concisely summarizing the story of the

 neural network.

By

 using non-negative matrix factorization we can reduce the large number

of neurons to a small set of groups that concisely summarize the story

of the network.

[Reproduce in a

#### Input image

#### Activations

 of neuron groups

#### neuron groups based on matrix factorization of mixed4d layer

6 groups

color key

 hover to isolate

#### Effect of neuron groups on output classes

Labrador retrieverbeagletiger catlynxtennis ball

2.249

3.298

-0.350

0.111

0.920

3.755

0.599

-0.994

-0.642

1.336

-1.193

-0.110

-1.607

-0.057

0.152

-1.141

-0.356

0.116

0.117

-0.885

1.117

-0.133

0.248

1.120

1.227

-1.892

-2.618

0.205

0.152

-0.480

 This figure only focuses at a single layer but, as we saw earlier,

it can be useful to look across multiple layers to understand how a

neural network assembles together lower-level detectors into

higher-level concepts.

 The groups we constructed before were optimized to understand a

single layer independent of the others. To understand multiple layers

together, we would like each layer’s factorization to be

“compatible” — to have the groups of earlier layers naturally compose

into the groups of later layers. This is also something we can optimize

the factorization for

 We formalize this “compatibility” in a manner described below,

although we’re not confident it’s the best formalization and won’t be

surprised if it is superseded in future work.

 The basic idea is to split each entry in the activation matrix into *N*

 entries on the channel dimension, spreading the values proportional to

the absolute value of its attribution to the corresponding group.

 Any factorization of this matrix induces a factorization of the

original matrix by collapsing the duplicated entries in the column

factors.

 However, the resulting factorization tries to create separate

factors when the activation of the same channel has different

attributions in different places.

  .

#### Input Image

![](The%20Building%20Blocks%20of%20Interpretability_files/dog_cat.jpeg)

To understand multiple layers together, we would like

each layer's factorization to be "compatible"—to have the groups of

 earlier layers naturally compose into the groups of later

layers. This is also something we can optimize the factorization

 for.

  positive influence

 negative influence

#### Attribution by Factorized Groups

##### Mixed4a

8 groups

##### Mixed4d

6 groups

##### Output class

 Align layer factors

tiger catbeagleLabrador retrieverlynxtennis ball

 In this section, we recognize that the way in which we break apart

the cube of activations is an important interface decision. Rather than

resigning ourselves to the natural slices of the cube of activations, we

 construct more optimal groupings of neurons. These improved groupings

are both more meaningful and more human-scale, making it less tedious

for users to understand the behavior of the network.

 Our visualizations have only begun to explore the potential of

alternate bases in providing better atoms for understanding neural

networks.

 For example, while we focus on creating smaller numbers of

directions to explain individual examples, there’s recently been

The Space of Interpretability Interfaces

----------------------------------------

We can think of an interface as a union of individual elements.

#### Layers

* output

* hidden

* input

#### Atoms

* group

* spatial

* channel

* neuron

#### Content

* activations

* attribution

#### Presentation

* information visualization

* feature visualization

 We think this is a powerful technique and present this grammar as an

initial exploration of how it might apply to interpretability

interfaces.

 `Id = Int

Atom = Neuron | Spatial | Channel | Group | Whole

Layer = Input | Hidden Int | Out

Substrate = Network Id Atom Layer

| Dataset Id

 | Parameters Id

 Content = Substrate

| Activation Substrate

| Attribution Substrate Substrate

| Transform Content *Content?*

Element = InfoVis Content | FeatureVis Content

Interface = [ Element ]`,

 but we find it helpful to think about the space visually.

 We can represent the network’s substrate (which layers we display,

and how we break them apart) as a grid, with the content and style of

presentation plotted on this grid as points and connections.

![](The%20Building%20Blocks%20of%20Interpretability_files/empty.svg)

 For instance, let us consider our teaser figure again.

![](The%20Building%20Blocks%20of%20Interpretability_files/teaser.svg)

**1. Feature visualization**

![](The%20Building%20Blocks%20of%20Interpretability_files/teaser-1.png)

**2. Filter by output attribution**

![](The%20Building%20Blocks%20of%20Interpretability_files/teaser-2.png)

 Next, we filter for specific classes by calculating the output attribution.

**3. Drill down on hover**

![](The%20Building%20Blocks%20of%20Interpretability_files/teaser-3.png)

 Hovering over channels, we get a heatmap of spatial activations.

 figure .teaser-thumb {

 width: 100%;

 border: 30px solid rgb(248, 248, 248);

 border-radius: 10px;

 box-sizing: border-box;

 margin: 12px 0 10px;

 }

 In this article, we have only scratched the surface of

possibilities.

 There are lots of combinations of our building blocks left to

explore, and the design space gives us a way to do so systematically.

 Moreover, each building block represents a broad class of

techniques.

 Our interfaces take only one approach but, as we saw in each

section, there are a number of alternatives for feature visualization,

attribution, and matrix factorization.

 An immediate next step would be to try using these alternate

techniques, and research ways to improve them.

 We can think of dataset examples as another substrate in our design

space, thus becoming another building block that fully composes with the

 others.

 In doing so, we can now imagine interfaces that not only allow us to

 inspect the influence of dataset examples on the final output

classification (as Koh & Liang proposed), but also how examples

influence the features of hidden layers, and how they influence the

relationship between these features and the output.

 For example, if we consider our “Labrador retriever” image, we can

not only see which dataset examples most influenced the model to arrive

at this classification, but also which dataset examples most caused the

“floppy ear” detectors to fire, and which dataset examples most caused

these detectors to increase the “Labrador retriever” classification.

![](The%20Building%20Blocks%20of%20Interpretability_files/dataset.svg)

 A new substrate.

 An interface showing how examples influence the channels of hidden layers.

 An interface for identifying which dataset examples most caused

particular detectors to increase the output classification.

 While most models today are trained to optimize simple objective

functions that one can easily describe, many of the things we’d like

models to do in the real world are subtle, nuanced, and hard to describe

 mathematically.

 An extreme example of the subtle

objective problem is something like “creating interesting art”, but much

 more mundane examples arise more or less whenever humans are involved.

 However, even with human feedback, it may still be hard to train

models to behave the way we want if the problematic aspect of the model

doesn’t surface strongly in the training regime where humans are giving

feedback.

 There are lots of reasons why problematic behavior may not surface

 or may be hard for an evaluator to give feedback on.

 For example, discrimination and bias may be subtly present

throughout the model’s behavior, such that it’s hard for a human

evaluator to critique.

 Or the model may be making a decision in a way that has

problematic consequences, but those consequences never play out in the

problems we’re training it on.

 Human feedback on the model’s decision making process, facilitated

by interpretability interfaces, could be a powerful solution to these

problems.

 (There is however a danger here: we are optimizing our model to look

 the way we want in our interface — if we aren’t careful, this may lead

 Another exciting possibility is interfaces for comparing multiple

models.

 For instance, we might want to see how a model evolves during

training, or how it changes when you transfer it to a new task.

 Or, we might want to understand how a whole family of models

compares to each other.

 Existing work has primarily focused on comparing the output behavior

 One of the unique challenges of this work is that we may want to

align the atoms of each model; if we have completely different models,

can we find the most analogous neurons between them?

 Zooming out, can we develop interfaces that allow us to evaluate

large spaces of models at once?

How Trustworthy Are These Interfaces?

-------------------------------------

 In order for interpretability interfaces to be effective, we must

trust the story they are telling us.

 We perceive two concerns with the set of building blocks we

currently use.

 First, do neurons have a relatively consistent meaning across

different input images, and is that meaning accurately reified by

feature visualization?

 Semantic dictionaries, and the interfaces that build on top of them,

 are premised off this question being true.

 Second, does attribution make sense and do we trust any of the

attribution methods we presently have?

 validated this in a number of ways: we visualized them without a

generative model prior, so that the content of the visualizations

 was causally linked to the neuron firing; we inspected the spectrum

of examples that cause the neuron to fire; and used diversity

 visualizations to try to create different inputs that cause the

neuron to fire.

For more details, see

 Besides these neurons, however, we also found many neurons that do

not have as clean a meaning including “poly-semantic” neurons that

respond to a mixture of salient ideas (e.g., “cat” and “car”).

 There are natural ways that interfaces could respond to this: we

could use diversity visualizations to reveal the variety of meanings the

 neuron can take, or rotate our semantic dictionaries so their

components are more disentangled.

 Of course, just like our models can be fooled, the features that

make them up can be too — including with adversarial examples .

 In fact, it can be interesting to identify when a detector misfires.

 One might even wonder if the idea is fundamentally flawed, since a

function’s output could be the result of non-linear interactions between

 its inputs.

 One way these interactions can pan out is as attribution being

“path-dependent”.

 A natural response to this would be for interfaces to explicitly

surface this information: how path-dependent is the attribution?

 A deeper concern, however, would be whether this path-dependency

dominates the attribution.

 Clearly, this is not a concern for attribution between adjacent

layers because of the simple (essentially linear) mapping between them.

 While there may be technicalities about correlated inputs, we

believe that attribution is on firm grounding here.

 And even with layers further apart, our experience has been that

attribution between high-level features at the output is much more

consistent than attribution to the input — we believe that

path-dependence is not a dominating concern here.

 Model behavior is extremely complex, and our current building blocks

 force us to show only speciﬁc aspects of it.

 An important direction for future interpretability research will be

developing techniques that achieve broader coverage of model behavior.

 But, even with such improvements, we anticipate that a key marker of

 trustworthiness will be interfaces that do not mislead.

 Interacting with the explicit information displayed should not cause

 users to implicitly draw incorrect assessments about the model (we see a

 similar principle articulated by Mackinlay for data visualization).

 Undoubtedly, the interfaces we present in this article have room to

improve in this regard.

 Fundamental research, at the intersection of machine learning and

human-computer interaction, is necessary to resolve these issues.

 Trusting our interfaces is essential for many of the ways we want to

 use interpretability.

 This is both because the stakes can be high (as in safety and

fairness) and also because ideas like training models with

interpretability feedback put our interpretability techniques in the

middle of an adversarial setting.

Conclusion & Future Work

------------------------

 There is a rich design space for interacting with enumerative

algorithms, and we believe an equally rich space exists for interacting

with neural networks.

 We have a lot of work left ahead of us to build powerful and

trustworthy interfaces for interpretability.

 But, if we succeed, interpretability promises to be a powerful tool

in enabling meaningful human oversight and in building fair, safe, and

aligned AI systems.

Thought as a Technology

Visualizing Representations: Deep Learning and Human Beings

Understanding neural networks through deep visualization

Using Artificial Intelligence to Augment Human Intelligence

Visualizing higher-layer features of a deep network

Feature Visualization

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Visualizing and understanding convolutional networks

Striving for simplicity: The all convolutional net

Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization

Interpretable Explanations of Black Boxes by Meaningful Perturbation

PatternNet and PatternLRP--Improving the interpretability of neural networks

The (Un)reliability of saliency methods

Axiomatic attribution for deep networks

Visualizing data using t-SNE

LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent neural networks

ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models

Do convolutional neural networks learn class hierarchy?

Going deeper with convolutions

Deconvolution and Checkerboard Artifacts

Learning deep features for discriminative localization

Information foraging

TCAV: Relative concept importance testing with Linear Concept Activation Vectors

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks

Understanding Black-box Predictions via Influence Functions

Deep reinforcement learning from human preferences

Interactive optimization for steering machine classification

Interacting with predictions: Visual inspection of black-box machine learning models

Modeltracker: Redesigning performance analysis tools for machine learning

Network dissection: Quantifying interpretability of deep visual representations

Intriguing properties of neural networks

Efficient estimation of word representations in vector space

Unsupervised representation learning with deep convolutional generative adversarial networks

Adversarial manipulation of deep representations

Automating the design of graphical presentations of relational information

The Building Blocks of Interpretability.html