Gabriel Goh
Nick Cammarata †
Chelsea Voss †
Shan Carter
Michael Petrov
Ludwig Schubert
Alec Radford
Chris Olah

Abstract: In  2005, a letter published in Nature described human neurons responding  to specific people, such as Jennifer Aniston or Halle Berry .  The exciting thing wasn’t just that they selected for particular  people, but that they did so regardless of whether they were shown  photographs, drawings, or even images of the person’s name. The neurons  were multimodal. As the lead author would put it: "You are looking at  the far end of the transformation from metric, visual shapes to  conceptual… information." Quiroga's full quote, from New Scientist  reads: "I think that’s the excitement to these results. You are looking  at the far end of the transformation from metric, visual shapes to  conceptual memory-related information. It is that transformation that  underlies our ability to understand the world. It’s not enough to see  something familiar and match it. It’s the fact that you plug visual  information into the rich tapestry of memory that brings it to life." We  elided the portion discussing memory since it was less relevant.

#### Contents

* [Emotion Neurons](#emotion-neurons)

* [Region Neurons](#region-neurons)

* [Feature Properties](#feature-properties)

* [Understanding Language](#understanding-language)

* [Emotion Composition](#emotional-intelligence)

* [Faceted Feature Visualization](#faceted-feature-visualization)

* [CLIP Training](#clip)

In

 2005, a letter published in Nature described human neurons responding

to specific people, such as Jennifer Aniston or Halle Berry .

 The exciting thing wasn’t just that they selected for particular

people, but that they did so regardless of whether they were shown

photographs, drawings, or even images of the person’s name. The neurons

were multimodal. As the lead author would put it: "You are looking at

the far end of the transformation from metric, visual shapes to

 reads: "I think that’s the excitement to these results. You are looking

 at the far end of the transformation from metric, visual shapes to

conceptual memory-related information. It is that transformation that

underlies our ability to understand the world. It’s not enough to see

something familiar and match it. It’s the fact that you plug visual

information into the rich tapestry of memory that brings it to life." We

 elided the portion discussing memory since it was less relevant.

We

 report the existence of similar multimodal neurons in artificial neural

 networks. This includes neurons selecting for prominent public figures

 important to note that the vast majority of people these models

recognize don’t have a specific neuron, but instead are represented by a

 combination of neurons. Often, the contributing neurons are

conceptually related. For example, we found a Donald Trump neuron which

fires (albeit more weakly) for Mike Pence, contributing to representing

him.Some of the

neurons we found seem strikingly similar to those described in

neuroscience. A Donald Trump neuron we found might be seen as similar to

 And although we don’t find an exact Jennifer Aniston neuron, we do find

 a neuron for the TV show “Friends” which fires for her.

Like the biological multimodal neurons, these artificial neurons respond

 to the same subject in photographs, drawings, and images of their name:

 neurons only scratch the surface of the highly abstract neurons we've

found. Some neurons seem like topics out of a kindergarten curriculum:

weather, seasons, letters, counting, or primary colors. All of these

features, even the trivial-seeming ones, have rich multimodality, such

We find these multimodal neurons in the recent CLIP models ,

 although it's possible similar undiscovered multimodal neurons may

 authors also kindly shared an alternative version from earlier

experiments, where the training objective was an autoregressive language

 modelling objective, instead of a contrastive objective. The features

seem pretty similar. There are several CLIP models of

varying sizes; we find multimodal neurons in all of them, but focus on

studying the mid-sized RN50-x4 model. We

found it challenging to make feature visualization work on the largest

CLIP models. The reasons why remain unclear. See faceted feature

 for more detailed discussion of CLIP’s architecture and performance.

Our analysis will focus on CLIP's vision side, so when we talk about a

multimodal neuron responding to text we mean the model "reading" text in

 images. The alignment with the text side

of the model might be seen as an additional form of multimodality,

perhaps analogous to a human neuron responding to hearing a word rather

than seeing it (see Quiroga’s later work). But since that is an expected

 result of the training objective, it seems less interesting.

CLIP’s

 abstract visual features might be seen as the natural result of

aligning vision and text. We expect word embeddings (and language models

 generally) to learn abstract "topic" features .

 Either the side of the model which processes captions (the “language

side”) needs to give up those features, or its counterpart, the “vision

side”, needs to build visual analogues. Many

 researchers are interested in “grounding” language models by training

them on tasks involving another domain, in the hope of them learning a

more real world understanding of language. The abstract features we find

 in vision models can be seen as a kind of “inverse grounding”: vision

taking on more abstract features by connection to language. This

 includes some of the classic kinds of bias we see in word embeddings,

such as a “terrorism”/”Islam” neuron, or an “Immigration”/”Mexico”

neuron. See discussion in the [region neurons section](#region-neurons).

 But even if these features seem natural in retrospect, they are

qualitatively different from neurons previously studied in vision models

 (eg. ).

 They also have real world implications: these models are vulnerable to a

 kind of “typographic attack” where adding adversarial text to images

can cause them to be systematically misclassified.

 A typographic attack.

---

A Guided Tour of Neuron Families

--------------------------------

What

 features exist in CLIP models? In this section, we examine neurons

found in the final convolutional layer of the vision side across four

models. A majority of these neurons seem to be interpretable.We

 checked a sample of 50 neurons from this layer and classified them as

interpretable, polysemantic, or uninterpretable. We found that 76% of

the sampled neurons were interpretable. (As a 95% confidence interval,

that’s between 64% and 88%.) A further 18% were polysemantic but with

interpretable facets, and 6% were as yet uninterpretable.

Each layer consists of thousands of neurons, so for our preliminary

analysis we looked at feature visualizations, the dataset examples that

most activated the neuron, and the English words which most activated

the neuron when rastered as images. This revealed an incredible

diversity of features, a sample of which we share below:

 neurons respond to content associated with with a geographic region,

with neurons ranging in scope from entire hemispheres to individual

 this, we mean both that it responds to people presenting as this

gender, as well as that it responds to concepts associated with that

 neurons detect features that an image might contain, whether it's

normal object recognition or detection of more exotic features such as

 despite being able to “read” words and map them to semantic features,

the model keeps a handful of more typographic features in its high-level

 representations. Like a child spelling out a word they don’t know, we

 many of the neurons in the model contribute to recognizing an

incredible diversity of abstract concepts that cannot be cleanly

 neurons respond to any visual information that contextualizes the image

 in a particular time – for some it's a season, for others it's a

 This diagram presents selected neurons from the final layer of four

CLIP models, hand organized into "families" of similar neurons. Each

neuron is represented by a feature visualization (selected from regular

or [faceted feature visualization](#faceted-feature-visualization)

 to best illustrate the neuron) with human-chosen labels to help quickly

 provide a sense of each neuron. Labels were picked after looking at

hundreds of stimuli that activate the neuron, in addition to feature

visualizations.

 You can click on any neuron to open it up in OpenAI Microscope to

see feature visualizations, dataset examples that maximally activate the

 neuron, and more.

These neurons don’t just select for a single

object. They also fire (more weakly) for associated stimuli, such as a

Barack Obama neuron firing for Michelle Obama or a morning neuron firing

 for images of breakfast. They also tend to be maximally inhibited by

stimuli which could be seen, in a very abstract way, as their opposite. Some

 neurons seem less abstract. For example, typographic features like the

“-ing” detector seem to roughly fire based on how far a string is away

in Levenshtein distance. Although, even these show remarkable

generalization, such as responding to different font sizes and rotated

text.

How should we think of these neurons? From an

interpretability perspective, these neurons can be seen as extreme

examples of “multi-faceted neurons” which respond to multiple distinct

 is a classic example in neuroscience of a hypothetical neuron that

responds in a highly specific way to some complex concept or stimulus –

 but this framing might encourage people to overinterpret these

artificial neurons. Instead, the authors generally think of these

neurons as being something like the visual version of a topic feature,

activating for features we might expect to be similar in a word

embedding.

Many of these neurons deal with sensitive topics, from

political figures to emotions. Some neurons explicitly represent or are

closely related to protected characteristics: age, gender, race,

religion, sexual orientation,There’s a

neuron we conceptualize as an LGBT neuron, which responds to the Pride

flag, rainbows, and images of words like “LGBT”. Previous work (Wang

& Kosinski) has suggested that neural networks might be able to

determine sexual orientation from facial structure. This work has since

been thoroughly rebutted and we wish to emphasize that we see no

 neurons related to age and gender, see "person trait neurons." Region

neurons seem closely related to race and national origin, responding to

ethnicities associated with given regions of the world. For sexual

 These neurons may reflect prejudices in the “associated” stimuli they

respond to, or be used downstream to implement biased behavior. There

are also a small number of people detectors for individuals who have

committed crimes against humanity, and a “toxic” neuron which responds

to hate speech and sexual content. Having neurons corresponding to

sensitive topics doesn’t necessarily mean a network will be prejudiced.

You could even imagine explicit representations helping in some cases:

the toxic neuron might help the model match hateful images with captions

 that refute them. But they are a warning sign for a wide range of

possible biases, and studying them may help us find potential biases

which might be less on our radar.Examples

 of bias in AI models, and work drawing attention to it, has helped the

research community to become somewhat “alert” to potential bias with

regards to gender and race. However, CLIP could easily have biases which

 we are less alert to, such as biased behavior towards parents when

there’s a child’s drawing in the background.

CLIP

contains a large number of interesting neurons. To allow detailed

examination we’ll focus on three of the “neuron families” shown above:

people neurons, emotion neurons, and region neurons. We invite you to

explore others in Microscope.

### Person Neurons

This

 section will discuss neurons representing present and historical

figures. Our discussion is intended to be descriptive and frank about

what the model learned from the internet data it was trained on, and is

not endorsement of associations it makes or of the figures discussed,

who include political figures and people who committed crimes against

humanity. This content may be disturbing to some readers.To

caption images on the Internet, humans rely on cultural knowledge. If

you try captioning the popular images of a foreign place, you’ll quickly

 find your object and scene recognition skills aren't enough. You can't

caption photos at a stadium without recognizing the sport, and you may

even need to know specific players to get the caption right. Pictures of

 politicians and celebrities speaking are even more difficult to caption

 if you don’t know who’s talking and what they talk about, and these are

 some of the most popular pictures on the Internet. Some public figures

elicit strong reactions, which may influence online discussion and

captions regardless of other content.

With this in mind, perhaps

it’s unsurprising that the model invests significant capacity in

representing specific public and historical figures — especially those

 detects Christian symbols like crosses and crowns of thorns, paintings

of Jesus, his written name, and feature visualization shows him as a

 recognizes the masked hero and knows his secret identity, Peter Parker.

 It also responds to images, text, and drawings of heroes and villians

 learns to detect his face and body, symbols of the Nazi party, relevant

 historical documents, and other loosely related concepts like German

food. Feature visualization shows swastikas and Hitler seemingly doing a

 Nazi salute.

Which

 people the model develops dedicated neurons for is stochastic, but

seems correlated with the person's prevalence across the datasetThe

 model’s dataset was collected in 2019 and likely emphasizes content

from around that time. In the case of the Donald Trump neuron, it seems

likely there would have also been a Hillary Clinton neuron if data had

 and the intensity with which people respond to them. The one person

we’ve found in every CLIP model is Donald Trump. It strongly responds to

 images of him across a wide variety of settings, including effigies and

 caricatures in many artistic mediums, as well as more weakly activating

 for people he’s worked closely with like Mike Pence and Steve Bannon.

It also responds to his political symbols and messaging (eg. “The Wall”

and “Make America Great Again” hats). On the other hand, it most

\*negatively\* activates to musicians like Nicky Minaj and Eminem, video

games like Fortnite, civil rights activists like Martin Luther King Jr.,

 and LGBT symbols like rainbow flags.

To understand the Trump neuron in more depth, we collected about 650

images that cause it to fire different amounts and labeled them by hand

into categories we created. This lets us estimate the conditional

 for details. As the black / LGBT category contains only a few images,

since they don't occur frequently in the dataset, we validated they

cause negative activations with a futher experimentAs

 we were labeling images for the conditional probability plot in Figure 2

 we were surprised that images related to black and gay rights

consistently caused strong negative activations. However, since there

were four images in that category, we decided to do a follow-up

experiment on more images.

We searched Google Images for the

terms "black rights" and "gay rights" and took ten top images for each

term without looking at their activations. Then we validated these

images reliably cause the Trump neuron to fire in the range of roughly

negative ~3-6 standard deviations from zero. The images that cause less

strong negative activations near -3 standard deviations tend to have

broad symbols such as an image of several black teenagers raising their

arm and fist that causes a -2.5 standard deviations. Conversely, images

of more easy to recognize and specific symbols such as rainbow flags or

photos of Martin Luther King Jr consistently cause activations of at

least -4 standard deviations. In Figure 3 we also show activations

related to photos of Martin Luther King Jr..

 Across

 all categories, we see that higher activations of the Trump neuron are

highly selective, as more than 90% of the images with a standard

deviation greater than 30 are related to Donald Trump.While

 labeling images for the previous experiment it became clear the neuron

activates different amounts for specific people. We can study this more

by searching the Internet for pictures of specific people and measuring

how the images of each person makes the neuron fire.

To see how the Trump neuron responds to different individuals, we

searched the query "X giving a speech at a microphone" for various

individuals on Google Images. We cleaned the data by hand, excluding

photos that are not clear photos of the individual's face. The bar

length for each individual shows the median activation of the person's

photos in standard deviations of the neuron over the dataset, and the

range over the bar shows the standard deviation of the activations of

the person's photos.Presumably, person

neurons also exist in other models, such as facial recognition models.

What makes these neurons unique is that they respond to the person

across modalities and associations, situating them in a cultural

context. In particular, we're struck by how the neuron's response tracks

 an informal intuition with how associated people are. In this sense,

person neurons can be thought of as a landscape of person-associations,

with the person themself as simply the tallest peak.

### Emotion Neurons

This

 section will discuss neurons representing emotions, and a neuron for

“mental illness.” Our discussion is intended to be descriptive and frank

 about what the model learned from the internet data it was trained on

and is not endorsement. This content may be disturbing to some readers.Since

 a small change in someone's expression can radically change the meaning

 of a picture, emotional content is essential to the task of captioning.

 The model dedicates dozens of neurons to this task, each representing a

 different emotion.

These emotion neurons don’t just respond to

facial expressions associated with an emotion -- they’re flexible,

responding to body language and facial expressions in humans and

 activates even when the majority of the face is obscured. It responds

to slang like "OMG!" and "WTF", and text feature visualization produces

similar words of shock and surprise. There are even some emotion neurons

 in discussion of emotions because (1) they are sometimes included in

emotion wheels, (2) they seem to play a role in captioning emotions and

feelings, and (3) being more inclusive in our discussion allow us to

explore more of the model. Of course, these neurons simply

respond to cues associated with an emotion and don’t necessarily

correspond to the mental state of subjects in an image.In

 addition to CLIP neurons potentially incorrectly recognizing cues, they

 cues themselves don’t necessarily reflect people’s mental states. For

example, facial expressions don’t reliably correspond to someone

 addition to these emotion neurons, we also find which neurons respond

to an emotion as a secondary role, but mostly respond to something else.

 which primarily responds to jail and incarceration helps represent

emotions such as “persecuted.” Similarly, a neuron that primarily

detects pornographic content seems to have a secondary function of

to see some of these different facets. In particular, the face facet

shows facial expressions corresponding to different emotions, such as

smiling, crying, or wide-eyed shock. Click on any neuron to open it in

Microscope to see more information, including dataset examples.While

 most emotion neurons seem to be very abstract, there are also some

neurons which simply respond to specific body and facial expressions,

 It activates most to the internet-born duckface expression and peace

signs, and we'll see later that both words show up in the maximally

corresponding captions.

One

 neuron that doesn't represent a single emotion but rather a high level

 This neuron activates when images contain words associated with

negative mental states (eg. “depression,” “anxiety,” “lonely,”

“stressed”), words associated with clinical mental health treatment

(“psychology”, “mental,” “disorder”, “therapy”) or mental health

pejoratives (“insane,” “psycho”). It also fires more weakly for images

of drugs, and for facial expressions that look sad or stressed, and for

the names of negative emotions.

 we wouldn’t think of mental illness as a dimension of emotion. However,

 a couple things make this neuron important to frame in the emotion

context. First, in its low-mid range activations, it represents common

negative emotions like sadness. Secondly, words like “depressed” are

often colloquially used to describe non-clinical conditions. Finally,

we’ll see in a later section that this neuron plays an important role in

 captioning emotions, composing with other emotion neurons to

differentiate “healthy” and “unhealthy” versions of an emotion.

To

 better understand this neuron we again estimated the conditional

probabilities of various categories by activation magnitude. The

strongest positive activations are concepts related to mental illness.

Conversely, the strongest negative activations correspond to activities

like exercise, sports, and music events.

 To understand the "mental illness neuron" in more depth, we collected

images that cause it to fire different amounts and labeled them by hand

into categories we created. This lets us estimate the conditional

 for details. During the labeling process we couldn't see how much it

made the neuron fire. We see that the strongest activations all belong

to labels corresponding to low-valence mental states. On the other hand,

 many images with a negative pre-ReLU activation are of scenes we may

typically consider high-valence, like photos with pets, travel photos,

and or pictures of sporting events.### Region Neurons

This

 section will discuss neurons representing regions of the world, and

indirectly ethnicity. The model’s representations are learned from the

internet, and may reflect prejudices and stereotypes, sensitive regional

 situations, and colonialism. Our discussion is intended to be

descriptive and frank about what the model learned from the internet

data it was trained on, and is not endorsement of the model’s

representations or associations. This content may be disturbing to some

readers.From local weather and food, to travel and immigration,

 to language and race: geography is an important implicit or explicit

context in a great deal of online discourse. Blizzards are more likely

 They respond to a wide variety of modalities and facets associated with

 a given region: country and city names, architecture, prominent public

figures, faces of the most common ethnicity, distinctive clothing,

wildlife, and local script (if not the Roman alphabet). If shown a world

 map, even without labels, these neurons fire selectively for the

relevant region on the map.Map responses

seem to be strongest around distinctive geographic landmarks, such as

the Gulf Of Carpentaria and Cape York Peninsula for Australia, or the

Gulf of Guinea for Africa.

 which responds to bears, moose, coniferous forest, and the entire

Northern third of a world map — down to sub-regions of countries, such

as the US West Coast.One interesting

property of the regional neuron “hierarchy” is that the parent neuron

often doesn’t fire when a child is uniquely implicated. So while the

Europe neuron fires for the names of European cities, the general United

 States neuron generally does not, and instead lets neurons like the

West Coast neuron fire. See also another example of a neuron “hierarchy”

 Some region neurons seem to form more consistently than others. Which

neurons form doesn't seem to be fully explained by prevalence in the

 but not all models seem to have a UK neuron. Why is that? One intuition

 is that there’s more variance in neurons when there’s a natural

supercategory they can be grouped into. For example, when an individual

UK neuron doesn’t exist, it seems to be folded into a Europe neuron. In

Africa, we sometimes see multiple different Africa neurons (in

particular a South/West Africa neuron and an East Africa neuron), while

other times there seems to be a single unified Africa neuron. In

contrast, Australia is perhaps less subdividable, since it’s both a

continent and country.

Geographical Activation of Region Neurons  **Unlabeled map activations**:

 Spatial activations of neurons in response to unlabeled geographical

world map. Activations averaged over random crops. Note that neurons for

 Countries colored by activations of neurons in response to rastered

images of country names. Activations averaged over font sizes, max over

word positions.  **City name activations**:

 Cities colored by activations of neurons in response to rastered images

 of city names. Activations averaged over font sizes, max over word

positions.      Selected Region Neurons Most Activating Words

Hover on a neuron to isolate activations. Click to open in Microscope.

 This diagram contextualizes region neurons with a map.

 Each neuron is mapped to a hue, and then regions where it

activates are colored in that hue, with intensity proportional to

activiation. If multiple neurons of opposing hues fire, the region will

be colored in a desaturated gray.

 It can show their response

 to an unlabeled geographical map,

 to country names,

 and to city names.

 "large region neurons"

 (such as the "Northern Hemisphere" neuron)

 and at

 "secondarily regional neurons"

 "entrepreneurship"

 or

 may not. This means that visualizing behavior on a global map

underrepresents the sheer number of region neurons that exist in CLIP.

Using the top-activating English words as a heuristic, we estimate

around 4% of neurons are regional.To

estimate the fraction of neurons that are regional, we looked at what

fraction of each neuron's top-activating words (ie. words it responds to

 when rastered as images) were explicitly linked to geography, and used

this as a heuristic for whether a neuron was regional. To do this, we

created a list of geographic words consisting of continent / country /

 We found 2.5% (64) of RN50-x4 neurons had geographic words for all of

the five maximally activating words. This number varied between 2-4% in

other CLIP models. However, looking only at neurons for which all top

five words are explicitly geographic misses many region neurons which

respond strongly to words with implicit regional connotations (eg.

“hockey” for a Canada neuron, “volkswagen” for a German neuron, “palm”

for an equatorial neuron). We bucketed neurons by fraction of five most

activating words that are geographic, then estimated the fraction of

each bucket that were regional. With many neurons, the line was quite

blurry (should we include polysemantic neurons where one case is

regional? What about “secondarily regional neurons”?). For a relatively

conservative definition, this seems to get us about 4%, but with a more

liberal one you might get as high as 8%.

 caution is needed in interpreting these neurons as truly regional,

rather than spuriously weakly firing for part of a world map. Important

validations are that they fire for the same region on multiple different

 maps, and if they respond to words for countries or cities in that

region. These neurons don’t have a region as the primary

focus, but have some kind of geographic information baked in, firing

 also find that the linear combination of neurons that respond to Russia

 on a map strongly responds to Pepe the frog, a symbol of white

nationalism in the United States allegedly promoted by Russia. Our

impression is that Russians probably wouldn’t particularly see this as a

 symbol of Russia, suggesting it is more “Russia as understood by the

US.”

#### Case Study: Africa Neurons

Despite these

 examples of neurons learning Americentric caricatures, there are some

areas where the model seems slightly more nuanced than one might fear,

especially given that CLIP was only trained on English language data.

For example, rather than blurring all of Africa into a monolithic

entity, the RN50-x4 model develops neurons for three regions within

Africa. This is significantly less detailed than its representation of

many Western countries, which sometimes have neurons for individual

countries or even sub-regions of countries, but was still striking to

us.It’s important to keep in mind that

the model can represent many more things using combinations of neurons.

Where the model dedicates neurons may give us some sense of the level of

 nuance, but we shouldn’t infer, for example, that it doesn’t somehow

represent individual African countries. To

 contextualize this numerically, the model seems to dedicate ~4% of its

regional neurons to Africa, which accounts for ~20% of the world’s

landmass, and ~15% of the world’s population.

 early explorations it quickly became clear these neurons “know” more

about Africa than the authors. For example, one of the first feature

visualizations of the South African regional neuron drew the text

 Learning about a TV drama might not be the kind of deep insights one

might have envisioned, but it is a charming proof of concept.

We

 chose the East Africa neuron for more careful investigation, again

using a conditional probability plot. It fires most strongly for flags,

 activations tend to follow an exponential distribution in their tails, a

 point that was made to us by Brice Menard. This means that strong

activations are more common than you’d expect in a Gaussian (where the

tail decays at exp(-x^2)), but are much less common than weaker

activations. — have a significantly different distribution

and seems to be mostly about ethnicity. Perhaps this is because

ethnicity is implicit in all images of people, providing weak evidence

for a region, while features like flags are far less frequent, but

provide strong evidence when they do occur. This is the first neuron

we've studied closely with a distinct regime change between medium and

strong activations.

We labeled more than 400 images that causes a neuron that most strongly

responds to the word “Ghana” to fire at different levels of activation,

without access to how much each image caused the neuron to fire while

labeling. See [the appendix](#conditional-probability) for details.

 It fires most strongly for people of African descent as well as African

 words like country names. It’s pre-ReLU activation is negative for

symbols associated with other countries, like the Tesla logo or British

flag, as well as people of non-African descent. Many of its strongest

negative activations are for weaponry such as military vehicles and

handguns. Ghana, the country name it responds to most strongly, has a

Global Peace Index rating higher than most African countries, and

perhaps it learns this anti-association.We

 also looked at the activations of the other two Africa neurons. We

suspect they have interesting differences beyond their detection of

different country names and flags — why else would the model dedicate

three neurons — but we lacked the cultural knowledge to appreciate their

 subtleties.

### Feature properties

So

 far, we’ve looked at particular neurons to give a sense of the kind of

features that exist in CLIP models. It's worth noting several properties

 that might be missed in the discussion of individual features:

**Image-Based Word Embedding:**

 Despite being a vision model, one can produce “image-based word

embeddings” with the visual CLIP model by rastering words into images

and then feeding these images into the model, and then subtracting off

the average over words. Like normal word embeddings, the nearest

Original

Word

Nearest Neighbors

Collobert embeddings

Nearest Neighbors

CLIP image-based embeddings

France

French, Francis, Paris, Les, Des, Sans, Le, Pairs, Notre, Et

Jesus

God, Sati, Christ, Satan, Indra, Vishnu, Ananda, Parvati, Grace

Christ, God, Bible, Gods, Praise, Christians, Lord, Christian, Gospel, Baptist

xbox

Amiga, Playstation, Msx, Ipod, Sega, Ps#, Hd, Dreamcast, Geforce, Capcom

*V(Img(*“King”*)) - V(Img(*“Man”*)) + V(Img(*“Woman”*)) = V(Img(*“Queen”*))*

 work in some cases if we mask non-semantic lexicographic neurons (eg.

“-ing” detectors). It seems likely that mixed arithmetic of words and

images should be possible.

**Limited Multilingual Behavior:**

Although CLIP’s training data was filtered to be English, many features

 responds to images of English “Thank You”, French “Merci”, German

“Danke”, and Spanish “Gracias,” and also to English “Congratulations”,

German “Gratulieren”, Spanish “Felicidades”, and Indonesian “Selamat”.

As the example of Indonesian demonstrates, the model can recognize some

words from non Romance/Germanic languages. However, we were unable to

find any examples of the model mapping words in non-Latin script to

semantic meanings. It can recognize many scripts (Arabic, Chinese,

Japanese, etc) and will activate the corresponding regional neurons, but

 doesn’t seem to be able to map words in those scripts to their

meanings.One interesting question is why

the model developed reading abilities in latin alphabet languages, but

not others. Was it because more data of that type slipped into the

training data, or (the more exciting possibility) because it’s easier to

 learn a language from limited data if you already know the alphabet?

 The most striking examples are likely racial and religious bias. As

 which responds to images of words such as “Terrorism”, “Attack”,

“Horror”, “Afraid”, and also “Islam”, “Allah”, “Muslim”. This isn’t just

 an illusion from looking at a single neuron: the image-based word

embedding for “Terrorist” has a cosine similarity of 0.52 with

“Muslims”, the highest value we observe for a word that doesn’t include

“terror.”By “image-based word embedding”,

 we mean the activation for an image of that word, with the average

activation over images of 10,000 English words subtracted off. The

intuition is that this removes generic “black text on white background”

fatures. If one measures the cosine similarity between “Terrorism” and

“Muslim” without subtracting off the average, it’s much higher at about

0.98, but that’s because all values are shifted up due to sharing the

**Polysemanticity and Conjoined Neurons:**

 Our qualitative experience has been that individual neurons are more

interpretable than random directions; this mirrors observations made in

previous work.Although

 we’ve focused on neurons which seem to have a single clearly defined

concept they respond to, many CLIP neurons are “polysemantic” ,

 responding to multiple unrelated features. Unusually, polysemantic

neurons in CLIP often have suspicious links between the different

 The concepts in these neurons seem “conjoined”, overlapping in a

superficial way in one facet, and then generalizing out in multiple

directions. We haven’t ruled out the possibility that these are just

coincidences, given the large number of facets that could overlap for

each concept. But if conjoined features genuinely exist, they hint at

new potential explanations of polysemanticity.In

 the past, when we've observed seemingly polysemantic neurons, we've

considered two possibilities: either it is responding to some shared

feature of the stimuli, in which case it isn’t really polysemantic, or

it is genuinely responding to two unrelated cases. Usually we

distinguish these cases with feature visualization. For example,

InceptionV1 4e:55 responds to cars and cat heads. One could imagine it

being the case that it’s responding to some shared feature — perhaps cat

 eyes and car lights look similar. But feature visualization establishes

 a facet selecting for a globally coherent cat head, whiskers and all,

as well as the metal chrome and corners of a car. We concluded that it

was genuinely *OR(cat, car)*.

Conjoined features can be seen

 as a kind of mid-point between detecting a shared low-level feature and

 detecting independent cases. Detecting Santa Claus and “turn” are

clearly true independent cases, but there was a different facet where

they share a low-level feature.

Why would models have conjoined

features? Perhaps they’re a vestigial phenomenon from early in training

when the model couldn’t distinguish between the two concepts in that

facet. Or perhaps there’s a case where they’re still hard to

distinguish, such as large font sizes. Or maybe it just makes concept

packing more efficient, as in the superposition hypothesis.

---

Using Abstractions

------------------

We

 typically care about features because they’re useful, and CLIP’s

features are more useful than most. These features, when ensembled,

allow direct retrieval on a variety of queries via the dot product

alone.

Untangling the image into its semantics

 enables the model to perform a wide variety of downstream tasks

including imagenet classification, facial expression detection,

geolocalization and more. How do they do this? Answering these questions

 will require us to look at how neurons work in concert to represent a

broader space of concepts.

### The Imagenet Challenge

To

 study how CLIP classifies Imagenet, it helps to look at the simplest

case. We use a sparse linear model for this purpose, following the

methodology of Radford et al .

 With each class using only 3 neurons on average, it is easy to look at

all of the weights. This model, by any modern standard, fares poorly

with a top-5 accuracy of 56.4%, but the surprising thing is that such a

miserly model can do anything at all. How is each weight carrying so

much weight?

ImageNet

 organizes images into categories borrowed from another project called

WordNet.

Neural networks typically classify images treating ImageNet classes as

structureless labels. But WordNet actually gives them a rich structure

of higher level nodes. For example, a Labrador Retriever is a Canine

which is a Mammal which is an Animal.

We find that the weights and

 neurons of CLIP reflect some of this structure. At the highest levels

we find conventional categories such as

 This diagram visualizes a submatrix of the full weight matrix that

takes neurons in the penultimate layer of Resnet 4x to the imagenet

classes. Each grey circle represents a positive weight. We see the model

 fails in ways that close but incorrect, such as its labeling of

scorpion as a fish.

 arrive at a surprising discovery: it seems as though the neurons appear

 to arrange themselves into a taxonomy of classes that appear to mimic,

very approximately, the imagenet hierarchy. While there have been

attempts to explicitly integrate this information ,

 CLIP was not given this information as a training signal. The fact that

 these neurons naturally form a hierarchy — form a hierarchy without

even being trained on ImageNet — suggests that such hierarchy may be a

universal feature of learning systems.We’ve

 seen hints of similar structure in region neurons, with a whole world

neuron, a northern hemisphere neuron, a USA neuron, and then a West

Coast neuron.

### Understanding Language

The

 most exciting aspect of CLIP is its ability to do zero-shot

classification: it can be “programmed” with natural language to classify

 images into new categories, without fitting a model. Where linear

probes had fixed weights for a limited set of classes, now we have

dynamic weight vectors that can be generated automatically from text.

Indeed, CLIP makes it possible for end-users to ‘roll their own

classifier’ by programming the model via intuitive, natural language

commands - this will likely unlock a broad range of downstream uses of

CLIP-style models.

Recall that CLIP has two sides, a vision side

(which we’ve discussed up to this point) and a language side. The two

sides meet at the end, going through some processing and then performing

 a dot product to create a logit. If we ignore spatial structureIn

 order to use a contrastive loss, the 3d activation tensor of the last

convolutional layer must discard spatial information and be reduced to a

 single vector which can be dot producted with the language embedding.

CLIP does this with an attention layer, first generating attention

 is the text embedding. We focus on the bilinear interaction term, which

 governs local interactions in most directions. Although this

We’ll

 mostly be focusing on using text to create zero-shot weights for

images. But it’s worth noting one tool that the other direction gives

us. If we fix a neuron on the vision side, we can search for the text

that maximizes the logit. We do this with a hill climbing algorithm to

find what amounts to the text maximally corresponding to that neuron.

### Emotion Composition

As

 we see above, English has far more descriptive words for emotions than

the vision side has emotion neurons. And yet, the vision side recognizes

 these more obscure emotions. How can it do that?

We can see what

different emotion words correspond to on the vision side by taking

attribution, as described in the previous section, to "I feel X" on the

language side. This gives us a vector of image neurons for each emotion

word.Since the approximations we made in

the previous section aren’t exact, we double-checked these attribution

vectors for all of the “emotion equations” shown by taking the top image

 neuron in each one, artificially increasing its activation at the last

layer on the vision side when run on a blank image, and confirming that

the logit for the corresponding emotion word increases on the language

 for the prompts "i am feeling {emotion}, "Me feeling {emotion} on my

face", "a photo of me with a {emotion} expression on my face" on each

one of the emotion-words on the emotion-wheel. We assign each prompt a

label corresponding to the emotion-word, and then we then run sparse

logistic regression to find the neurons that maximally discriminate

between the attribution vectors. For the purposes of this article, these

 vectors are then cleaned up by hand by removing neurons that respond to

 bigrams. This may relate to a line of thinking in

psychology where combinations of basic emotions form the “complex

emotions” we experience.The theory of constructed emotion.

For

 example, the jealousy emotion is success + grumpy. Bored is relaxed +

grumpy. Intimate is soft smile + heart - sick. Interested is question

mark + heart and inquisitive is question mark + shocked. Surprise is

celebration + shock.

 physical objects contribute to representing emotions.

For example, part of "powerful" is a lightning neuron, part of

"creative" is a painting neuron, part of "embarrassed" is a neuron

 also see concerning use of sensitive topics in these emotion vectors,

suggesting that problematic spurious correlations are used to caption

expressions of emotion. For instance, "accepted" detects LGBT.

"Confident" detects overweight. "Pressured" detects Asian culture.

 can also search for examples where particular neurons are used, to

explore their role in complex emotions. We see the mental illness neuron

 contributes to emotions like “stressed,” “anxious,” and “mad.”

 far, we’ve only looked at a subset of these emotion words. We can also

see a birds-eye view of this broader landscape of emotions by

visualizing every attribution vector together.

 of complex emotions by applying non-negative matrix factorization to

the emotion attribution vectors and using the factors to color each

cell. The atlas resembles common feeling wheels

 hand-crafted by psychologists to explain the space of human emotions,

indicating that the vectors have a high-level structure that resembles

emotion research in psychology.This atlas has a

 few connections to classical emotion research. When we use just 2

factors, we roughly reconstruct the canonical mood-axes used in much of

psychology: valence and arousal. If we increase to 7 factors, we nearly

reconstruct a well known categorization of these emotions into happy,

surprised, sad, bad, disgusted, fearful, and angry, except with

“disgusted” switched for a new category related to affection that

includes “valued,” “loving,” “lonely,” and “insignificant.”

---

Typographic Attacks

-------------------

As

 we’ve seen, CLIP is full of multimodal neurons which respond to both

images and text for a given concept. Given how strongly these neurons

react to text, we wonder: can we perform a kind of non-programmatic

adversarial attack – a *typographic attack* – simply using handwriting?

To

 test this hypothesis, we took several common items and deliberately

mislabeled them. We then observed how this affects ImageNet

#### No label

#### Labeled “ipod”

#### Labeled “library”

#### Labeled “pizza”

 Physical typographic attacks.

 Above we see the CLIP RN50-4x model's classifications of

objects labeled with incorrect ImageNet classes. Each row corresponds to

 an object, and each column corresponds to a labeling. Some attacks are

more effective than others, and some objects are more resilient to

attack.

 Expand more examples

 Recall that there are two ways to use CLIP for ImageNet

classification: zero-shot and linear probes.

 For this style of attack, we observe that the zero-shot

methodology is somewhat consistently effective, but that the linear

probes methodology is ineffective. Later on, we show an attack style

 Displayed ImageNet classification method:

 Zero-shot

 Linear probes

 Adversarial patches are stickers that can be placed on real-life

objects in order to cause neural nets to misclassify those objects as

something else – for example, as toasters. Physical adversarial examples

 are complete 3D objects that are reliably misclassified from all

perspectives, such as a 3D-printed turtle that is reliably misclassified

 as a rifle. Typographic attacks are both weaker and stronger than

these. On the one hand, they only work for models with multimodal

neurons. On the other hand, once you understand this property of the

### Evaluating Typographic Attacks

Our

 physical adversarial examples are a proof of concept, but they don’t

give us a very good sense of how frequently typographic attacks succeed.

 Duct tape and markers don't scale, so we create an automated setup to

measure the attack’s success rate on the ImageNet validation set.

Target class:

`pizza`

Attack text:

We found text snippets for our attacks in

two different ways. Firstly, we manually looked through the multimodal

model's neurons for those that appear sensitive to particular kinds of

 attacks. Secondly, we brute-force searched through all of the ImageNet

class names looking for short class names which are, in and of

 this setup, we found several attacks to be reasonably effective. The

most successful attacks achieve a 97% attack success rate with only

around 7% of the image's pixels changed. These results are competitive

with the results found in *Adversarial Patch*, albeit on a different model.

| Target class | Attack text | Pixel cover | Success Linear probes |

| --- | --- | --- | --- |

| `waste container` | *trash* | 7.59% | 95.4% |

| `iPod` | *iPod* | 6.8% | 94.7% |

| `rifle` | *rifle* | 6.41% | 91% |

| `pizza` | *pizza* | 8.11% | 92.3% |

| `radio` | *radio* | 7.73% | 77% |

| `great white shark` | *shark* | 8.33% | 62.2% |

| `library` | *library* | 9.95% | 75.9% |

| `Siamese cat` | *meow* | 8.44% | 46.5% |

| `piggy bank` | *$\$\$\$$* | 6.99% | 36.4% |

 **Pixel cover** measures the attack's impact on the original

image: the average percentage of pixels that were changed by any amount

(an L0-norm) in order to add the attack.

 **Success rate** is measured over 1000 ImageNet validation

images with an attack considered to have succeeded if the attack class

is the most likely. We do not consider an attack to have succeeded if

the attack-free image was already classified as the attack class.

### Comparison with the Stroop Effect

 Just as our models make errors when adversarial text is added to

images, humans are slower and more error prone when images have

incongruent labels.

 is harder than normal. To compare CLIP’s behavior to these human

experiments, we had CLIP classify these stimuli by color, using its

zero-shot classification. Unlike humans, CLIP can’t slow down to

compensate for the harder task. Instead of taking a longer amount of

time for the incongruent stimuli, it has a very high error rate.

 A Stroop effect experiment.

 Above we see the CLIP RN50-4x model's classifications of

various words colored with various colors. Activations were gathered

 Expand more examples

---

Appendix: Methodological Details

--------------------------------

### Conditional Probability Plots

If

 we really want to understand the behavior of a neuron, it’s not enough

to look at the cases where it maximally fires. We should look at the

full spectrum: the cases where it weakly fired, the cases where it was

on the border of firing, and the cases where it was strongly inhibited

from firing. This seems especially true for highly abstract neurons,

where weak activations can reveal “associated stimuli,” such as a Donald

 Trump neuron firing for Mike Pence.

Since we have access to a

validation set from the same distribution the model was trained on, we

can sample the distribution of stimuli that cause a certain level of

activation by iterating through the validation set until we find an

image that causes that activation.

To more rigorously characterize

 this, we create a plot showing the conditional probability of various

categories as a function of neuron activation, following the example of

Curve Detectors . To do

 this, we defined uniformly spaced buckets between the maximally

inhibitory and maximally excitatory activation values, and sampled a

fixed number of stimuli for each activation range. Filling in the most

extreme buckets requires checking the neuron activations for millions of

 stimuli. Once we have a full set of stimuli in each bucket, we blind a

labeler to the activation of each stimuli, and have them select salient

categories they observed, informed by the hypothesis we have for the

neuron. The human labeler then categorized each stimuli into these

categories, while blinded to the activation.

We plot the

activation axis in terms of standard deviations of activation from zero,

 since activations have an arbitrary scale. But keep in mind that

activations aren’t Gaussian distributed, and have much thicker tails.

In

 reading these graphs, it’s important to keep in mind that different

activation levels can have many orders of magnitude differences in

probability density. In particular, probability density peaks around

zero and decays exponentially to the tails. This means that false

negatives for a rare category will tend to not be very visible, because

they’ll be crowded out at zero: these graphs show a neuron’s precision,

but not recall. Curve Detectors  discusses these issues in more detail.

An

 alternative possibility is to look at the distribution of activations

conditioned on a category. We take this approach in our second plot for

the Trump neuron. These plots can help characterize how the neuron

responds to rare categories in regions of higher density, and can help

resolve concerns about recall. However, one needs some way to get

samples conditioned on a category for these experiments, and it’s

possible that your process may not be representative. For our purposes,

since these neurons are so high-level, we used a popular image search to

 sample images in a category.

### Faceted Feature Visualization

A neuron is said to have multiple facets

 if it responds to multiple, distinct categories of images. For example,

 a pose-invariant dog-head detector detects dog heads tilted to the

 look for a difference in texture from one side to the other but doesn’t

 care which is which. A neuron may even fire for many different,

unrelated categories of images . We refer to these as polysemantic neurons.

[Feature visualization](https://distill.pub/2017/feature-visualization/)is

 a technique where the input to a neural network is optimized to create a

 stimuli demonstrating some behavior, typically maximizing the

activation of a neuron. Neurons that possess multiple facets present

particular challenges to feature visualization as the multiple facets

are difficult to represent as a single image. When such neurons are

encountered, feature visualization often tries to draw both facets at

once (making it nonsensical), or just reveal one facet The

 difference between the two is believed to be related to the phenomena

of mutual inhibition, see the InceptionV1 pose invariant dog head

circuit.. Both cases are inadequate.

We

 are aware of two past approaches to improving feature visualization for

 multi-faceted neurons. The first approach is to find highly diverse

images that activate a given neuron, and use them as seeds for the

feature visualization optimization process.

 The second tries to combine feature visualization together with a term

that encourages diversity of the activations on earlier layers.

 that allows us to steer the feature visualization towards a particular

theme (e.g. text, logos, facial features, etc), defined by a collection

of images. The procedure works as follows: first we collect examples of

images in this theme, and train a linear probe on the lower layers of

the model to discriminate between those images and generic natural

images. We then do feature visualization by maximizing the penalized

The reader may be curious why we do not maximize f(g(x)) + w^Tg(x)

 instead. We have found that, in practice, the former objective produces

 far higher quality feature visualizations; we believe this is because

Invariant visual representation by single neurons in the human brain

Explicit encoding of multimodal percepts by single neurons in the human brain

Learning Transferable Visual Models From Natural Language Supervision

Deep Residual Learning for Image Recognition

Attention is all you need

Improved deep metric learning with multi-class n-pair loss objective

Contrastive multiview coding

Linear algebraic structure of word senses, with applications to polysemy

Visualizing and understanding recurrent networks

Object detectors emerge in deep scene cnns

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Zoom In: An Introduction to Circuits

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Sparse but not ‘grandmother-cell’ coding in the medial temporal lobe

Concept cells: the building blocks of declarative memory functions

Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements

Geographical evaluation of word embeddings

Using Artificial Intelligence to Augment Human Intelligence

Visualizing Representations: Deep Learning and Human Beings

Natural language processing (almost) from scratch

Linguistic regularities in continuous space word representations

Man is to computer programmer as woman is to homemaker? debiasing word embeddings

Intriguing properties of neural networks

Visualizing higher-layer features of a deep network

Feature Visualization

How does the brain solve visual object recognition?

Imagenet: A large-scale hierarchical image database

BREEDS: Benchmarks for Subpopulation Shift

Global Weighted Average Pooling Bridges Pixel-level Localization and Image-level Classification

Separating style and content with bilinear models

The feeling wheel: A tool for expanding awareness of emotions and increasing spontaneity and intimacy

Activation atlas

Adversarial Patch

Synthesizing Robust Adversarial Examples

Studies of interference in serial verbal reactions.

Curve Detectors

An overview of early vision in inceptionv1

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Sun database: Large-scale scene recognition from abbey to zoo

The pascal visual object classes (voc) challenge

Fairface: Face attribute dataset for balanced race, gender, and age

A style-based generator architecture for generative adversarial networks

Multimodal Neurons in Artificial Neural Networks.html


Visualizing Weights

Chelsea Voss
Nick Cammarata
Gabriel Goh
Michael Petrov
Ludwig Schubert
Ben Egan
Swee Kiat Lim
Chris Olah

Abstract: This article is part of the Circuits thread,  an experimental format collecting invited short articles and critical  commentary delving into the inner workings of neural networks.

![](Visualizing%20Weights_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

[Curve Circuits](https://distill.pub/2020/circuits/curve-circuits/)

Introduction

------------

The problem of understanding a neural network is a little bit

like reverse engineering a large compiled binary of a computer program.

In this analogy, the weights of the neural network are the compiled

assembly instructions. At the end of the day, the weights are the

fundamental thing you want to understand: how does this sequence of

convolutions and matrix multiplications give rise to model behavior?

Trying to understand artificial neural networks also has a lot in

 common with neuroscience, which tries to understand biological neural

networks. As you may know, one major endeavor in modern neuroscience is

mapping the [connectomes](https://en.wikipedia.org/wiki/Connectome)

 of biological neural networks: which neurons connect to which. These

connections, however, will only tell neuroscientists which weights are

non-zero. Getting the weights – knowing whether a connection excites or

inhibits, and by how much – would be a significant further step. One

imagines neuroscientists might give a great deal to have the access to

weights that those of us studying artificial neural networks get for

free.

And so, it’s rather surprising how little attention we actually

give to looking at the weights of neural networks. There are a few

exceptions to this, of course. It’s quite common for researchers to show

 pictures of the first layer weights in vision models

 (these are directly connected to RGB channels, so they’re easy to

understand as images). In some work, especially historically, we see

researchers reason about the weights of toy neural networks by hand. And

 we quite often see researchers discuss aggregate statistics of weights.

 But actually looking at the weights of a neural network other than the

first layer is quite uncommon – to the best of our knowledge, mapping

weights between hidden layers to meaningful algorithms is novel to the

circuits project.

In this article, we’re focusing on visualizing weights. But

people often visualize activations, attributions, gradients, and much

more. How should we think about the meaning of visualizing these

different objects?

* **Activations:** We generally think of these as being “what”

the network saw. If understanding a neural network is like reverse

compiling a computer program, the neurons are the variables, and the

activations are the values of those variables.

* **Weights:** We generally think of these as being “how” the

neural network computes one layer from the previous one. In the reverse

engineering analogy, these are compiled assembly instructions.

 often think of this as “why” the neuron fired. We need to be careful

with attributions, because they’re a human-defined object on top of a

neural network rather than a fundamental object. They aren’t always well

 defined, and people mean different things by them. (They are very well

defined if you are only operating across adjacent layers!)

Why it’s non-trivial to study weights in hidden layers

------------------------------------------------------

It seems to us that there are three main barriers to making sense

 of the weights in neural networks, which may have contributed to

researchers tending to not directly inspect them:

* **Lack of Contextualization:** Researchers often visualize

weights in the first layer, because they are linked to RGB values that

we understand. That connection makes weights in the first layer

meaningful. But weights between hidden layers are meaningless by

default: knowing nothing about either the source or the destination, how

 can we make sense of them?

* **Indirect Interaction:** Sometimes, the meaningful weight

interactions are between neurons which aren’t literally adjacent in a

neural network. For example, in a residual network, the output of one

neuron can pass through the additive residual stream and linearly

interact with a neuron much later in the network. In other cases,

neurons may interact through intermediate neurons without significant

nonlinear interactions. How can we efficiently reason about these

interactions?

* **Dimensionality and Scale:** Neural networks have lots of

neurons. Those neurons connect to lots of other neurons. There’s a lot

of data to display! How can we reduce it to a human-scale amount of

information?

 The goal of this article is to show how similar ideas can be applied to

 weights instead of activations. Of course, we’ve already implicitly

used these methods in various circuit articles,

 but in those articles the methods have been of secondary interest to

the results. It seems useful to give some dedicated discussion to the

methods.

Aside: One Simple Trick

-----------------------

Interpretability methods often fail to take off because they’re

hard to use. So before diving into sophisticated approaches, we wanted

to offer a simple, easy to apply method.

 is large. (If this is the first convolutional layer, visualize it as

![](Visualizing%20Weights_files/screenshot_1.png)

[1](#figure-1):

 NMF of input weights in InceptionV1 `mixed4d_5x5`,

for a selection of ten neurons. The red, green, and blue channels on

each grid indicate the weights for each of the 3 NMF factors.

This visualization doesn’t tell you very much about what your

weights are doing in the context of the larger model, but it does show

you that they are learning nice spatial structures. This can be an easy

sanity check that your neurons are learning, and a first step towards

understanding your neuron’s behavior. We’ll also see later that this

general approach of factoring weights can be extended into a powerful

tool for studying neurons.

Despite this lack of contextualization, one-sided NMF can be a

great technique for investigating multiple channels at a glance. One

thing you may quickly discover using this method is that, in models with

 global average pooling at the end of their convolutional layers, the

last few layers will have all their weights be horizontal bands.

![](Visualizing%20Weights_files/screenshot_2.png)

[2](#figure-2):

 Horizontally-banded weights in InceptionV1 `mixed5b_5x5`,

 for a selection of eight neurons. As in Figure 1, the red, green, and

blue channels on each grid indicate the weights for each of the 3 NMF

factors.

Contextualizing Weights with Feature Visualization

--------------------------------------------------

 The challenge of contextualization is a recurring challenge in

understanding neural networks: we can easily observe every activation,

every weight, and every gradient; the challenge lies in determining what

 those values represent.

`[relative x position, relative y position,

 input channels, output channels]`

If we fix the input channel and the output channel, we get a 2D

array we can present with traditional data visualization. Let’s assume

we know which neuron we’re interested in understanding, so we have the

output channel. We can pick the input channels with high magnitude

weights to our output channel.

But what does the input represent? What about the output?

The key trick is that techniques like feature visualization

 (or deeper investigations of neurons) can help us understand what the

input and output neurons represent, contextualizing the graph. Feature

visualizations are especially attractive because they’re automatic, and

produce a single image which is often very informative about the neuron.

 As a result, we often represent neurons as feature visualizations in

weight diagrams.

[3](#figure-3): Contextualizing weights.

We can liken this to how, when reverse-engineering a normal

compiled computer program, one would need to start assigning variable

names to the values stored in registers to keep track of them. Feature

visualizations are essentially automatic variable names for neurons,

which are roughly analogous to those registers or variables.

### Small Multiples

And if we have two families of related neurons interacting, it

can sometimes even be helpful to show the weights between all of them as

 a grid of small multiples:

Advanced Approaches to Contextualization with Feature Visualization

-------------------------------------------------------------------

Although we most often use feature visualization to visualize

neurons, we can visualize any direction (linear combination of neurons).

 This opens up a very wide space of possibilities for visualizing

weights, of which we’ll explore a couple particularly useful ones.

### Visualizing Spatial Position Weights

 matrix. But an alternative approach is to think of there as being a

vector over input neurons at each spatial position, and to apply feature

 visualization to each of those vectors. You can think of this as

telling us what the weights in that position are collectively looking

for.

![](Visualizing%20Weights_files/screenshot_6.png)

[6](#figure-6). **Left:** Feature visualization of a car neuron. **Right:**

 Feature visualizations of the vector over input neurons at each spatial

 position of the car neuron’s weights. As we see, the car neuron broadly

 responds to window features above wheel features.

 from Building Blocks. It can be a nice, high density way to get an

overview of what the weights for one neuron are doing. However, it will

be unable to capture cases where one position responds to multiple very

different things, as in a multi-faceted or polysemantic neuron.

### Visualizing Weight Factors

Feature visualization can also be applied to factorizations of

the weights, which we briefly discussed earlier. This is the weight

analogue to the “Neuron Groups” visualization from Building Blocks.

 or black and white vs color detectors that look are all mostly looking

for a small number of factors. For example, a large number of high-low

frequency detectors can be significantly understood as combining just

two factors – a high frequency factor and a low-frequency factor – in

different patterns.

 .upstream-nmf {

 display: grid;

 grid-row-gap: .5rem;

 margin-bottom: 2rem;

 }

 .upstream-nmf .row {

 display: grid;

 grid-template-columns: min-content 1fr 6fr;

 grid-column-gap: 1rem;

 grid-row-gap: .5rem;

 }

 .units,

 .weights {

 display: grid;

 grid-template-columns: repeat(6, 1fr);

 grid-gap: 0.5rem;

 grid-column-start: 3;

 }

 img.fv {

 max-width: 100%;

 border-radius: 8px;

 }

 div.units img.full {

 margin-left: 1px;

 margin-bottom: 0px;

 }

 img.full {

 width: unset;

 object-fit: none;

 object-position: center;

 image-rendering: optimizeQuality;

 }

 img.weight {

 width: 100%;

 image-rendering: pixelated;

 align-self: center;

 border: 1px solid #ccc;

 }

 .annotated-image {

 display: grid;

 grid-auto-flow: column;

 align-items: center;

 }

 .annotated-image span {

 writing-mode: vertical-lr;

 }

 .layer-label {

 grid-row-start: span 2;

 text-align: end;

 }

 .layer-label label {

 display: inline-block;

 writing-mode: vertical-lr;

 }

 .layer-label.hidden {

 border-color: transparent;

 }

 .layer-label.nonhidden {

 margin-left: 1rem;

 }

 .layer-label.hidden label {

 visibility: hidden;

 }

mixed3a

HF-factor

![](Visualizing%20Weights_files/conv2d2-hi.png)

![](Visualizing%20Weights_files/neuron136-layermaxpool1-factor1.png)

![](Visualizing%20Weights_files/neuron108-layermaxpool1-factor1.png)

![](Visualizing%20Weights_files/neuron132-layermaxpool1-factor1.png)

![](Visualizing%20Weights_files/neuron88-layermaxpool1-factor1.png)

![](Visualizing%20Weights_files/neuron110-layermaxpool1-factor1.png)

![](Visualizing%20Weights_files/neuron180-layermaxpool1-factor1.png)

LF-factor

![](Visualizing%20Weights_files/conv2d2-lo.png)

![](Visualizing%20Weights_files/neuron136-layermaxpool1-factor0.png)

![](Visualizing%20Weights_files/neuron108-layermaxpool1-factor0.png)

![](Visualizing%20Weights_files/neuron132-layermaxpool1-factor0.png)

![](Visualizing%20Weights_files/neuron88-layermaxpool1-factor0.png)

![](Visualizing%20Weights_files/neuron110-layermaxpool1-factor0.png)

![](Visualizing%20Weights_files/neuron180-layermaxpool1-factor0.png)

[7](#figure-7):

 layer `conv2d2`.

 .upstream-neurons {

 display: grid;

 grid-gap: 1em;

 margin-bottom: 1em;

 }

 h5 {

 margin-bottom: 0px;

 }

 .upstream-neurons .row {

 display: grid;

 grid-column-gap: .25em;

 column-gap: .25em;

 align-items: center;

 }

 .units,

 .weights {

 display: grid;

 grid-template-columns: repeat(6, 1fr);

 grid-gap: 0.5rem;

 grid-column-start: 3;

 }

 img.fv {

 display: block;

 max-width: 100%;

 border-radius: 8px;

 }

 img.full {

 width: unset;

 object-fit: none;

 object-position: center;

 image-rendering: optimizeQuality;

 }

 img.weight {

 width: 100%;

 image-rendering: pixelated;

 align-self: center;

 border: 1px solid #ccc;

 }

 .layer-label {

 grid-row-start: span 2;

 }

 .layer-label label {

 display: inline-block;

 /\* transform: rotate(-90deg); \*/

 }

 .annotation {

 font-size: 1.5em;

 font-weight: 200;

 color: #666;

 margin-bottom: 0.2em;

 }

 .equal-sign {

 padding: 0 0.25em;

 }

 .ellipsis {

 padding: 0 0.25em;

 /\* vertically align the ellipsis \*/

 position: relative;

 bottom: 0.5ex;

 }

 .unit {

 display: block;

 min-width: 50px;

 }

 .factor {

 box-shadow: 0 0 8px #888;

 }

 .unit .bar {

 display: block;

 margin-top: 0.5em;

 background-color: #CCC;

 height: 4px;

 }

 .row h4 {

 border-bottom: 1px solid #ccc;

 }

![](Visualizing%20Weights_files/conv2d2-hi.png)

=

+

+

+

+

+

…

HF-factor

 × 0.93

 × 0.73

 × 0.66

 × 0.59

 × 0.55

![](Visualizing%20Weights_files/conv2d2-lo.png)

=

+

+

+

+

+

…

LF-factor

 × 0.44

 × 0.41

 × 0.38

 × 0.36

 × 0.34

 each factor.

Dealing with Indirect Interactions

----------------------------------

As we mentioned earlier, sometimes the meaningful weight

interactions are between neurons which aren’t literally adjacent in a

neural network, or where the weights aren’t directly represented in a

single weight tensor. A few examples:

* In a residual network, the output of one neuron can pass

through the additive residual stream and linearly interact with a neuron

 much later in the network.

* In a bottleneck architecture, neurons in the bottleneck may

primarily be a low-rank projection of neurons from the previous layer.

* The behavior of an intermediate layer simply doesn’t introduce

 much non-linear behavior, leaving two neurons in non-adjacent layers

with a significant linear interaction.

As a result, we often work with “expanded weights” – that is, the

 result of multiplying adjacent weight matrices, potentially ignoring

non-linearities. We generally implement expanded weights by taking

gradients through our model, ignoring or replacing all non-linear

operations with the closest linear one.

These expanded weights have the following properties:

* If two layers interact **linearly**, the expanded weights

will give the true linear map, even if the model doesn’t explicitly

represent the weights in a single weight matrix.

* If two layers interact **non-linearly**, the expanded

weights can be seen as the expected value of the gradient up to a

constant factor, under the assumption that all neurons have an equal

(and independent) probability of firing.

They also have one additional benefit, which is more of an

implementation detail: because they’re implemented in terms of

gradients, you don’t need to know how the weights are represented. For

example, in TensorFlow, you don’t need to know which variable object

represents the weights. This can be a significant convenience when

you’re working with unfamiliar models!

### Benefits of Expanded Weights

Expanding out the weights allows us to see an important aggregate

 effect of these connections: together, they look for the absence of

color in the center one layer further back.

A particularly important use of this method – which we’ve been

implicitly using in earlier examples – is to jump over “bottleneck

layers.” Bottleneck layers are layers of the network which squeeze the

number of channels down to a much smaller number, typically in a branch,

 of InceptionV1 are one example. Since so much information is

compressed, these layers are often polysemantic, and it can often be

more helpful to jump over them and understand the connection to the

wider layer before them.

### Cases where expanded weights are misleading

 excited by high-frequency patterns on one side and inhibited on the

other (and vice versa for low frequency), detecting both directions

means that the expanded weights cancel out! As a result, expanded

weights appear to show that boundary detectors are neither excited or

inhibited by high frequency detectors two layers back, when in fact they

 are *both* excited and also inhibited by high frequency, depending

 on the context, and it’s just that those two different cases cancel

out.

[12](#figure-12).

More sophisticated techniques for describing multi-layer

interactions can help us understand cases like this. For example, one

can determine what the “best case” excitation interaction between two

neurons is (that is, the maximum achievable gradient between them). Or

you can look at the gradient for a particular example. Or you can factor

 the gradient over many examples to determine major possible cases.

These are all useful techniques, but we’ll leave them for a future

article to discuss.

### Qualitative properties

One qualitative property of expanding weights across many layers

deserves mention before we end our discussion of them. Expanded weights

often get this kind of “electron orbital”-like smooth spatial

structures:

 Although the exact structures present may vary from neuron to

neuron, this example is not cherry-picked: this smoothness is typical of

 most multiple-layer expanded weights.

Dimensionality and Scale

------------------------

So far, we’ve addressed the challenges of contextualization and

indirection interactions. But we’ve only given a bit of attention to our

 third challenge of dimensionality and scale. Neural networks contain

many neurons and each one connects to many others, creating a huge

amount of weights. How do we pick which connections between neurons to

look at?

For the purposes of this article, we’ll put the question of which

 neurons we want to study outside of our scope, and only discuss the

problem of picking which connections to study. (We may be trying to

comprehensively study a model, in which case we want to study all

neurons. But we might also, for example, be trying to study neurons

we’ve determined related to some narrower aspect of model behavior.)

Generally, we chose to look at the largest weights, as we did at

the beginning of the section on contextualization. Unfortunately, there

tends to be a long tail of small weights, and at some point it generally

 gets impractical to look at these. How much of the story is really

hiding in these small weights? We don’t know, but polysemantic neurons

suggest there could be a very important and subtle story hiding here!

There’s some hope that sparse neural networks might make this much

better, by getting rid of small weights, but whether such conclusions

can be drawn about non-sparse networks is presently speculative.

An alternative strategy that we’ve brushed on a few times is to

reduce your weights into a few components and then study those factors

(for example, with NMF). Often, a very small number of components can

explain much of the variance. In fact, sometimes a small number of

factors can explain the weights of an entire set of neurons! Prominent

examples of this are high-low frequency detectors (as we saw earlier)

and black and white vs color detectors.

However, this approach also has downsides. Firstly, these

components can be harder to understand and even polysemantic. For

example, if you apply the basic version of this method to a boundary

detector, one component will contain both high-to-low and low-to-high

frequency detectors which will make it hard to analyze. Secondly, your

factors no longer align with activation functions, which makes analysis

much messier. Finally, because you will be reasoning about every neuron

in a different basis, it is difficult to build a bigger picture view of

the model unless you convert your components back to neurons.

![](Visualizing%20Weights_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

[Curve Circuits](https://distill.pub/2020/circuits/curve-circuits/)

Imagenet classification with deep convolutional neural networks

Understanding neural networks through deep visualization

Visualizing and understanding convolutional networks

The Building Blocks of Interpretability

Zoom In: An Introduction to Circuits

An Overview of Early Vision in InceptionV1

Curve Detectors

Feature Visualization

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Visualizing and understanding recurrent networks

Visualizing higher-layer features of a deep network


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Example Researchers Need to Expand What is Meant by 'Robustness'

Abstract: This article is part of a discussion of the Ilyas et al. paper         “Adversarial examples are not bugs, they are features”.         You can learn more in the                      main discussion article         .

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

Detailed Response

-----------------

 commonly accepted in the robustness to distributional shift literature

: a model’s lack of

[1](https://distill.pub/2019/advex-bugs-discussion/response-1/#figure-1)

 high frequency features.

 Fourier basis vector,

  from 85.7% to 55.3%. Adversarial training similarly degrades robustness to

 contrast and low-pass

 settings.

[2](https://distill.pub/2019/advex-bugs-discussion/response-1/#figure-2)

 an

 is smaller compared to that of the naturally trained model).

 detached from security and real-world robustness . While often thought an

 idiosyncratic quirk of deep

 to noise . They should not be surprising given the brittleness observed in

 numerous synthetic — and even

 natural  — conditions. Models reliably exhibit poor performance when they are

 evaluated on distributions

 comprehensive benchmarks accordingly . As long as models lack

 robustness to

 distributional shift, there will always be errors to find adversarially.

 To cite Ilyas et al.’s response, please cite their

**Response Summary**: The demonstration of models that learn from

 high-frequency components of the data is interesting and nicely aligns with our

 findings. Now, even though susceptibility to noise could indeed arise from

 of ML models has been so far predominantly viewed as a consequence of model

 “bugs” that will be eliminated by “better” models. Finally, we agree that our

 models need to be robust to a much broader set of perturbations — expanding the

 set of relevant perturbations will help identify even more non-robust features

 and further distill the useful features we actually want our models to rely on.

**Response**: The fact that models can learn to classify correctly based

 purely on the high-frequency component of the training set is neat! This nicely

 Also, while non-robustness to noise can be an indicator of models using

 More often than not, the brittleness of ML models to noise was instead regarded

 expected that progress towards “better”/”bug-free” models will lead to them

 being more robust to noise and adversarial examples.

 small subset of the perturbations we want our models to be robust to. Note,

 however, that the focus of our work is human-alignment — to that end, we

 demonstrate that models rely on features sensitive to patterns that are

 imperceptible to humans. Thus, the existence of other families of

 incomprehensible but useful features would provide even more support for our

 future research.

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Measuring the tendency of CNNs to Learn Surface Statistical Regularities

Nightmare at Test Time: Robust Learning by Feature Deletion

A Robust Minimax Approach to Classification

Generalisation in humans and deep neural networks

A Fourier Perspective on Model Robustness in Computer Vision

Motivating the Rules of the Game for Adversarial Example Research

Adversarial Examples Are a Natural Consequence of Test Error in Noise

Robustness of classifiers: from adversarial to random noise

Natural Adversarial Examples

{MNIST-C:} {A} Robustness Benchmark for Computer Vision

{NICO:} {A} Dataset Towards Non-I.I.D. Image Classification

Do ImageNet Classifiers Generalize to ImageNet?

The Elephant in the Room

Using Videos to Evaluate Image Model Robustness

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'_ Adversarial Example Researchers Need to Expand What is Meant by 'Robustness'.html


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Learning from Incorrectly Labeled Data

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

[Comment by Ilyas et al.](#rebuttal)

 about the trained model is being “leaked” into the dataset.

 Nevertheless, we show that this intuition fails — a model *can* generalize.

 examples on the left of the Figure below:

[1](#figure-1)

 incorrectly labeled, unperturbed images but can still non-trivially generalize.

This is Model Distillation Using Incorrect Predictions

------------------------------------------------------

 of model distillation — training on this dataset allows a new

 model to somewhat recover the features of the original model.

 another task.

### Two-dimensional Illustration of Model Distillation

 (panel (a) in the Figure below).

[2](#figure-2)

 panel (c).

 predictions.

###

 Other Peculiar Forms of Distillation

 we learn about the original model? Can we use only *out-of-domain* data?

 labeled as an “8″.

[3](#figure-3)

 training on erroneous predictions.

###

 Summary

 Ilyas et al. (2019) are not necessary to enable learning.

 To cite Ilyas et al.’s response, please cite their

**Response

 Summary**: Note that since our experiments work across different architectures,

 “distillation” in weight space does not occur. The only distillation that can

 have been able to “distill” a useful model from them. (In fact, one might think

 of normal model training as just “feature distillation” of the humans that

 labeled the dataset.) Furthermore, the hypothesis that all we need is enough

 model-consistent points in order to recover a model, seems to be disproven by

 and other (e.g. ) settings.

**Response**: Since our experiments work across different architectures,

 “distillation” in weight space cannot arise. Thus, from what we understand, the

 “distillation” hypothesis suggested here is referring to “feature distillation”

 (i.e. getting models which use the same features as the original), which is

 actually precisely our hypothesis too. Notably, this feature distillation would

 are good for classification (see [World

 1](https://distill.pub/2019/advex-bugs-discussion/original-authors/#world1) and

 model would only use features that generalize poorly, and would thus generalize

 poorly itself.

 Moreover, we would argue that in the experiments presented (learning from

 mislabeled data), the same kind of distillation is happening. For instance, a

 moderately accurate model might associate “green background” with “frog” thus

 labeling “green” images as “frogs” (e.g., the horse in the comment’s figure).

 Training a new model on this dataset will thus associate “green” with “frog”

 from Fashion-MNIST” experiment in the comment). This corresponds exactly to

 learning features from labels, akin to how deep networks “distill” a good

 decision boundary from human annotators. In fact, we find these experiments

 a very interesting illustration of feature distillation that complements

 our findings.

 We also note that an analogy to logistic regression here is only possible

 due to the low VC-dimension of linear classifiers (namely, these classifiers

 have dimension ddd). In particular, given any classifier with VC-dimension

 networks have been shown to have extremely large VC-dimension (in particular,

 bigger than the size of the training set ). So even though

 labelling d+1d+1d+1 random

 points model-consistently is sufficient to recover a linear model, it is not

 necessarily sufficient to recover a deep neural network. For instance, Milli et

 al.  are not able to reconstruct a ResNet-18

 using only its predictions on random Gaussian inputs. (Note that we are using a

 ResNet-50 in our experiments.)

 Finally, it seems that the only potentially problematic explanation for

 our experiments (namely, that enough model-consistent points can recover a

 In particular, Preetum is able to design a

 dataset where training on mislabeled inputs *that are model-consistent*

 does not at all recover the decision boundary of the original model. More

 generally, the “model distillation” perspective raised here is unable to

 distinguish between the dataset created by Preetum below, and those created

 with standard PGD (as in our D^det\widehat{\mathcal{D}}\_{det}D

det and

 D^rand\widehat{\mathcal{D}}\_{rand}D

rand datasets).

Distilling the Knowledge in a Neural Network

Model reconstruction from model explanations

Understanding deep learning requires rethinking generalization

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features' Learning from Incorrectly Labeled Data.html


Logan Engstrom
Justin Gilmer
Gabriel Goh
Dan Hendrycks
Andrew Ilyas
Aleksander Madry
Reiichiro Nakano
Preetum Nakkiran
Shibani Santurkar
Brandon Tran
Dimitris Tsipras
Eric Wallace

Abstract: On May 6th, Andrew Ilyas and colleagues published a paper  outlining two sets of experiments.             Firstly, they showed that models trained on adversarial examples can transfer to real data,             and secondly that models trained on a dataset derived from the representations of robust neural networks             seem to inherit non-trivial robustness.             They proposed an intriguing interpretation for their results:             adversarial examples are due to “non-robust features” which are highly predictive but imperceptible to             humans.

 outlining two sets of experiments.

 seem to inherit non-trivial robustness.

 They proposed an intriguing interpretation for their results:

 humans.

 The paper was received with intense interest and discussion

 on social media, mailing lists, and reading groups around the world.

 How should we interpret these experiments?

 Would they replicate?

 disciplines of machine learning,

 because it requires researchers to play both attack and defense.

 It’s easy for even very rigorous researchers to accidentally use a weak attack.

 However, as we’ll see, Ilyas et al’s results have held up to initial scrutiny.

 And if non-robust features exist… what are they?

 Why not just have everyone write private blog posts like Ferenc?

 can give more researchers license to invest energy in discussing other’s work

 published.

 We invited a number of researchers

 The Machine Learning community

 [sometimes](https://www.machinelearningdebates.com/program)

 that peer review isn’t thorough enough.

 In contrast to this, we were struck by how deeply respondents engaged.

 deeply about the original paper.

 and forth!

 even running new experiments in response to comments.

 discussion articles in the future.

Discussion Themes

-----------------

**Clarifications**:

 Discussion between the respondents and original authors was able

 to surface several misunderstandings or opportunities to sharpen claims.

 The original authors summarize this in their rebuttal.

**Successful Replication**:

 the non-robust dataset experiments.

 Preetum reproduced the D^det\widehat{\mathcal{D}}\_{det}D

det non-robust dataset experiment as described in the

 paper, for L∞L\_\inftyL∞ and L2L\_2L2 attacks.

 Gabriel repproduced both D^det\widehat{\mathcal{D}}\_{det}D

det and D^rand\widehat{\mathcal{D}}\_{rand}D

rand for L2L\_2L2

 attacks.

 Preetum also replicated part of the robust dataset experiment by

 and hyperparameters he tried.

**Exploring the Boundaries of Non-Robust Transfer**:

 where training on adversarial examples transfers to real data.

 When, how, and why does it happen?

 Gabriel Goh explores an alternative mechanism for the results,

 Preetum Nakkiran shows a special construction where it doesn’t happen,

**Properties of Robust and Non-Robust Features**:

 distribution shift,

Comments

--------

 Distill collected six comments on the original paper.

 They are presented in alphabetical order by the author’s last name,

###

[Adversarial Example Researchers Need to Expand What is Meant by

 “Robustness”](https://distill.pub/2019/advex-bugs-discussion/response-1/)

### Authors

### Affiliations

[Justin Gilmer](https://www.linkedin.com/in/jmgilmer)

[Google Brain Team](https://g.co/brain)

[Dan Hendrycks](https://people.eecs.berkeley.edu/~hendrycks/)

[UC Berkeley](https://www.berkeley.edu/)

 Justin and Dan discuss “non-robust features” as a special case

 of models being non-robust because they latch on to superficial correlations,

 a view often found in the distributional robustness literature.

 They emphasize we should think about a broader notion of robustness.

#### Comment from original authors:

 appears “meaningless” to humans.

 to rely on.

###

### Authors

### Affiliations

[Gabriel Goh](https://gabgoh.github.io/)

[OpenAI](https://openai.com/)

 results.

 D^rand\widehat{\mathcal{D}}\_{rand}D

rand experiment,

 but finds no evidence for it effecting the D^det\widehat{\mathcal{D}}\_{det}D

det experiment.

#### Comment from original authors:

 motivations for designing the D^det\widehat{\mathcal{D}}\_{det}D

det dataset.

###

### Authors

### Affiliations

[Gabriel Goh](https://gabgoh.github.io/)

[OpenAI](https://openai.com/)

 He provides two constructions:

 and “ensembles” that could be candidates for true useful non-robust features.

#### Comment from original authors:

 features for real datasets (and thus a neat corroboration of their existence).

 interesting direction of developing a more fine-grained definition of features.

###

### Authors

###

[Reiichiro Nakano](https://reiinakano.com/)

 Reiichiro shows that adversarial robustness makes neural style transfer

 work by default on a non-VGG architecture.

 to humans.

#### Comment from original authors:

 trained models will have in neural network art!

 interesting links between robustness and style transfer.

###

### Authors

### Affiliations

[Preetum Nakkiran](https://preetum.nakkiran.org/)

[OpenAI](https://openai.com/) &

 [Harvard University](https://www.harvard.edu/)

 has no “non-robust features”.

#### Comment from original authors:

 example of adversarial examples that arise from “bugs”.

###

### Authors

### Affiliations

[Eric Wallace](https://www.ericswallace.com/)

[Allen Institute for AI](https://allenai.org/)

 Eric shows that training on a model’s training

errors,

 or on how it predicts examples form an unrelated

dataset,

 can both transfer to the true test set.

 These experiments are analogous to the original

paper’s non-robust transfer results — all three results are examples of a

 kind of “learning from incorrectly labeled data.”

#### Comment from original authors:

 settings.

Original Author Discussion and Responses

----------------------------------------

###

### Authors

### Affiliations

[Logan Engstrom](http://loganengstrom.com/),

 [Andrew Ilyas](http://andrewilyas.com/),

 [Aleksander Madry](https://people.csail.mit.edu/madry/),

 [Shibani Santurkar](http://people.csail.mit.edu/shibani/),

 Brandon Tran,

 [Dimitris Tsipras](http://people.csail.mit.edu/tsipras/)

MIT

 conversation.

 This article also contains their responses to each comment.

Adversarial examples are not bugs, they are features

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'.html