Activation Atlas

Shan Carter
Zan Armstrong
Ludwig Schubert
Ian Johnson
Chris Olah

Introduction

------------

Neural networks can learn to classify images more accurately than

any system humans directly design. This raises a natural question: What

have these networks learned that allows them to classify images so well?

Feature visualization is a thread of research that tries to

answer this question by letting us “see through the eyes” of the network

 and trying to determine what they respond to. Because neurons don’t

 But there was still a problem — what combinations of neurons should we

These approaches are exciting because they can make the hidden

layers of networks comprehensible. These layers are the heart of how

neural networks outperform more traditional approaches to machine

learning and historically, we’ve had little understanding of what

Unfortunately, visualizing activations has a major

weakness — it is limited to seeing only how the network sees a single

input. Because of this, it doesn’t give us a big picture view of the

network. When what we want is a map of an entire forest, inspecting one

tree at a time will not suffice.

 gives a global view of a dataset by taking each image and organizing

 them by their activation values from a neural network.

 Showing which images the model sees as similar does help us infer

some ideas about what features the network is responding to,

 but feature visualization makes those connections much more

explicit.

 Nguyen, et al  use t-SNE to make more diverse neuron visualizations,

In this article we introduce *activation atlases* to this

quiver of techniques. (An example is shown at the top of this article.)

Broadly speaking, we use a technique similar to the one in CNN codes,

but instead of showing input data, we show feature visualizations of

averaged activations. By combining these two techniques, we can get the

advantages of each in one view — a global map seen through the eyes of

the network.

 would give us the global view of a network that we are seeking. In

practice, however, neurons are rarely used by the network in isolation,

and it may be difficult to understand them that way. As an analogy,

while the 26 letters in the alphabet provide a basis for English, seeing

 how letters are commonly combined to make words gives far more insight

into the concepts that can be expressed than the letters alone.

Similarly, activation atlases give us a bigger picture view by showing

common combinations of neurons.

#### Individual Neurons

![](Activation%20Atlas_files/overview-neuron.jpg)

![diagram](Activation%20Atlas_files/manifold-1.jpg)

 make hidden layers somewhat meaningful, but misses interactions between

 neurons — it only shows us one-dimensional, orthogonal probes of the

high-dimensional activation space.

#### Pairwise Interactions

![](Activation%20Atlas_files/overview-pairwise.jpg)

![diagram](Activation%20Atlas_files/manifold-2.jpg)

 reveal interaction effects, but they only show two-dimensional slices

of a space that has hundreds of dimensions, and many of the combinations

 are not realistic.

#### Spatial Activations

![](Activation%20Atlas_files/activation-grid.png)

![diagram](Activation%20Atlas_files/manifold-3.jpg)

 show us important combinations of many neurons by sampling the

sub-manifold of likely activations, but they are limited to those that

occur in the given example image.

#### Activation Atlas

![](Activation%20Atlas_files/overview-atlas.jpg)

![diagram](Activation%20Atlas_files/manifold-4.jpg)

 These atlases not only reveal visual abstractions within a model,

but later in the article we will show that they can reveal high-level

misunderstandings in a model that can be exploited. For example, by

looking at an activation atlas we will be able to see why a picture of a

 baseball can switch the classification of an image from “grey whale” to

 “great white shark”.

![great white shark / grey whale](Activation%20Atlas_files/whale-baseball.png)

|  |  |  |

| --- | --- | --- |

| 1. | grey whale | 91.0% |

| 2. | killer whale | 7.5% |

| 3. | great white shark | 0.7% |

| 4. | gar | 0.4% |

|  |  |  |

| --- | --- | --- |

| 1. | great white shark | 66.7% |

| 2. | baseball | 7.4% |

| 3. | grey whale | 4.1% |

| 4. | sombrero | 3.2% |

Of course, activation atlases do have limitations. In particular,

they’re dependent on the distribution of the data we choose to sample

activations from (in our examples, we use one million images chosen at

random from the ImageNet dataset

 training data). As a result, they will only show the activations that

exist within the distribution of the sample data. However, while it’s

important to be aware of these limitations — we’ll discuss them in much

more depth later! — Activation Atlases still give us a new kind of

overview of what neural networks can represent.

Looking at a Single Image

-------------------------

Before we dive into Activation Atlases, let’s briefly review how we

 use feature visualization to make activation vectors meaningful (“see

 (also known as “GoogLeNet”).

 InceptionV1 consists of a number of layers, which we refer to as

“mixed3a”, “mixed3b”, “mixed4a”, etc., and sometimes shortened to just

“3a”. Each layer successively builds on the previous layers.

![model](Activation%20Atlas_files/model.svg)

To visualize how InceptionV1 sees an image, the first step is to

feed the image into the network and run it through to the layer of

interest. Then we collect the activations — the numerical values of how

much each neuron fired. If a neuron is excited by what it is shown, its

activation value will be positive.

Unfortunately these vectors of activation values are just

vectors of unitless numbers and not particularly interpretable by

 comes in.

 Roughly speaking, we can think of feature visualization as creating an

 idealized image of what the network thinks would produce a particular

activation vector. Whereas we normally use a network to transform an

image into an activation vector, in feature visualization we go in the

opposite direction. Starting with an activation vector at a particular

layer, we create an image through an iterative optimization process.

 there is not just one activation vector per layer per image.

 This means that the same neurons are run on each patch of the previous

 layer.

 Thus, when we pass an entire image through the network, each neuron

will be evaluated hundreds of times, once for each overlapping patch of

the image. We can consider the vectors of how much each neuron fired for

 each patch separately.

The result is a grid of feature visualizations, one for each

 patch. This shows us how the network sees different parts of the input

image.

![input image](Activation%20Atlas_files/dogcat.jpg)

Input image from ImageNet.

![activation grid](Activation%20Atlas_files/dogcat-grid.jpg)

Activation grid from InceptionV1, layer mixed4d.

Aggregating Multiple Images

---------------------------

Activation grids show how the network sees a single image, but what

 if we want to see more? What if we want to understand how it reacts to

millions of images?

Of course, we could look at individual activation grids for

those images one by one. But looking at millions of examples doesn’t

scale, and human brains aren’t good at comparing lots of examples

without structure. In the same way that we need a tool like a histogram

in order to understand millions of numbers, we need a way to aggregate

and organize activations if we want to see meaningful patterns in

millions of them.

 This gives us one million activation vectors. Each of the vectors is

high-dimensional, perhaps 512 dimensions! With such a complex set of

data, we need to organize and aggregate it if we want a big picture

view.

 can project high-dimensional data like our collection of activation

vectors into useful 2D layouts, preserving some of the local structure

of the original space. This takes care of organizing our activation

vectors, but we also need to aggregate into a more manageable number of

elements — one million dots would be hard to interpret. We’ll do this by

 drawing a grid over the 2D layout we created with dimensionality

reduction. For each cell in our grid, we average all the activations

that lie within the boundaries of that cell, and use feature

visualization to create an iconic representation.

[0.700, 0.498, …]

 [0.818, 0.437, …]

 [0.421, 0.027, …]

The activations are fed through UMAP to

reduce them to two dimensions. They are then plotted, with similar

activations placed near each other.

We then draw a grid and average the

activations that fall within a cell and run feature inversion on the

averaged activation. We also optionally size the grid cells according to

 the density of the number of activations that are averaged within.

 However, we use a slightly non-standard objective.

 We find it helpful to use an objective that emphasizes angle more

heavily by multiplying the dot product by cosine similarity, leading to

objectives of the following form:

 We don’t yet fully understand this phenomenon.

 For each activation vector, we also compute an *attribution*

vector.

 The attribution vector has an entry for each class, and approximates

 the amount that the activation vector influenced the logit for each

class.

 Attribution vectors generally depend on the surrounding context.

 This is similar to Grad-CAM ,

 but without the spatial averaging of the gradient.

 Instead, we reduce noise in the gradient by using a continuous

relaxation of the gradient for max pooling in computing the gradient (as

 in ).

 This average attribution can be thought of as showing what classes

that cell tends to support, marginalizing over contexts.

 At early layers, the average attribution is very small and the top

classes are fairly arbitrary because low-level visual features like

textures tend to not be very discriminative without context.

Grid size:

 20x20

 40x40

 80x80

 160x160

 attribution labels

This atlas can be a bit overwhelming at first

glance — there’s a lot going on! This diversity is a reflection of the

variety of abstractions and concepts the model has developed. Let’s take

 a tour to examine this atlas in more depth.

If we look at the top-left of the atlas, we see images which

look like animal heads. There is some differentiation between different

types of animals, but it seems to be more a collection of elements of

generic mammals — eyes, fur, noses — rather than a collection of

different classes of animals. We’ve also added labels that show which

class each averaged activation most contributes to. Please note, in some

 areas of a layer this early in the network these attribution labels can

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

Below the feet we start to lose any identifiable parts of

animals, and see isolated grounds and floors. We see attribution toward

environments like “sandbar” and also toward things that are found on the

 ground, like “doormat” or “ant”.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

These sandy, rocky backgrounds slowly blend into beaches and

 bodies of water. Here we see lakes and oceans, both above and below

water. Though the network does have certain classes like “seashore”, we

see attribution toward many sea animals, without any visual references

to the animals themselves. While not unexpected, it is reassuring to see

 that the activations that are used to identify the sea for the class

“seashore” are the same ones used when classifying “starfish” or “sea

lion”. There is also no real distinction at this point between lakes and

 ocean — “lakeside” and “hippopotamus” attributions are intermingled

with “starfish” and “stingray”.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

Now let’s jump to the other side of the atlas, where we can

see many variations of text detectors. These will be useful when

identifying classes such as “menu”, “web site” or “book jacket”.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

Moving upward, we see many variations of people. There are

very few classes that specifically identify people in ImageNet, but

people are present in lots of the images. We see attribution toward

things people use (“hammer”, “flute”), clothes that people wear (“bow

tie”, “maillot”) and activities that people participate in

(“basketball”). There is a uniformity to the skin color in these

visualizations which we suspect is a reflection of the distribution of

the data used for training. (You can browse the ImageNet training data

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

And finally, moving back to the left, we can see round food

and fruit organized mostly by colors — we see attribution toward

“lemon”, “orange” and “fig”.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

We can also trace curved paths through this manifold that

we’ve created. Not only are regions important, but certain movements

through the space seem to correspond to human interpretable qualities.

With the fruit, we can trace a path that seems to correlate with the

size and number of fruits in the frame.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

Similarly, with people, we can trace a path that seems to

correspond to how many people are in the frame, whether it’s a single

person or a crowd.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

In the plants region, we can trace a path that seems to

correspond to how blurry the plant is. This could possibly be used to

determine relative size of objects because of the typical focal lengths

of cameras. Close up photos of small insects have more opportunity for

blurry background foliage than photos of larger animals, like monkeys.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

It is important to note that these paths are constructed

after the fact in the low-dimensional projection. They are smooth paths

in this reduced projection but we don’t necessarily know how the paths

operate in the original higher-dimensional activation space.

Looking at Multiple Layers

--------------------------

In the previous section we focused on one layer of the network,

mixed4c, which is in the middle of our network. Convolutional networks

are generally deep, consisting of many layers that progressively build

up more powerful abstractions. In order to get a holistic view, we must

look at how the model’s abstractions develop over several layers.

![model](Activation%20Atlas_files/model2.svg)

To start, let’s compare three layers from different areas of

 the network to try to get a sense for the different personalities of

 .vertical-layer-comparison {

 display: grid;

 grid-column: page;

 grid-template-columns: 1fr 1fr 1fr;

 grid-gap: 16px 20px;

 }

 .vertical-layer-comparison h4 {

 margin-bottom: 0;

 padding-bottom: 4px;

 width: 100%;

 border-bottom: solid #CCC 1px;

 }

 .vertical-layer-comparison.progress {

 display: block;

 max-width: 704px;

 padding: 0 20px;

 margin: 0 auto;

 }

 .vertical-layer-comparison.progress > div {

 display: grid;

 grid-template-columns: 2.5fr 1fr;

 grid-gap: 20px;

 margin-bottom: 16px;

 }

 .vertical-layer-comparison.progress h4 {

 margin-top: 0;

 margin-bottom: 16px;

 }

 .vertical-layer-comparison.progress .figcaption {

 margin-top: 8px;

 }

#### Mixed3b

#### Mixed4c

#### Mixed5b

![](Activation%20Atlas_files/layers-plant-0.jpg)

![thumbnail for mixed3b](Activation%20Atlas_files/thumbnail-mixed3b.jpg)

You'll immediately notice that

the early layer is very nonspecific in comparison to the others. The

icons that emerge are of patterns and splotches of color. It is

suggestive of the final class, but not particularly evocative.

![](Activation%20Atlas_files/layers-plant-1.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

By the middle layer, icons

definitely resemble leaves, but they could be any type of plant.

Attributions are focused on plants, but are a little all over the board.

![](Activation%20Atlas_files/layers-plant-2.jpg)

![thumbnail for mixed5b](Activation%20Atlas_files/thumbnail-mixed5b.jpg)

Here we see foliage with

textures that are specific to cabbage, and curved into rounded balls.

There are full heads of cabbage rather than individual leaves.

As you move through the network, the later layers seem to

get much more specific and complex. This is to be expected, as each

layer builds its activations on top of the preceding layer’s

activations. The later layers also tend to have larger receptive fields

than the ones that precede them (meaning they are shown larger subsets

of the image) so the concepts seem to encompass more of the whole of

objects.

There is another phenomenon worth noting: not only are concepts

 being refined, but new concepts are appearing out of combinations of

old ones. Below, you can see how sand and water are distinct concepts in

 a middle layer, mixed4c, both with strong attributions to the

classification of “sandbar”. Contrast this with a later layer, mixed5b,

where the two ideas seem to be fused into one activation.

#### Mixed4c

#### Mixed5b

![](Activation%20Atlas_files/layers-water-0.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

![](Activation%20Atlas_files/layers-water-1.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

…and a separate area of **watery textures**.

![](Activation%20Atlas_files/layers-water-2.jpg)

![thumbnail for mixed5b](Activation%20Atlas_files/thumbnail-mixed5b.jpg)

Finally, if we zoom out a little, we can see how the broader

 shape of the activation space changes from layer to layer. By looking

at similar regions in several consecutive layers, we can see concepts

getting refined and differentiated — In mixed4a we see very vague,

generic blob, which gets refined into much more specific “peninsulas” by

 mixed4e.

![mixed4a](Activation%20Atlas_files/progress-mixed4a.jpg)

#### Mixed4a

 In mixed4a there is a vague “mammalian” area.

![mixed4b](Activation%20Atlas_files/progress-mixed4b.jpg)

#### Mixed4b

![mixed4c](Activation%20Atlas_files/progress-mixed4c.jpg)

#### Mixed4c

![mixed4d](Activation%20Atlas_files/progress-mixed4d.jpg)

#### Mixed4d

 The specialization continues in mixed4d.

![mixed4e](Activation%20Atlas_files/progress-mixed4e.jpg)

#### Mixed4e

 And further still in mixed4e.

Below you can browse many more of the layers of InceptionV1. You can compare the

 [curved edge detectors of mixed4a](#) with the

 [bowls and cups of mixed5b](#). Mixed4b has some

 [fabrics](#). In later layers, you’ll see

 [specific types of clothing](#).

Class Filter

------------

show all

fireboat

streetcar

speedboat

scuba diver

snorkel

dalmatian

Labrador retriever

Siberian husky

English setter

Doberman

Tibetan terrier

sea lion

great white shark

goldfish

lionfish

oil filter

digital clock

digital watch

comic book

spider web

barn spider

head cabbage

artichoke

yellow lady's slipper

croquet ball

volleyball

parachute

sleeping bag

lakeside

coral reef

seashore

vine snake

water snake

Indian cobra

green mamba

goldfinch

house finch

vulture

lorikeet

peacock

Layer

-----

mixed3a

mixed3b

mixed4a

mixed4b

mixed4c

mixed4d

mixed4e

mixed5a

mixed5b

![thumbnail for mixed4e](Activation%20Atlas_files/thumbnail-mixed4e.jpg)

 attribution labels

 show tooltip

 scroll to zoom

### Grid size

 20x20

 40x40

 80x80

 160x160

 320x320

 auto

auto threshold: 0.5

### Icon density: 1

### Class filter

 positive influence

 negative influence

Intensity: 1

### Location

x: 0.500

y: 0.500

scale: 1.000

fewer options

Focusing on a Single Classification

-----------------------------------

Looking at an atlas of all activations can be a little

overwhelming, especially when you’re trying to understand how the

network goes about ranking one particular class. For instance, let’s

investigate how the network classifies a “fireboat”.

![fireboat](Activation%20Atlas_files/fireboat-01.jpg)

An image labeled “fireboat” from ImageNet.

We’ll start by looking at an atlas for the last layer,

mixed5b. Instead of showing all the activations, however, we’ll

calculate the amount that each activation contributes toward a

classification of “fireboat” and then map that value to the opacity of

the activation icon.In the case of

mixed5b, determining this contribution is fairly straightforward because

 the relationship between activations at mixed5b and the logit values is

 linear. When there are multiple layers between our present one and the

output — and as a result, the relationship is non-linear — it’s a little

 less clear what to do. In this article, we take the simple approach of

forming a linear approximation of these future layers and use it to

approximate the effect of our activations. The areas that

contribute a lot toward a classification of “fireboat” will be clearly

visible, whereas the areas that contribute very little (or even

contribute negatively) will be completely transparent.

![](Activation%20Atlas_files/focus-1-1.jpg)

![](Activation%20Atlas_files/focus-1-2.jpg)

When we map opacity to the

amount that each activation contributes to "fireboat" in the layer

mixed5b, we see a main cluster of icons showing red boats and splashing,

 spraying water. While there are some stray areas elsewhere, it seems

that this is region of the atlas that is dedicated specifically to

classifying red boats with splashing water nearby.

The layer we just looked at, mixed5b, is located just before

 the final classification layer so it seems reasonable that it would be

closely aligned with the final classes. Let’s look at a layer a little

earlier in the network, say mixed4d, and see how it differs.

![](Activation%20Atlas_files/focus-2-1.jpg)

3

2

1

![](Activation%20Atlas_files/focus-2-4.jpg)

1

![](Activation%20Atlas_files/focus-2-3.jpg)

2

![](Activation%20Atlas_files/focus-2-2.jpg)

3

In mixed4d we see we see the

attribution toward "fireboat" is high in several clusters located in

different positions around the atlas. One is very focused on windows,

another on geysers and splashing water, and yet another on crane-like

objects.

Here we see a much different pattern. If we look at some

more input examples, this seems entirely reasonable. It’s almost as if

we can see a collection of the component concepts the network will use

in later layers to classify “fireboat”. Windows + crane + water =

“fireboat”.

![fireboat](Activation%20Atlas_files/fireboat-10.jpg)

![fireboat](Activation%20Atlas_files/fireboat-02.jpg)

![fireboat](Activation%20Atlas_files/fireboat-03.jpg)

One of the clusters, the one with windows, has strong

attribution to “fireboat”, but taken on its own, it has an even stronger

 attribution toward “streetcar”. So, let’s go back to the atlas at

mixed4d, but isolate “streetcar” and compare it to the patterns seen for

 “fireboat”. Let’s look more closely at the four highlighted areas: the

three areas we highlighted for fireboat plus one additional area that is

 highly activated for streetcars.

#### Activations for Fireboat

![](Activation%20Atlas_files/focus-2-1.jpg)

1234

#### Activations for Streetcar

![](Activation%20Atlas_files/focus-3-1.jpg)

1234

If we zoom in, we can get a better look at what

distinguishes the two classifications at this layer. (We’ve

cherry-picked these examples for brevity, but you can explore all the

layers and activations in detail in a explorable playground below.)

#### Fireboat

![](Activation%20Atlas_files/focus-2-2.jpg)

1![](Activation%20Atlas_files/focus-2-3.jpg)

2![](Activation%20Atlas_files/focus-2-4.jpg)

3![](Activation%20Atlas_files/focus-2-5.jpg)

4

#### Streetcar

![](Activation%20Atlas_files/focus-3-2.jpg)

1![](Activation%20Atlas_files/focus-3-3.jpg)

2![](Activation%20Atlas_files/focus-3-4.jpg)

3![](Activation%20Atlas_files/focus-3-5.jpg)

4

"Fireboat" activations have

much stronger attributions from water than "streetcar", where there is

virtually no positive evidence.

If we look at a couple of input examples, we can see how

buildings and water backgrounds are an easy way to differentiate between

 a “fireboat” and a “streetcar”.

![fireboat](Activation%20Atlas_files/fireboat-01.jpg)

![streetcar](Activation%20Atlas_files/streetcar-01.jpg)

Images from ImageNet

By isolating the activations that contribute strongly to one

 class and comparing it to other class activations, we can see which

activations are conserved among classes and which are recombined to form

 more complex activations in later layers. Below you can explore the

activation patterns of many classes in ImageNet through several layers

of InceptionV1. You can even explore negative attributions, which we

ignored in this discussion.

Class Filter

------------

show all

fireboat

streetcar

speedboat

scuba diver

snorkel

dalmatian

Labrador retriever

Siberian husky

English setter

Doberman

Tibetan terrier

sea lion

great white shark

goldfish

lionfish

oil filter

digital clock

digital watch

comic book

spider web

barn spider

head cabbage

artichoke

yellow lady's slipper

croquet ball

volleyball

parachute

sleeping bag

lakeside

coral reef

seashore

vine snake

water snake

Indian cobra

green mamba

goldfinch

house finch

vulture

lorikeet

peacock

Layer

-----

mixed3a

mixed3b

mixed4a

mixed4b

mixed4c

mixed4d

mixed4e

mixed5a

mixed5b

![thumbnail for mixed4d](Activation%20Atlas_files/thumbnail-mixed4d.jpg)

 attribution labels

 show tooltip

 scroll to zoom

### Grid size

 20x20

 40x40

 80x80

 160x160

 320x320

 auto

auto threshold: 0.5

### Icon density: 1

### Class filter

 positive influence

 negative influence

Intensity: 1

### Location

x: 0.500

y: 0.500

scale: 1.000

fewer options

Further Isolating Classes

-------------------------

Highlighting the class-specific activations in situ of a full atlas

 is helpful for seeing how that class relates to the full space of what a

 network “can see.” However, if we want to really isolate the

activations that contribute to a specific class we can remove all the

other activations rather than just dimming them, creating what we’ll

 class activations we generally have better results from using t-SNE for

 the dimensionality reduction step rather than UMAP. We suspect it is

#### Class Activation Atlas for “snorkel” from mixed5b

 scuba diver

 snorkel

 red fox

 ibex

 giant panda

 lesser panda

 zebra

 tiger

 fireboat

 dalmatian

 English setter

 great white shark

 goldfish

 lionfish

 comic book

 plunger

 knot

 tile roof

 pill bottle

 measuring cup

 crossword puzzle

 cauliflower

 head cabbage

 artichoke

 green mamba

 goldfinch

 house finch

 vulture

 chainlink fence

 wok

 grey whale

 waffle iron

 conch

 snowmobile

 sax

 French horn

 stage

 spider web

 barn spider

 grasshopper

 monarch

A class activation atlas gives us a much clearer view of

which detectors the network is using to rank a specific class. In the

“snorkel” example we can clearly see ocean, underwater, and colorful

masks.

In the previous example, we are only showing those activations

whose strongest attribution is toward the class in question. This will

show us activations that contribute mostly to our class in question,

even if their overall strength is low (like in background detectors). In

 some cases, though, there are strong correlations that we’d like to see

 (like fish with snorkelers). These activations on their own might

contribute more strongly to a different class than the one we’re

interested in, but their existence can also contribute strongly to our

class of interest. For these we need to choose a different filtering

method.

#### “snorkel” filtered by top rank

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

We pluck only those

activations whose top attribution is toward the class in question. The

results are often much more focused and isolated, exclusive to the

class. Some are low magnitude, like backgrounds, and we miss

correlations or concepts that are shared among many classes.

#### “snorkel” filtered by overall magnitude

drake

red-breasted merganser

swimming trunks

maillot

balance beam

dugong

killer whale

snorkel

snorkel

snorkel

canoe

rock beauty

puffer

dugong

snorkel

snorkel

bathing cap

lifeboat

canoe

rock beauty

vine snake

agama

scuba diver

snorkel

snorkel

green mamba

snorkel

snorkel

bathing cap

neck brace

snorkel

oxygen mask

gasmask

football helmet

neck brace

water bottle

oxygen mask

oxygen mask

sunglass

ballpoint

swab

sunglass

sunglasses

Here we sort all the

activations by the magnitude toward the class in question (independent

of other classes) and take the top 2,000 activations. We see more

correlated activations that could, on their own, contribute to another

classification (We label each activation with the class that it

contributes toward most). Notice that certain fish that are commonly

seen while snorkeling now show up with that class, and a “coffee mug”

now shows up with “crossword puzzle”. Some of them appear spurious,

however.

 scuba diver

 snorkel

 red fox

 ibex

 giant panda

 lesser panda

 zebra

 tiger

 fireboat

 dalmatian

 English setter

 great white shark

 goldfish

 lionfish

 comic book

 plunger

 knot

 tile roof

 pill bottle

 measuring cup

 crossword puzzle

 cauliflower

 head cabbage

 artichoke

 green mamba

 goldfinch

 house finch

 vulture

 chainlink fence

 wok

 grey whale

 waffle iron

 conch

 snowmobile

 sax

 French horn

 stage

 spider web

 barn spider

 grasshopper

 monarch

Using the magnitude filtering method, let’s try to compare two

related classes and see if we can more easily see what distinguishes

them. (We could have instead used rank, or a combination of the two, but

 magnitude will suffice to show us a good variety of concepts).

#### “snorkel”

#### “scuba diver”

It can be a little hard to immediately understand all the

differences between classes. To help make the comparison easier, we can

combine the two views into one. We’ll plot the difference between the

attributions of the “snorkel” and “scuba diver” horizontally, and use

t-SNE  to cluster similar activations vertically.

In this comparison we can see some bird-like creatures and

clear tubes on the left, implying a correlation with “snorkel”, and some

 shark-like creatures and something round, shiny, and metallic on the

right, implying correlation with “scuba diver” (This activation has a

strong attribution toward the class “steam locomotive”). Let’s take an

image from the ImageNet dataset labeled as “snorkel” and add something

that resembles this icon to see how it affects the classification

scores.

![snorkel / scuba diver](Activation%20Atlas_files/snorkel-train.png)

|  |  |  |

| --- | --- | --- |

| 1. | snorkel | 55.6% |

| 2. | coral reef | 18.6% |

| 3. | scuba diver | 13.5% |

| 4. | loggerhead | 5.5% |

| 5. | lionfish | 1.7% |

| 6. | sea snake | 1.4% |

![snorkel / scuba diver](Activation%20Atlas_files/snorkel-train-medium.png)

|  |  |  |

| --- | --- | --- |

| 1. | scuba diver | 71.3% |

| 2. | coral reef | 14.7% |

| 3. | snorkel | 4.5% |

| 4. | lionfish | 3.2% |

| 5. | sea snake | 2.4% |

| 6. | loggerhead | 0.9% |

By adding a picture of one

of the concepts seen in the visualization above we can change the

classification. With an added picture of a steam train the confidence of

 “scuba diver” classification rises and “snorkel” drops significantly.

![snorkel / scuba diver](Activation%20Atlas_files/snorkel-train-large.png)

|  |  |  |

| --- | --- | --- |

| 1. | steam locomotive | 89.6% |

| 2. | snowplow | 6.2% |

| 3. | jeep | 0.8% |

| 4. | tractor | 0.5% |

| 5. | scuba diver | 0.4% |

| 6. | passenger car | 0.3% |

The failure mode here seems to be that the model is using

its detectors for the class “steam locomotive” to identify air tanks to

help classify “scuba diver”. We’ll call these “multi-use”

features — detectors that react to very different concepts that are

nonetheless visually similar. Let’s look at the differences between a

“grey whale” and a “great white shark” to see another example of this

issue.

In this example we see another detector that seems to be

playing two roles: detecting red stitching on a baseball and a sharks’s

![great white shark / grey whale](Activation%20Atlas_files/whale-baseball.png)

|  |  |  |

| --- | --- | --- |

| 1. | grey whale | 91.0% |

| 2. | killer whale | 7.5% |

| 3. | great white shark | 0.7% |

| 4. | gar | 0.4% |

| 5. | sea lion | 0.1% |

| 6. | tiger shark | 0.1% |

|  |  |  |

| --- | --- | --- |

| 1. | great white shark | 66.7% |

| 2. | baseball | 7.4% |

| 3. | grey whale | 4.1% |

| 4. | sombrero | 3.2% |

| 5. | sea lion | 3.1% |

| 6. | killer whale | 2.7% |

|  |  |  |

| --- | --- | --- |

| 1. | baseball | 100.0% |

| 2. | rugby ball | 0.0% |

| 3. | golf ball | 0.0% |

| 4. | ballplayer | 0.0% |

| 5. | drum | 0.0% |

| 6. | sombrero | 0.0% |

The results follow the pattern in previous examples pretty

closely.

 Adding a small-sized baseball does change the top classification to

“great white shark”, and as it gets bigger it overpowers the

classification, so the top slot goes to “baseball”.

Let’s look at one more example: “frying pan” and “wok”.

One difference stands out here — the type of related foods

present. On the right we can clearly see something resembling noodles

(which have a strong attribution toward the class “carbonara”). Let’s

take a picture from ImageNet labeled as “frying pan” and add an inset of

 some noodles.

![frying pan / wok](Activation%20Atlas_files/frying-pan-noodles.png)

|  |  |  |

| --- | --- | --- |

| 1. | frying pan | 76.5% |

| 2. | wok | 15.8% |

| 3. | stove | 5.4% |

| 4. | spatula | 1.0% |

| 5. | Dutch oven | 0.5% |

| 6. | mixing bowl | 0.2% |

![frying pan / wok](Activation%20Atlas_files/frying-pan-noodles-medium.png)

|  |  |  |

| --- | --- | --- |

| 1. | wok | 63.2% |

| 2. | frying pan | 35.1% |

| 3. | spatula | 0.6% |

| 4. | hot pot | 0.5% |

| 5. | mixing bowl | 0.1% |

| 6. | stove | 0.1% |

![frying pan / wok](Activation%20Atlas_files/frying-pan-noodles-large.png)

|  |  |  |

| --- | --- | --- |

| 1. | carbonara | 96.4% |

| 2. | plate | 2.3% |

| 3. | wok | 0.6% |

| 4. | meat loaf | 0.1% |

| 5. | dishwasher | 0.1% |

| 6. | rotisserie | 0.1% |

As we make the picture of

noodles larger, its influence overpowers the other classifications, but

“wok” remains ranked above “frying pan”.

Here the patch was not as effective at lowering the initial

classification, which makes sense since the noodle-like icons were

plotted more toward the center of the visualization thus having less of a

 difference in attribution. We suspect that the training set simply

contained more images of woks with noodles than frying pans with

noodles.

### Testing dozens of patches on thousands of images

So far we’ve only shown single examples of these patches. Below we

show the result of ten sample patches (each set includes the one example

 we explored above), run on 1,000 images from the ImageNet training set

for the class in question. While they aren’t effective in all cases,

they do flip the image classification to the target class in about 2 in 5

 images. The success rate reaches about 1 in 2 images if we also allow

to position the patch in the best of the four corners of the image (top

left, top right, bottom left, bottom right) at the most effective size.

To ensure our attack isn’t just blocking out evidence for the original

class, we also compare each attack to a random noise image patch.

 #adversarial-analysis {

 display: none;

 grid-column: text;

 margin-bottom: 0;

 }

 #adversarial-analysis table {

 width: 100%;

 text-align: right;

 }

 #adversarial-analysis table th {

 vertical-align: initial;

 }

 #adversarial-analysis table th .figcaption {

 display: block;

 font-weight: initial;

 }

 #adversarial-analysis table th img {

 width: initial;

 }

 #adversarial-analysis table .detail {

 /\* padding-left: 1em; \*/

 font-size: 85%;

 color: rgba(0, 0, 0, 0.6);

 }

 #adversarial-analysis table .synset {

 font-family: monospace;

 white-space: nowrap;

 }

 #adversarial-analysis table .from,

 #adversarial-analysis table .to {

 white-space: nowrap;

 }

 #adversarial-analysis table .overall {

 font-weight: bold;

 }

 @supports (font-variant-numeric: tabular-nums) {

 #adversarial-analysis table {

 font-variant-numeric: tabular-nums;

 }

 }

#### Dynamic corner position

| icon

Choosing the strongest size and corner position for each example. | Success rate

Mean probability assigned to the target class before and after patch. |

| --- | --- | --- |

| snorkel → scuba diver | 85.0% | 7,031 / 8,270 | +56.8% | 3.7% → 60.5% |

| frying pan → wok | 44.9% | 2,799 / 6,240 | +24.1% | 10.3% → 34.4% |

#### Consistent corner position

| icon

| --- | --- | --- |

| snorkel → scuba diver | 65.4% | 5,409 / 8,270 | +43.0% | 3.7% → 46.7% |

| frying pan → wok | 25.6% | 1,595 / 6,240 | +12.7% | 10.3% → 23.0% |

#### details for each patch tested

| snorkel → scuba divergrey whale → great white sharkfrying pan → wok

| --- | --- | --- |

| patch 0

Patch 01 | 88.1% | 729 / 827 | +60.3% | 3.7% → 64.0% |

| patch 1

Patch 02 | 83.4% | 690 / 827 | +53.0% | 3.7% → 56.7% |

| patch 2

Patch 03 | 92.4% | 764 / 827 | +66.9% | 3.7% → 70.6% |

| patch 3

Patch 04 | 93.0% | 769 / 827 | +66.7% | 3.7% → 70.4% |

| patch 4

Patch 05 | 88.0% | 728 / 827 | +58.8% | 3.7% → 62.5% |

| patch 5

Patch 06 | 85.5% | 707 / 827 | +52.0% | 3.7% → 55.7% |

| patch 6

Patch 07 | 86.2% | 713 / 827 | +57.4% | 3.7% → 61.2% |

| patch 7

Patch 08 | 56.1% | 464 / 827 | +30.7% | 3.7% → 34.5% |

| patch 8

Patch 09 | 92.0% | 761 / 827 | +64.8% | 3.7% → 68.5% |

| patch 9

Patch 10 | 85.4% | 706 / 827 | +57.6% | 3.7% → 61.3% |

| patch undefined

Random noise | 13.9% | 115 / 827 | +9.5% | 3.7% → 13.2% |

 Our “attacks” can be seen as part of a larger trend (eg. ) of researchers

 In many ways, our attacks are most similar to adversarial patches ,

 which also add a small patch to the input image.

 From this perspective, adversarial patches are far more effective,

working much more reliably. Instead, we see our attacks as interesting

because they are synthesized by humans from their understanding of the

model,

 and seem to be attacking the model at a higher level of abstraction.

 We also want to emphasize that not all class comparisons reveal

these type of patches and not all icons in the visualization have the

same (or any) effectiveness and we’ve only tested them on one model. If

we wanted to find these patches more systematically, a different

approach would most likely be more effective. However, the class

activation atlas technique was what revealed the existence of these

patches before we knew to look for them. If you’d like to explore your

own comparisons and search for your own patches, we’ve provided a

Conclusion and Future Work

--------------------------

Activation atlases give us a new way to peer into convolutional

vision networks. They give us a global, hierarchical, and

human-interpretable overview of concepts within the hidden layers. Not

only does this allow us to better see the inner workings of these

complicated systems, but it’s possible that it could enable new

interfaces for working with images.

### Surfacing Inner Properties of Models

The vast majority of neural network research focuses on

quantitative evaluations of network behavior. How accurate is the model?

 What’s the precision-recall curve?

 it behaves the way it does. To truly understand why a network behaves

the way it does, we would need to fully understand the rich inner world

of the network — it’s hidden layers. For example, understanding better

how InceptionV1 builds up a classifier for a fireboat from component

parts in mixed4d can help us build confidence in our models and can

surface places where it isn’t doing what we want.

Engaging with this inner world also invites us to do deep

learning research in a new way. Normally, each neural network experiment

 gives only a few bits of feedback — whether the loss went up or

down — to inform the next round of experiments. We design architectures

by almost blind trial and error, guided by vague intuitions that we

build up over years. In the future, we hope that researchers will get

rich feedback on what each layer in their model is doing in a way that

will make our current approach seem like stumbling in the dark.

Activation atlases, as they presently stand, are inadequate to

really help researchers iterate on models, in part because they aren’t

comparable. If you look at atlases for two slightly different models,

it’s hard to take away anything. In future work, we will explore how

similar visualizations can compare models, showing similarities and

differences beyond error rates.

### New interfaces

Machine learning models are usually deployed as black boxes that

automate a specific task, executing it on their own. But there’s a

growing sense that there might be an alternate way for us to relate to

them: that instead of increasingly automating a task, they could be used

 more directly by a person. One vision of this augmentation that we find

 particularly compelling is the idea that the internal representations

 and music

  .

We think of activation atlases as revealing a machine-learned

alphabet for images — a collection of simple, atomic concepts that are

combined and recombined to form much more complex visual ideas. In the

same way that we use word processors to turn letters into words, and

words into sentences, we can imagine a tool that would allow us to

create images from a machine-learned language system for images. Similar

 to GAN painting , imagine

using something like an activation atlas as a palette — one could dip a

brush into a “tree” activation and paint with it. A palette of concepts

rather than colors.

 has shown that this is entirely possible. In this particular instance,

we imagine constructing a grid of activations by selecting them from an

atlas (or some derivation), then optimizing an output image that would

correspond to the user’s constructed activation matrix.

 have shown us that we can use these vision networks to create nuanced

visual expression outside the explicit distribution of visual data it

was trained on. We speculate that activation atlases could be helpful in

 manipulating artistic styles without having to find an existing

reference image, or they could help in guiding and modifying automated

style transfer techniques.

We could also use these atlases to query large image datasets.

In the same way that we probe large corpuses of text with words, we

could, too, use activation atlases to find types of images in large

image datasets. Using words to search for something like a “tree” is

quite powerful, but as you get more specific, human language is often

ill-suited to describe specific visual characteristics. In contrast, the

 hidden layers of neural networks are a language optimized for the sole

purpose of representing visual concepts. Instead of using the proverbial

 thousand words to uniquely specify the image one is seeking, we can

imagine someone using the language of the activation atlas.

And lastly, we can also liken activation atlases to histograms.

 In the same way that traditional histograms give us good summaries of

large datasets, activation atlases can be used to summarize large

numbers of images.

In the examples in this article we used the same dataset for

training the model as we did for collecting the activations. But, if we

use a different dataset to collect the activations, we could use the

atlas as a way of inspecting an unknown dataset. An activation atlas

could show us a histogram of *learned concepts* that exist within

 the images. Such a tool could show us the semantics of the data and not

 just visual similarities, like showing histograms of common pixel

values.

While we are excited about the potential of activation atlases,

 we are even more excited at the possibility of developing similar

techniques for other types of models. Imagine having an array of machine

 learned, but human interpretable, languages for images, audio and text.

Visualizing higher-layer features of a deep network

Feature Visualization

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Understanding deep image representations by inverting them

The Building Blocks of Interpretability

t-SNE visualization of CNN codes

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Imagenet: A large-scale hierarchical image database

Going deeper with convolutions

Imagenet large scale visual recognition challenge

Visualizing data using t-SNE

UMAP: Uniform Manifold Approximation and Projection

Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization

Adversarial Patch

Adversarial examples in the physical world

A rotation and a translation suffice: Fooling cnns with simple transformations

Unrestricted adversarial examples

Intriguing properties of neural networks

Using Artificial Intelligence to Augment Human Intelligence

Visualizing Representations: Deep Learning and Human Beings

Machine Learning for Visualization

Image-to-image translation with conditional adversarial networks

TopoSketch: Drawing in Latent Space

Generative visual manipulation on the natural image manifold

ML as Collaborator: Composing Melodic Palettes with Latent Loops

A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

A neural algorithm of artistic style

Exploring Histograms


How to Use t-SNE Effectively

Martin Wattenberg
Fernanda Viégas
Ian Johnson

Abstract: A popular method for exploring high-dimensional data is something called t-SNE, introduced by van der Maaten and Hinton in 2008 [1].  The technique has become widespread in the field of machine learning,  since it has an almost magical ability to create compelling  two-dimensonal “maps” from data with hundreds or even thousands of  dimensions. Although impressive, these images can be tempting to misread. The  purpose of this note is to prevent some common misreadings.

How to Use t-SNE Effectively

============================

Although extremely useful for visualizing

high-dimensional data, t-SNE plots can sometimes be mysterious or

misleading. By exploring how it behaves in simple cases, we can learn to

 use it more effectively.

 #playground {

 overflow: hidden;

 font-family: 'Open Sans', sans-serif;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 /\*border-bottom: 1px solid rgba(0, 0, 0, 0.1);\*/

 margin-top: 36px;

 padding: 36px 0 0 0;

 /\*background: #fcfcfc;\*/

 z-index: 1000;

 }

 #playground \* {

 box-sizing: border-box;

 }

 #playground.modal {

 position: fixed;

 left: 10px;

 right: 10px;

 top: 50px;

 }

 /\* Playground Canvas \*/

 #playground-canvas {

 float: left;

 width: 55%;

 }

 #playground-canvas canvas {

 width: 100%;

 }

 /\* Data Menu \*/

 #data-menu {

 /\*float: left;\*/

 /\*width: 25%\*/

 width: 40%;

 float: left;

 margin-bottom: 24px;

 overflow: hidden;

 margin-left: 5%;

 }

 #data-menu .demo-data {

 cursor: pointer;

 position: relative;

 font-size: 10px;

 line-height: 1.2em;

 box-sizing: border-box;

 float: left;

 margin: 2px;

 padding: 4px;

 width: calc(33% - 4px);

 background: white;

 border: 1px solid rgba(0, 0, 0, 0.1);

 border-radius: 4px;

 box-shadow: 0 0 3px rgba(0, 0, 0, 0.08);

 }

 @media(min-width: 480px) {

 #data-menu .demo-data {

 width: calc(25% - 8px);

 padding: 8px;

 margin: 4px;

 }

 }

 @media(min-width: 768px) {

 #data-menu .demo-data {

 width: calc(16.5% - 8px);

 padding: 8px;

 margin: 4px;

 }

 }

 #data-menu .demo-data:hover {

 border: 1px solid rgba(0, 0, 0, 0.2);

 }

 #data-menu .demo-data.selected::after {

 content: "";

 border: 2px solid rgba(70, 130, 180, 0.8);

 width: 100%;

 height: 100%;

 position: absolute;

 top: 0;

 left: 0;

 box-sizing: border-box;

 border-radius: 4px;

 }

 #data-menu .demo-data span {

 display: none;

 }

 #data-menu .demo-data:hover canvas {

 opacity: 1;

 }

 #data-menu .demo-data canvas {

 width: 100%;

 opacity: 0.3;

 }

 #data-menu .demo-data.selected canvas {

 opacity: 1;

 }

 /\* Data Details \*/

 #data-details {

 position: relative;

 }

 @media(min-width: 768px) {

 #data-details {

 width: 40%;

 float: right;

 }

 }

 #data-details #data-controls {

 width: 40%;

 float: right;

 position: relative;

 overflow: hidden;

 font-size: 13px;

 }

 @media(min-width: 768px) {

 #data-details #data-controls {

 width: 50%;

 margin-right: 10%;

 float: left;

 }

 }

 #data-details #play-controls {

 margin-bottom: 18px;

 overflow: hidden;

 position: relative;

 }

 #data-details #play-controls button {

 cursor: pointer;

 outline: none;

 border-radius: 50%;

 background: steelblue;

 color: white;

 width: 25%;

 margin-right: 5%;

 padding-top: 25%;

 padding-bottom: 0;

 border: none;

 float: left;

 position: relative;

 }

 #play-controls i {

 display: block;

 position: absolute;

 top: 50%;

 left: 0;

 width: 100%;

 height: 36px;

 font-size: 24px;

 line-height: 0;

 }

 @media(min-width: 768px) {

 #play-controls i {

 font-size: 30px;

 }

 }

 #play-controls #play-pause i {

 display: none;

 position: absolute;

 }

 #play-controls #play-pause.paused i:nth-child(1) {

 display: block;

 }

 #play-controls #play-pause.playing i:nth-child(2) {

 display: block;

 }

 #steps-display {

 float: left;

 text-align: center;

 width: 25%;

 line-height: 1.5em;

 font-size: 13px;

 }

 @media(min-width: 1024px) {

 #steps-display {

 font-size: 16px;

 line-height: 1.6em;

 }

 }

 #data-details #data-description {

 width: 50%;

 margin-right: 10%;

 float: right;

 font-size: 14px;

 line-height: 1.6em;

 }

 @media(min-width: 768px) {

 #data-details #data-description {

 width: 40%;

 float: left;

 margin-right: 0;

 }

 }

 /\* Options \*/

 #data-details #options {

 float: left;

 padding-left: 36px;

 font-size: 13px;

 line-height: 1.5em;

 width: 45%;

 }

 #data-details input {

 display: block;

 width: 100%;

 margin: 8px 0 16px 0;

 }

 #options #data-options {

 width: 45%;

 margin-left: 4px;

 }

 #options #tsne-options {

 width: 45%;

 margin-left: 4px;

 }

 #data-details #share {

 margin-top: 8px;

 display: block;

 color: rgba(0, 0, 0, 0.4);

 text-decoration: none;

 font-size: 12px;

 }

 #data-details #share:hover {

 text-decoration: underline;

 }

 #data-details #share i {

 line-height: 0px;

 position: relative;

 top: 7px;

 }

*play\_arrow**pause*

*refresh*

 Step

102

Points Per Side 20

Perplexity 10Epsilon 5

[*link* Share this view](#perplexity=10&epsilon=5&demo=0&demoParams=20)

 dt-byline {

 font-size: 12px;

 line-height: 18px;

 display: block;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 color: rgba(0, 0, 0, 0.5);

 padding-top: 12px;

 padding-bottom: 12px;

 }

 dt-article.centered dt-byline {

 text-align: center;

 }

 dt-byline a,

 dt-article dt-byline a {

 text-decoration: none;

 border-bottom: none;

 }

 dt-article dt-byline a:hover {

 text-decoration: underline;

 border-bottom: none;

 }

 dt-byline .authors {

 text-align: left;

 }

 dt-byline .name {

 display: inline;

 text-transform: uppercase;

 }

 dt-byline .affiliation {

 display: inline;

 }

 dt-byline .date {

 display: block;

 text-align: left;

 }

 dt-byline .year, dt-byline .month {

 display: inline;

 }

 dt-byline .citation {

 display: block;

 text-align: left;

 }

 dt-byline .citation div {

 display: inline;

 }

 @media(min-width: 768px) {

 dt-byline {

 }

 }

 @media(min-width: 1080px) {

 dt-byline {

 border-bottom: none;

 margin-bottom: 70px;

 }

 dt-byline a:hover {

 color: rgba(0, 0, 0, 0.9);

 }

 dt-byline .authors {

 display: inline-block;

 }

 dt-byline .author {

 display: inline-block;

 margin-right: 12px;

 /\*padding-left: 20px;\*/

 /\*border-left: 1px solid #ddd;\*/

 }

 dt-byline .affiliation {

 display: block;

 }

 dt-byline .author:last-child {

 margin-right: 0;

 }

 dt-byline .name {

 display: block;

 }

 dt-byline .date {

 border-left: 1px solid rgba(0, 0, 0, 0.1);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .year, dt-byline .month {

 display: block;

 }

 dt-byline .citation {

 border-left: 1px solid rgba(0, 0, 0, 0.15);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .citation div {

 display: block;

 }

 }

[Martin Wattenberg](http://hint.fm/)

[Google Brain](http://g.co/brain)

[Fernanda Viégas](http://hint.fm/)

[Google Brain](http://g.co/brain)

[Ian Johnson](http://enjalot.github.io/)

[Google Cloud](http://cloud.google.com/)

Oct. 13

2016

[Citation:

Wattenberg, et al., 2016](#citation)

 The technique has become widespread in the field of machine learning,

since it has an almost magical ability to create compelling

two-dimensonal “maps” from data with hundreds or even thousands of

dimensions.

Although impressive, these images can be tempting to misread. The

purpose of this note is to prevent some common misreadings.

We’ll walk through a series of simple examples to illustrate what

t-SNE diagrams can and cannot show. The t-SNE technique really is

useful—but only if you know how to interpret it.

Before diving in: if you haven’t encountered t-SNE before, here’s

what you need to know about the math behind it. The goal is to take a

set of points in a high-dimensional space and find a faithful

representation of those points in a lower-dimensional space, typically

the 2D plane. The algorithm is non-linear and adapts to the underlying

data, performing different transformations on different regions. Those

differences can be a major source of confusion.

A second feature of t-SNE is a tuneable parameter, “perplexity,”

which says (loosely) how to balance attention between local and global

aspects of your data. The parameter is, in a sense, a guess about the

number of close neighbors each point has. The perplexity value has a

 But the story is more nuanced than that. Getting the most from t-SNE

may mean analyzing multiple plots with different perplexities.

That’s not the end of the complications. The t-SNE algorithm

doesn’t always produce similar output on successive runs, for example,

and there are additional hyperparameters related to the optimization

process.

---

1. Those hyperparameters really matter

--------------------------------------

Let’s start with the “hello world” of t-SNE: a data set of two

widely separated clusters. To make things as simple as possible, we’ll

consider clusters in a 2D plane, as shown in the lefthand diagram. (For

clarity, the two clusters are color coded.) The diagrams at right show

t-SNE plots for five different perplexity values.

With perplexity values in the range (5 - 50) suggested by van der

Maaten & Hinton, the diagrams do show these clusters, although with

very different shapes. Outside that range, things get a little weird.

With perplexity 2, local variations dominate. The image for perplexity

100, with merged clusters, illustrates a pitfall: for the algorithm to

operate properly, the perplexity really should be smaller than the

number of points. Implementations can give unexpected behavior

otherwise.

Each of the plots above was made with 5,000 iterations with a

learning rate (often called “epsilon”) of 10, and had reached a point of

 stability by step 5,000. How much of a difference do those values make?

 In our experience, the most important thing is to iterate until

reaching a stable configuration.

The images above show five different runs at perplexity 30. The

first four were stopped before stability. After 10, 20, 60, and 120

steps you can see layouts with seeming 1-dimensional and even pointlike

images of the clusters. If you see a t-SNE plot with strange “pinched”

shapes, chances are the process was stopped too early. Unfortunately,

there’s no fixed number of steps that yields a stable result. Different

data sets can require different numbers of iterations to converge.

Another natural question is whether different runs with the same

hyperparameters produce the same results. In this simple two-cluster

example, and most of the others we discuss, multiple runs give the same

global shape. Certain data sets, however, yield markedly different

diagrams on different runs; we’ll give an example of one of these later.

From now on, unless otherwise stated, we’ll show results from 5,000

 iterations. That’s generally enough for convergence in the (relatively

small) examples in this essay. We’ll keep showing a range of

perplexities, however, since that seems to make a big difference in

every case.

---

2. Cluster sizes in a t-SNE plot mean nothing

---------------------------------------------

So far, so good. But what if the two clusters have different

standard deviations, and so different sizes? (By size we mean bounding

box measurements, not number of points.) Below are t-SNE plots for a

mixture of Gaussians in plane, where one is 10 times as dispersed as the

 other.

Surprisingly, the two clusters look about same size in the t-SNE

plots.

 What’s going on? The t-SNE algorithm adapts its notion of “distance”

to regional density variations in the data set. As a result, it

naturally expands dense clusters, and contracts sparse ones, evening out

 cluster sizes. To be clear, this is a different effect than the

run-of-the-mill fact that any dimensionality reduction technique will

distort distances. (After all, in this example all data was

two-dimensional to begin with.) Rather, density equalization happens by

design and is a predictable feature of t-SNE.

---

3. Distances between clusters might not mean anything

-----------------------------------------------------

At perplexity 50, the diagram gives a good sense of the global

geometry. For lower perplexity values the clusters look equidistant.

When the perplexity is 100, we see the global geometry fine, but one of

the cluster appears, falsely, much smaller than the others.

 Since perplexity 50 gave us a good picture in this example, can we

always set perplexity to 50 if we want to see global geometry?

Sadly, no. If we add more points to each cluster, the perplexity

has to increase to compensate. Here are the t-SNE diagrams for three

Gaussian clusters with 200 points each, instead of 50. Now none of the

trial perplexity values gives a good result.

It’s bad news that seeing global geometry requires fine-tuning

perplexity. Real-world data would probably have multiple clusters with

different numbers of elements. There may not be one perplexity value

that will capture distances across all clusters—and sadly perplexity is a

 global parameter. Fixing this problem might be an interesting area for

future research.

---

4. Random noise doesn’t always look random.

-------------------------------------------

A classic pitfall is thinking you see patterns in what is really

just random data. Recognizing noise when you see it is a critical skill,

 but it takes time to build up the right intuitions. A tricky thing

about t-SNE is that it throws a lot of existing intuition out the

window.

 The next diagrams show genuinely random data, 500 points drawn from a

unit Gaussian distribution in 100 dimensions. The left image is a

projection onto the first two coordinates.

The plot with perplexity 2 seems to show dramatic clusters. If you

were tuning perplexity to bring out structure in the data, you might

think you’d hit the jackpot.

Of course, since we know the cloud of points was generated

randomly, it has no statistically interesting clusters: those “clumps”

aren’t meaningful. If you look back at previous examples, low perplexity

 values often lead to this kind of distribution. Recognizing these

clumps as random noise is an important part of reading t-SNE plots.

There’s something else interesting, though, which may be a win for

t-SNE. At first the perplexity 30 plot doesn’t look like a Gaussian

distribution at all: there’s only a slight density difference across

different regions of the cloud, and the points seem suspiciously evenly

distributed. In fact, these features are saying useful things about

high-dimensional normal distributions, which are very close to uniform

distributions on a sphere: evenly distributed, with roughly equal spaces

 between points. Seen in this light, the t-SNE plot is more accurate

than any linear projection could be.

---

5. You can see some shapes, sometimes

-------------------------------------

It’s rare for data to be distributed in a perfectly symmetric way.

Let’s take a look at an axis-aligned Gaussian distribution in 50

dimensions, where the standard deviation in coordinate i is 1/i. That

is, we’re looking at a long-ish ellipsoidal cloud of points.

For high enough perplexity values, the elongated shapes are easy to

 read. On the other hand, at low perplexity, local effects and

meaningless “clumping” take center stage. More extreme shapes also come

through, but again only at the right perplexity. For example, here are

two clusters of 75 points each in 2D, arranged in parallel lines with a

bit of noise.

Even in the best cases, though, there’s a subtle distortion: the

lines are slightly curved outwards in the t-SNE diagram. The reason is

that, as usual, t-SNE tends to expand denser regions of data. Since the

middles of the clusters have less empty space around them than the ends,

 the algorithm magnifies them.

---

6. For topology, you may need more than one plot

------------------------------------------------

Sometimes you can read topological information off a t-SNE plot,

but that typically requires views at multiple perplexities.

 One of the simplest topological properties is containment. The plots

below show two groups of 75 points in 50 dimensional space. Both are

sampled from symmetric Gaussian distributions centered at the origin,

but one is 50 times more tightly dispersed than the other. The “small”

distribution is in effect contained in the large one.

The perplexity 30 view shows the basic topology correctly, but

again t-SNE greatly exaggerates the size of the smaller group of points.

 At perplexity 50, there’s a new phenomenon: the outer group becomes a

circle, as the plot tries to depict the fact that all its points are

about the same distance from the inner group. If you looked at this

image alone, it would be easy to misread these outer points as a

one-dimensional structure.

What about more complicated types of topology? This may be a

subject dearer to mathematicians than to practical data analysts, but

interesting low-dimensional structures are occasionally found in the

wild.

Consider a set of points that trace a link or a knot in three

dimensions. Once again, looking at multiple perplexity values gives the

most complete picture. Low perplexity values give two completely

separate loops; high ones show a kind of global connectivity.

The trefoil knot is an interesting example of how multiple runs

affect the outcome of t-SNE. Below are five runs of the perplexity-2

view.

The algorithm settles twice on a circle, which at least preserves

the intrinsic topology. But in three of the runs it ends up with three

different solutions which introduce artificial breaks. Using the dot

color as a guide, you can see that the first and third runs are far from

 each other.

Five runs at perplexity 50, however, give results that (up to

symmetry) are visually identical. Evidently some problems are easier

than others to optimize.

---

Conclusion

----------

There’s a reason that t-SNE has become so popular: it’s incredibly

flexible, and can often find structure where other

dimensionality-reduction algorithms cannot. Unfortunately, that very

flexibility makes it tricky to interpret. Out of sight from the user,

the algorithm makes all sorts of adjustments that tidy up its

visualizations.

 Don’t let the hidden “magic” scare you away from the whole technique,

though. The good news is that by studying how t-SNE behaves in simple

cases, it’s possible to develop an intuition for what’s going on.

.tsne-group {

 overflow: visible;

 display: -webkit-flex;

 display: flex;

 flex-direction: column;

 margin-top: 36px;

 margin-bottom: 36px;

}

@media(min-width: 640px) {

 .tsne-group {

 -webkit-flex-direction: row;

 flex-direction: row;

 }

}

.tsne-group .original,

.tsne-group .runner {

 box-sizing: border-box;

 -webkit-flex-grow: 1;

 flex-grow: 1;

 position: relative;

 display: -webkit-flex;

 display: flex;

 margin-bottom: 12px;

}

.tsne-group .original {

 position: relative;

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 padding-bottom: 12px;

}

@media(min-width: 640px) {

 .tsne-group .original,

 .tsne-group .runner {

 display: block;

 width: 100%;

 margin-bottom: 0;

 }

 .tsne-group .original {

 margin-left: 0;

 padding-right: 12px;

 border-right: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: none;

 padding-bottom: 0;

 }

 .tsne-group .runner {

 margin-left: 12px;

 }

}

.tsne-group .runner.no-click {

 cursor: default;

}

.runner.clickable {

 cursor: pointer;

}

.runner.clickable:hover .image {

 box-shadow: 0 2px 12px rgba(0, 0, 0, 0.08);

}

.runner:hover i {

 opacity: 1;

}

.runner.selected i {

 opacity: 1;

}

.runner i {

 font-size: 30px;

 position: absolute;

 top: 1px;

 left: 1px;

 background-color: hsla(207, 44%, 79%, 0.8);

 color: steelblue;

 border-radius: 4px;

 opacity: 0;

 transition: opacity 0.3s;

 line-height: 0;

 width: 150px;

 padding: 75px 0;

 text-align: center;

}

@media(min-width: 640px) {

 .runner i {

 width: 100%;

 padding: 50% 0;

 }

}

.runner.selected .image::after {

 content: "";

 border: 2px solid rgba(70, 130, 180, 1);

 width: 100%;

 height: 100%;

 position: absolute;

 top: 0;

 box-sizing: border-box;

 border-radius: 4px;

 z-index: 10;

}

.tsne-group .image {

 border: 1px solid rgba(0, 0, 0, 0.1);

 border-radius: 4px;

 box-shadow: 0 0 6px rgba(0, 0, 0, 0.08);

 width: 150px;

 padding-top: 150px;

 background: white;

 position: relative;

 transition: border-color 0.3s, box-shadow 0.3s;

 margin-right: 12px;

}

@media(min-width: 640px) {

 .tsne-group .image {

 width: 100%;

 padding-top: 100%;

 margin-right: 0;

 }

}

.tsne-group .original .image {

 box-shadow: none;

}

.tsne-group .image img {

 position: absolute;

 top: 5%;

 left: 5%;

 width: 90%;

}

.tsne-group h3.caption {

 font-size: 15px;

 margin-top: 12px;

}

.tsne-group .caption {

 font-size: 12px;

 line-height: 1.6em;

 margin-top: 12px;

 width: 100px;

}

@media(min-width: 640px) {

 .tsne-group .caption,

 .tsne-group h3.caption {

 margin-top: 8px;

 width: 100%;

 }

}

(function() {

 var format = d3.format(",");

 d3.selectAll(".tsne-group").data(figures, function(d) {

 if(!d) return this.dataset.id;

 return d.id

 })

 .each(generateFigure)

 function generateFigure(figure) {

 var FIGURE = figure.id

 var dis = d3.select(this)

 var original = dis.append("div").classed("original", true)

 original.append("div")

 .classed("image", true)

 .append("img")

 .classed("tsne-plot", true)

 .attr("src", "assets/figure\_" + FIGURE + "\_" + "original.png");

 original.append("h3").classed("caption", true).text("Original")

 // examples

 var runners = dis.selectAll(".runner").data(figure.examples)

 .enter().append("div").classed("runner", true)

 if(!figure.noclick) {

 runners

 .classed("clickable", true)

 .on("click", function(d) {

 d3.selectAll(".runner").classed("selected", false)

 updateStateFromFigure(figure, d, this);

 d3.select(this).classed("selected", true)

 });

 } else {

 runners.classed("no-click", true)

 }

 runners.append("div")

 .classed("image", true)

 .append("img")

 .attr("src", function(d,i) {

 return "assets/figure\_" + FIGURE + "\_" + d.id + ".png"

 });

 if(!figure.noclick) {

 runners.append("i")

 .classed("material-icons", true)

 .text("open\_in\_browser")

 }

 var caption = runners.append("div").classed("caption", true);

 caption.append("div").text(function(d) { return "Step: " + format(d.step); })

 }

})()