 Memorization in Recurrent Neural Networks (RNNs) continues to pose a challenge
 in many applications. We’d like RNNs to be able to store information over many
 timesteps and retrieve it when it becomes relevant — but vanilla RNNs often struggle to do this.

 Several network architectures have been proposed to tackle aspects of this problem, such
 as Long-Short-Term Memory (LSTM)
 units and Gated Recurrent Units (GRU).
 However, the practical problem of memorization still poses a challenge.
 As such, developing new recurrent units that are better at memorization
 continues to be an active field of research.

 To compare a recurrent unit against its alternatives, both past and recent
 papers, such as the Nested LSTM paper by Monzi et al.
 , heavily rely on quantitative
 comparisons. These comparisons often measure accuracy or
 cross entropy loss on standard problems such as Penn Treebank, Chinese
 Poetry Generation, or text8, where the task is to predict the
 next character given existing input.

 While quantitative comparisons are useful, they only provide partial
 insight into the how a recurrent unit memorizes. A model can, for example,
 achieve high accuracy and cross entropy loss by just providing highly accurate
 predictions in cases that only require short-term memorization, while
 being inaccurate at predictions that require long-term
 memorization.
 For example, when autocompleting words in a sentence, a model with only short-term understanding could still exhibit high accuracy completing the ends of words once most of the characters are present.
 However, without longer term contextual understanding it won’t be able to predict words when only a few characters are known.

 This article presents a qualitative visualization method for comparing
 recurrent units with regards to memorization and contextual understanding.
 The method is applied to the three recurrent units mentioned above: Nested LSTMs, LSTMs, and GRUs.

Recurrent Units
---------------

 The networks that will be analyzed all use a simple RNN structure:

 Output for layer  at time .

 Recurrent unit of choice.

 In theory, the time dependency allows it in each iteration to know
 about every part of the sequence that came before. However, this time
 dependency typically causes a vanishing gradient problem that results in
 long-term dependencies being ignored during training
 .

**Vanishing Gradient:**  where the contribution from the
 earlier steps becomes insignificant in the gradient for the vanilla RNN
 unit.

 Several solutions to the vanishing gradient problem have been proposed over
 the years. The most popular are the aforementioned LSTM and GRU units, but this
 is still an area of active research. Both LSTM and GRU are well known
 and [thoroughly explained in literature](http://colah.github.io/posts/2015-08-Understanding-LSTMs/). Recently, Nested LSTMs have also been proposed
  — an explanation of Nested LSTMs
 can be found [in the appendix](#appendix-nestedlstm).

![Nested LSTM Diagram](graphics/nlstm-web.svg)

**Recurrent Unit, Nested LSTM:** makes the cell update depend on another
 LSTM unit, supposedly this allows more long-term memory compared to
 stacking LSTM layers.

![Long Short Term Memory Diagram](graphics/lstm-web.svg)

**Recurrent Unit, LSTM:** allows for long-term
 memorization by gateing its update, thereby solving the vanishing gradient
 problem.

![Gated Recurrent Unit Diagram](graphics/gru-web.svg)

**Recurrent Unit, GRU:** solves the vanishing gradient
 problem without depending on an internal memory state.

 It is not entirely clear why one recurrent unit performs better than another
 in some applications, while in other applications it is another type of
 recurrent unit that performs better. Theoretically they all solve the vanishing
 gradient problem, but in practice their performance is highly application
 dependent.

 Understanding why these differences occur is likely an opaque and
 challenging problem. The purpose of this article is to demonstrate a
 visualization technique that can better highlight what these differences
 are. Hopefully, such an understanding can lead to a deeper understanding.

Comparing Recurrent Units
-------------------------

 Comparing different Recurrent Units is often more involved than simply comparing the accuracy or cross entropy
 loss. Differences in these high-level quantitative measures
 can have many explanations and may only be because of some small improvement
 in predictions that only requires short-term contextual understanding,
 while it is often the long-term contextual understanding that is of interest.

### A problem for qualitative analysis

 Therefore a good problem for qualitatively analyzing contextual
 understanding should have a human-interpretable output and depend both on
 long-term and short-term contextual understanding. The typical problems
 that are often used, such as Penn Treebank, Chinese Poetry Generation, or
 text8  generation do not have outputs that are easy to reason about, as they
 require an extensive understanding of either grammar, Chinese poetry, or
 only output a single letter.

 To this end, this article studies the autocomplete problem. Each character is mapped
 to a target that represents the entire word. The space leading up to the word should also map to that target.
 This prediction based on the space character is in particular useful for showing contextual understanding.

 The autocomplete problem is quite similar to the text8 generation
 problem: the only difference is that instead of predicting the next letter,
 the model predicts an entire word. This makes the output much more
 interpretable. Finally, because of its close relation to text8 generation,
 existing literature on text8 generation is relevant and comparable,
 in the sense that models that work well on text8 generation should work
 well on the autocomplete problem.

User types input sequence.
Recurrent neural network processes the sequence.
The output for the last character is used.
The most likely suggestions are extracted.

**Autocomplete:** An application that has a humanly
 interpretable output, while depending on both short and long-term
 contextual understanding. In this case, the network uses past information
 and understands the next word should be a country.

 The output in this figure was produced by the GRU model;
 all model setups are [described in the appendix](#appendix-autocomplete).

 Try [removing the last letters](javascript:arDemoShort();) to see
 that the network continues to give meaningful suggestions.

 ([reset](javascript:arDemoReset();)).

 The autocomplete dataset is constructed from the full
 [text8](http://mattmahoney.net/dc/textdata.html) dataset. The
 recurrent neural networks used to solve the problem have two layers, each
 with 600 units. There are three models, using GRU, LSTM, and Nested LSTM.
 See [the appendix](#appendix-autocomplete) for more details.

### Connectivity in the Autocomplete Problem

 In the recently published Nested LSTM paper
 , they qualitatively compared their
 Nested LSTM unit to other recurrent units, to show how it memorizes in
 comparison, by visualizing individual cell activations.

 This visualization was inspired by Karpathy et al.
  where they identify cells
 that capture a specific feature. To identify a specific
 feature, this visualization approach works well. However, it is not a useful
 argument for memorization in general as the output is entirely dependent
 on what feature the specific cell captures.

 Instead, to get a better idea of how well each model memorizes and uses
 memory for contextual understanding, the connectivity between the desired
 output and the input is analyzed. This is calculated as:

 Input time index.

 Output time index.

 Magnitude of the gradient, between the logits for the desired output  and the input
 .

 Exploring the connectivity gives a surprising amount of insight into the
 different models’ ability for long-term contextual understanding. Try and
 interact with the figure below yourself to see what information the
 different models use for their predictions.

**Connectivity:** the connection strength between
 the target for the selected character and the input characters is highlighted in green
 ([reset](javascript:connectivitySetIndex(null);)).
 *Hover over or tap the text to change the selected character.*

 Let’s highlight three specific situations:

1

 Observe how the models predict the word “learning” with [only the first two
 characters as input](javascript:connectivitySetIndex(106);). The Nested LSTM model barely uses past
 information and thus only suggests common words starting with the letter “l”.

 In contrast, the LSTM and GRU models both suggests the word “learning”.
 The GRU model shows stronger connectivity with the word “advanced”,
 and we see in the suggestions that it predicts a higher probability for “learning” than the LSTM model.

2

 Examine how the models predict the word “grammar”.
 This word appears twice; when it appears for the first time the models have very little context.
 Thus, no model suggests “grammar” until it has
 [seen at least 4 characters](javascript:connectivitySetIndex(32);).

 When “grammar” appears for the second time, the models have more context.
 The GRU model is able to predict the word “grammar” with only [1 character from the word itself](javascript:connectivitySetIndex(159);). The LSTM and Nested LSTM again
 need [at least 4 characters](javascript:connectivitySetIndex(162);).

3

 Finally, let’s look at predicting the word “schools”
 [given only the first character](javascript:connectivitySetIndex(141);). As in the other cases,
 the GRU model seems better at using past information for
 contextual understanding.

 What makes this case noteworthy is how the LSTM model appears to
 use words from almost the entire sentence as context. However,
 its suggestions are far from correct and have little to do
 with the previous words it seems to use in its prediction.
 This suggests that the LSTM model in this setup is capable of
 long-term memorization, but not long-term contextual understanding.

1
2
3

*The colored number links above change the connectivity figure’s displayed timestep and explanation.*

 These observations show that the connectivity visualization is a powerful tool
 for comparing models in terms of which previous inputs they use for contextual understanding.
 However, it is only possible to compare models on the same dataset, and
 on a specific example. As such, while these observations may show that
 Nested LSTM is not very capable of long-term contextual understanding in this example;
 these observations may not generalize to other datasets or hyperparameters.

### Future work; quantitative metric

 From the above observations it appears that short-term contextual understanding
 often involves the word that is being predicted itself. That is, the models switch to
 using previously seen letters from the word itself, as more letters become
 available. In contrast, at the beginning of predicting a word, models — especially the
 GRU network — use previously seen words as context for the prediction.

 This observation suggests a quantitative metric: measure the accuracy given
 how many letters from the word being predicted are already known.
 It is not clear that this is best quantitative metric: it is highly problem dependent,
 and it also doesn’t summarize the model to a single number, which one may wish for a more direct comparison.

**Accuracy Graph**: shows the accuracy
 given a fixed number of characters in a word that the RNN has seen.
 0 characters mean that the RNN has only seen the space leading up
 to the word, including all the previous text which should provide context.
 The different line styles, indicates if the correct word should appear
 among the top 1, 2, or 3 suggestions.

 These results suggest that the GRU model is better at long-term contextual
 understanding, while the LSTM model is better at short-term contextual
 understanding. These observations are valuable, as it justifies why the
 [overall accuracy of the GRU and LSTM models](#ar-overall-accuracy)  are almost identical, while the connectivity visualization shows that
 the GRU model is far better at long-term contextual understanding.

 While more detailed quantitative metrics like this provides new insight,
 qualitative analysis like the connectivity figure presented
 in this article still has great value. As the connectivity visualization gives an
 intuitive understanding of how the model works, which a quantitative metric
 cannot. It also shows that a wrong prediction can still be considered a
 useful prediction, such as a synonym or a contextually reasonable
 prediction.

Conclusion
----------

 Looking at overall accuracy and cross entropy loss in itself is not that
 interesting. Different models may prioritize either long-term or
 short-term contextual understanding, while both models can have similar
 accuracy and cross entropy.

 A qualitative analysis, where one looks at how previous input is used in
 the prediction is therefore also important when judging models. In this
 case, the connectivity visualization together with the autocomplete
 predictions, reveals that the GRU model is much more capable of long-term
 contextual understanding, compared to LSTM and Nested LSTM. In the case of
 LSTM, the difference is much higher than one would guess from just looking
 at the overall accuracy and cross entropy loss alone. This observation is
 not that interesting in itself as it is likely very dependent on the
 hyperparameters, and the specific application.

 Much more valuable is that this visualization method makes it possible
 to intuitively understand how the models are different, to a much higher
 degree than just looking at accuracy and cross entropy. For this application,
 it is clear that the GRU model uses repeating words and semantic meaning
 of past words to make its prediction, to a much higher degree than the LSTM
 and Nested LSTM models. This is both a valuable insight when choosing the
 final model, but also essential knowledge when developing better models
 in the future.

Visualizing memorization in RNNs

Summary: Inspecting gradient magnitudes in context can be a powerful tool to see when recurrent units use short-term or long-term contextual understanding.

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Nested LSTMs

The Penn Treebank: Annotating Predicate Argument Structure

text8 Dataset

On the difficulty of training recurrent neural networks

Visualizing and Understanding Recurrent Networks


Introduction
------------

Neural networks can learn to classify images more accurately than any system humans directly design. This raises a natural question: What have these networks learned that allows them to classify images so well?

Feature visualization is a thread of research that tries to answer this question by letting us “see through the eyes” of the network . It began with research into [visualizing individual neurons](https://distill.pub/2017/feature-visualization/) and trying to determine what they respond to. Because neurons don’t work in isolation, this led to applying feature visualization to [simple combinations of neurons](https://distill.pub/2017/feature-visualization/#interaction). But there was still a problem — what combinations of neurons should we be studying? A natural answer (foreshadowed by work on model inversion ) is to [visualize activations](https://distill.pub/2018/building-blocks/#ActivationGridSingle) , the combination of neurons firing in response to a particular input.

These approaches are exciting because they can make the hidden layers of networks comprehensible. These layers are the heart of how neural networks outperform more traditional approaches to machine learning and historically, we’ve had little understanding of what happens in themWith the exception of the first hidden layer.. Feature visualization addresses this by connecting hidden layers back to the input, making them meaningful.

Unfortunately, visualizing activations has a major weakness — it is limited to seeing only how the network sees a single input. Because of this, it doesn’t give us a big picture view of the network. When what we want is a map of an entire forest, inspecting one tree at a time will not suffice.

 There are techniques which give a more global view, but they tend to have other downsides.
 For example, Karpathy’s [CNN codes visualization](https://cs.stanford.edu/people/karpathy/cnnembed/)
 gives a global view of a dataset by taking each image and organizing them by their activation values from a neural network.
 Showing which images the model sees as similar does help us infer some ideas about what features the network is responding to,
 but feature visualization makes those connections much more explicit.
 Nguyen, et al  use t-SNE to make more diverse neuron visualizations,
 generating diverse starting points for the optimization process by clustering images in the t-SNE map.
 This reveals a broader picture of what the neuron detects but is still focused on individual neurons.

In this article we introduce *activation atlases* to this quiver of techniques. (An example is shown at the top of this article.) Broadly speaking, we use a technique similar to the one in CNN codes, but instead of showing input data, we show feature visualizations of averaged activations. By combining these two techniques, we can get the advantages of each in one view — a global map seen through the eyes of the network.

In theory, showing the feature visualizations of the [basis neurons](https://distill.pub/2017/feature-visualization/appendix/) would give us the global view of a network that we are seeking. In practice, however, neurons are rarely used by the network in isolation, and it may be difficult to understand them that way. As an analogy, while the 26 letters in the alphabet provide a basis for English, seeing how letters are commonly combined to make words gives far more insight into the concepts that can be expressed than the letters alone. Similarly, activation atlases give us a bigger picture view by showing common combinations of neurons.

 These atlases not only reveal visual abstractions within a model, but later in the article we will show that they can reveal high-level misunderstandings in a model that can be exploited. For example, by looking at an activation atlas we will be able to see why a picture of a baseball can switch the classification of an image from “grey whale” to “great white shark”.

Of course, activation atlases do have limitations. In particular, they’re dependent on the distribution of the data we choose to sample activations from (in our examples, we use one million images chosen at random from the ImageNet dataset  training data). As a result, they will only show the activations that exist within the distribution of the sample data. However, while it’s important to be aware of these limitations — we’ll discuss them in much more depth later! — Activation Atlases still give us a new kind of overview of what neural networks can represent.

Looking at a Single Image
-------------------------

Before we dive into Activation Atlases, let’s briefly review how we use feature visualization to make activation vectors meaningful (“see through the network’s eyes”). This technique was introduced in [Building Blocks](https://distill.pub/2018/building-blocks/) , and will be the foundation of Activation Atlases.

 Throughout this article, we’ll be focussing on a particular neural network: InceptionV1
 (also known as “GoogLeNet”).
 When it came out, it was notable for [winning](http://www.image-net.org/challenges/LSVRC/2014/results#clsin) the classification task in the 2014 ImageNet Large Scale Visual Recognition Challenge .

 InceptionV1 consists of a number of layers, which we refer to as “mixed3a”, “mixed3b”, “mixed4a”, etc., and sometimes shortened to just “3a”. Each layer successively builds on the previous layers.

InceptionV1 builds up its understanding of images over several layers (see [overview](https://distill.pub/2017/feature-visualization/appendix/) from ). It was trained on ImageNet ILSVRC . Each layer actually has several component parts, but for this article we’ll focus on these larger groups.

To visualize how InceptionV1 sees an image, the first step is to feed the image into the network and run it through to the layer of interest. Then we collect the activations — the numerical values of how much each neuron fired. If a neuron is excited by what it is shown, its activation value will be positive.

Unfortunately these vectors of activation values are just vectors of unitless numbers and not particularly interpretable by people. This is where [feature visualization](https://distill.pub/2017/feature-visualization/) comes in.

 Roughly speaking, we can think of feature visualization as creating an idealized image of what the network thinks would produce a particular activation vector. Whereas we normally use a network to transform an image into an activation vector, in feature visualization we go in the opposite direction. Starting with an activation vector at a particular layer, we create an image through an iterative optimization process.

Because InceptionV1 is a [convolutional network](http://colah.github.io/posts/2014-07-Conv-Nets-Modular/), there is not just one activation vector per layer per image.
 This means that the same neurons are run on each patch of the previous layer.
 Thus, when we pass an entire image through the network, each neuron will be evaluated hundreds of times, once for each overlapping patch of the image. We can consider the vectors of how much each neuron fired for each patch separately.

The result is a grid of feature visualizations, one for each patch. This shows us how the network sees different parts of the input image.

Aggregating Multiple Images
---------------------------

Activation grids show how the network sees a single image, but what if we want to see more? What if we want to understand how it reacts to millions of images?

Of course, we could look at individual activation grids for those images one by one. But looking at millions of examples doesn’t scale, and human brains aren’t good at comparing lots of examples without structure. In the same way that we need a tool like a histogram in order to understand millions of numbers, we need a way to aggregate and organize activations if we want to see meaningful patterns in millions of them.

Let’s start by collecting activations from one million images. We’ll randomly select one spatial activation per image.We avoid the edges due to boundary effects. This gives us one million activation vectors. Each of the vectors is high-dimensional, perhaps 512 dimensions! With such a complex set of data, we need to organize and aggregate it if we want a big picture view.

Thankfully, we have modern dimensionality reduction techniques at our disposal. These algorithms, such as t-SNE  and UMAP , can project high-dimensional data like our collection of activation vectors into useful 2D layouts, preserving some of the local structure of the original space. This takes care of organizing our activation vectors, but we also need to aggregate into a more manageable number of elements — one million dots would be hard to interpret. We’ll do this by drawing a grid over the 2D layout we created with dimensionality reduction. For each cell in our grid, we average all the activations that lie within the boundaries of that cell, and use feature visualization to create an iconic representation.

A randomized set of one million images is fed through the network, collecting one random spatial activation per image.
The activations are fed through UMAP to reduce them to two dimensions. They are then plotted, with similar activations placed near each other.
We then draw a grid and average the activations that fall within a cell and run feature inversion on the averaged activation. We also optionally size the grid cells according to the density of the number of activations that are averaged within.

 We perform feature visualization with the regularizations described in Feature Visualization  (in particular, [transformation robustness](https://distill.pub/2017/feature-visualization/#regularizer-playground-robust)).
 However, we use a slightly non-standard objective.
 Normally, to visualize a direction in activation space, vvv, one maximizes the dot product with the activation vector hhh at a position: hx,y⋅vh\_{x,y} \cdot vhx,y⋅v.
 We find it helpful to use an objective that emphasizes angle more heavily by multiplying the dot product by cosine similarity, leading to objectives of the following form:
 (hx,y⋅v)n+1(∣∣hx,y∣∣⋅∣∣v∣∣)n\frac{(h\_{x,y} \cdot v)^{n+1}}{(||h\_{x,y}|| \cdot ||v||)^{n}}(∣∣hx,y∣∣⋅∣∣v∣∣)n(hx,y⋅v)n+1.
 We also find that whitening the activation space to unstretch it can help improve feature visualization.
 We don’t yet fully understand this phenomenon.
 A reference implementation of this can be seen in the attached notebooks, and more general discussion can be found in this [github issue](https://github.com/tensorflow/lucid/issues/116).

 For each activation vector, we also compute an *attribution* vector.
 The attribution vector has an entry for each class, and approximates the amount that the activation vector influenced the logit for each class.
 Attribution vectors generally depend on the surrounding context.
 We follow Building Blocks  in computing the attribution of the activation vector at a position, hx,yh\_{x,y}hx,y, to a class logit, logitc\text{logit}\_clogitc as hx,y⋅∇hx,ylogitch\_{x,y} \cdot \nabla\_{h\_{x,y}} \text{logit}\_chx,y⋅∇hx,ylogitc.
 That is, we estimate that the effect of a neuron on a logit is the rate at which increasing the neuron affects the logit.
 This is similar to Grad-CAM , but without the spatial averaging of the gradient.
 Instead, we reduce noise in the gradient by using a continuous relaxation of the gradient for max pooling in computing the gradient (as in ).
 (A detailed reference implementation can be found in [this notebook](https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/building-blocks/AttrSpatial.ipynb).)
 The attribution shown for cells in the activation atlas is the average of attribution vectors for activations in that cell.

 This average attribution can be thought of as showing what classes that cell tends to support, marginalizing over contexts.
 At early layers, the average attribution is very small and the top classes are fairly arbitrary because low-level visual features like textures tend to not be very discriminative without context.

So, how well does this work? Well, let’s try applying it to InceptionV1  at [layer mixed4c](https://distill.pub/2017/feature-visualization/appendix/googlenet/4c.html).

This atlas can be a bit overwhelming at first glance — there’s a lot going on! This diversity is a reflection of the variety of abstractions and concepts the model has developed. Let’s take a tour to examine this atlas in more depth.

If we look at the top-left of the atlas, we see images which look like animal heads. There is some differentiation between different types of animals, but it seems to be more a collection of elements of generic mammals — eyes, fur, noses — rather than a collection of different classes of animals. We’ve also added labels that show which class each averaged activation most contributes to. Please note, in some areas of a layer this early in the network these attribution labels can be somewhat chaotic. In early layers the attribution vectors have a small magnitude since they don’t have a consistent effect on the output.

As we move further down we start to see different types of fur and the backs of four-legged animals.

Below this, we find different animal legs and feet resting on different types of ground.

Below the feet we start to lose any identifiable parts of animals, and see isolated grounds and floors. We see attribution toward environments like “sandbar” and also toward things that are found on the ground, like “doormat” or “ant”.

These sandy, rocky backgrounds slowly blend into beaches and bodies of water. Here we see lakes and oceans, both above and below water. Though the network does have certain classes like “seashore”, we see attribution toward many sea animals, without any visual references to the animals themselves. While not unexpected, it is reassuring to see that the activations that are used to identify the sea for the class “seashore” are the same ones used when classifying “starfish” or “sea lion”. There is also no real distinction at this point between lakes and ocean — “lakeside” and “hippopotamus” attributions are intermingled with “starfish” and “stingray”.

Now let’s jump to the other side of the atlas, where we can see many variations of text detectors. These will be useful when identifying classes such as “menu”, “web site” or “book jacket”.

Moving upward, we see many variations of people. There are very few classes that specifically identify people in ImageNet, but people are present in lots of the images. We see attribution toward things people use (“hammer”, “flute”), clothes that people wear (“bow tie”, “maillot”) and activities that people participate in (“basketball”). There is a uniformity to the skin color in these visualizations which we suspect is a reflection of the distribution of the data used for training. (You can browse the ImageNet training data by category online: [swimming trunks](http://image-net.org/synset?wnid=n04371430#), [diaper](http://image-net.org/synset?wnid=n03188531), [band aid](http://image-net.org/synset?wnid=n02786058), [lipstick](http://image-net.org/synset?wnid=n03676483), etc.)

And finally, moving back to the left, we can see round food and fruit organized mostly by colors — we see attribution toward “lemon”, “orange” and “fig”.

We can also trace curved paths through this manifold that we’ve created. Not only are regions important, but certain movements through the space seem to correspond to human interpretable qualities. With the fruit, we can trace a path that seems to correlate with the size and number of fruits in the frame.

Similarly, with people, we can trace a path that seems to correspond to how many people are in the frame, whether it’s a single person or a crowd.

With the ground detectors, we can trace a path from water to beach to rocky cliffs.

In the plants region, we can trace a path that seems to correspond to how blurry the plant is. This could possibly be used to determine relative size of objects because of the typical focal lengths of cameras. Close up photos of small insects have more opportunity for blurry background foliage than photos of larger animals, like monkeys.

It is important to note that these paths are constructed after the fact in the low-dimensional projection. They are smooth paths in this reduced projection but we don’t necessarily know how the paths operate in the original higher-dimensional activation space.

Looking at Multiple Layers
--------------------------

In the previous section we focused on one layer of the network, mixed4c, which is in the middle of our network. Convolutional networks are generally deep, consisting of many layers that progressively build up more powerful abstractions. In order to get a holistic view, we must look at how the model’s abstractions develop over several layers.

In early layers (eg. [mixed3b](”https://distill.pub/2017/feature-visualization/appendix/googlenet/3b.html”)), the network seems to represents textures and simple patterns.
 By mid layers (eg. [mixed4c](”https://distill.pub/2017/feature-visualization/appendix/googlenet/4c.html”)), icons evoke individual concepts like eyes, leaves, water, that are shared among many classes.
 In the final layers (eg. [mixed5b](”https://distill.pub/2017/feature-visualization/appendix/googlenet/5b.html”)), abstractions become more closely aligned with the output classes.

To start, let’s compare three layers from different areas of the network to try to get a sense for the different personalities of each — one very early layer ([mixed3b](”https://distill.pub/2017/feature-visualization/appendix/googlenet/3b.html”)), one layer from the middle ([mixed4c](”https://distill.pub/2017/feature-visualization/appendix/googlenet/4c.html”)), and the final layer ([mixed5b](”https://distill.pub/2017/feature-visualization/appendix/googlenet/5b.html”)) before the logits. We’ll focus on areas of each layer that contribute to the classification of “cabbage”.

 .vertical-layer-comparison {
 display: grid;
 grid-column: page;
 grid-template-columns: 1fr 1fr 1fr;
 grid-gap: 16px 20px;
 }
 .vertical-layer-comparison h4 {
 margin-bottom: 0;
 padding-bottom: 4px;
 width: 100%;
 border-bottom: solid #CCC 1px;
 }
 .vertical-layer-comparison.progress {
 display: block;
 max-width: 704px;
 padding: 0 20px;
 margin: 0 auto;
 }
 .vertical-layer-comparison.progress > div {
 display: grid;
 grid-template-columns: 2.5fr 1fr;
 grid-gap: 20px;
 margin-bottom: 16px;
 }
 .vertical-layer-comparison.progress h4 {
 margin-top: 0;
 margin-bottom: 16px;
 }
 .vertical-layer-comparison.progress .figcaption {
 margin-top: 8px;
 }

#### Mixed3b

#### Mixed4c

#### Mixed5b

As you move through the network, the later layers seem to get much more specific and complex. This is to be expected, as each layer builds its activations on top of the preceding layer’s activations. The later layers also tend to have larger receptive fields than the ones that precede them (meaning they are shown larger subsets of the image) so the concepts seem to encompass more of the whole of objects.

There is another phenomenon worth noting: not only are concepts being refined, but new concepts are appearing out of combinations of old ones. Below, you can see how sand and water are distinct concepts in a middle layer, mixed4c, both with strong attributions to the classification of “sandbar”. Contrast this with a later layer, mixed5b, where the two ideas seem to be fused into one activation.

#### Mixed4c

#### Mixed5b

Finally, if we zoom out a little, we can see how the broader shape of the activation space changes from layer to layer. By looking at similar regions in several consecutive layers, we can see concepts getting refined and differentiated — In mixed4a we see very vague, generic blob, which gets refined into much more specific “peninsulas” by mixed4e.

#### Mixed4a

 In mixed4a there is a vague “mammalian” area.

#### Mixed4b

 By mixed4b, animals and people have been disentangled, with some fruit and food emerging in the middle.

#### Mixed4c

 All the concepts are further refined and differentiated into small “peninsulas”.

#### Mixed4d

 The specialization continues in mixed4d.

#### Mixed4e

 And further still in mixed4e.

Below you can browse many more of the layers of InceptionV1. You can compare the
 [curved edge detectors of mixed4a](#) with the
 [bowls and cups of mixed5b](#). Mixed4b has some
 [interesting text and pattern detectors](#), whereas mixed5a appears to use those to differentiate
 [menus from crossword puzzles from rulers](#). In early layers, like mixed4b, you’ll see things that have similar textures near each other, like
 [fabrics](#). In later layers, you’ll see
 [specific types of clothing](#).

Focusing on a Single Classification
-----------------------------------

Looking at an atlas of all activations can be a little overwhelming, especially when you’re trying to understand how the network goes about ranking one particular class. For instance, let’s investigate how the network classifies a “fireboat”.

An image labeled “fireboat” from ImageNet.

We’ll start by looking at an atlas for the last layer, mixed5b. Instead of showing all the activations, however, we’ll calculate the amount that each activation contributes toward a classification of “fireboat” and then map that value to the opacity of the activation icon.In the case of mixed5b, determining this contribution is fairly straightforward because the relationship between activations at mixed5b and the logit values is linear. When there are multiple layers between our present one and the output — and as a result, the relationship is non-linear — it’s a little less clear what to do. In this article, we take the simple approach of forming a linear approximation of these future layers and use it to approximate the effect of our activations. The areas that contribute a lot toward a classification of “fireboat” will be clearly visible, whereas the areas that contribute very little (or even contribute negatively) will be completely transparent.

The layer we just looked at, mixed5b, is located just before the final classification layer so it seems reasonable that it would be closely aligned with the final classes. Let’s look at a layer a little earlier in the network, say mixed4d, and see how it differs.

Here we see a much different pattern. If we look at some more input examples, this seems entirely reasonable. It’s almost as if we can see a collection of the component concepts the network will use in later layers to classify “fireboat”. Windows + crane + water = “fireboat”.

One of the clusters, the one with windows, has strong attribution to “fireboat”, but taken on its own, it has an even stronger attribution toward “streetcar”. So, let’s go back to the atlas at mixed4d, but isolate “streetcar” and compare it to the patterns seen for “fireboat”. Let’s look more closely at the four highlighted areas: the three areas we highlighted for fireboat plus one additional area that is highly activated for streetcars.

If we zoom in, we can get a better look at what distinguishes the two classifications at this layer. (We’ve cherry-picked these examples for brevity, but you can explore all the layers and activations in detail in a explorable playground below.)

If we look at a couple of input examples, we can see how buildings and water backgrounds are an easy way to differentiate between a “fireboat” and a “streetcar”.

Images from ImageNet

By isolating the activations that contribute strongly to one class and comparing it to other class activations, we can see which activations are conserved among classes and which are recombined to form more complex activations in later layers. Below you can explore the activation patterns of many classes in ImageNet through several layers of InceptionV1. You can even explore negative attributions, which we ignored in this discussion.

Further Isolating Classes
-------------------------

Highlighting the class-specific activations in situ of a full atlas is helpful for seeing how that class relates to the full space of what a network “can see.” However, if we want to really isolate the activations that contribute to a specific class we can remove all the other activations rather than just dimming them, creating what we’ll call a *class activation atlas*. Similar to the general atlas, we run dimensionality reductionFor class activations we generally have better results from using t-SNE for the dimensionality reduction step rather than UMAP. We suspect it is because the data is much more sparse. over the class-specific activation vectors in order to arrange the feature visualizations shown in the class activation atlas.

A class activation atlas gives us a much clearer view of which detectors the network is using to rank a specific class. In the “snorkel” example we can clearly see ocean, underwater, and colorful masks.

In the previous example, we are only showing those activations whose strongest attribution is toward the class in question. This will show us activations that contribute mostly to our class in question, even if their overall strength is low (like in background detectors). In some cases, though, there are strong correlations that we’d like to see (like fish with snorkelers). These activations on their own might contribute more strongly to a different class than the one we’re interested in, but their existence can also contribute strongly to our class of interest. For these we need to choose a different filtering method.

Using the magnitude filtering method, let’s try to compare two related classes and see if we can more easily see what distinguishes them. (We could have instead used rank, or a combination of the two, but magnitude will suffice to show us a good variety of concepts).

It can be a little hard to immediately understand all the differences between classes. To help make the comparison easier, we can combine the two views into one. We’ll plot the difference between the attributions of the “snorkel” and “scuba diver” horizontally, and use t-SNE  to cluster similar activations vertically.

In this comparison we can see some bird-like creatures and clear tubes on the left, implying a correlation with “snorkel”, and some shark-like creatures and something round, shiny, and metallic on the right, implying correlation with “scuba diver” (This activation has a strong attribution toward the class “steam locomotive”). Let’s take an image from the ImageNet dataset labeled as “snorkel” and add something that resembles this icon to see how it affects the classification scores.

The failure mode here seems to be that the model is using its detectors for the class “steam locomotive” to identify air tanks to help classify “scuba diver”. We’ll call these “multi-use” features — detectors that react to very different concepts that are nonetheless visually similar. Let’s look at the differences between a “grey whale” and a “great white shark” to see another example of this issue.

In this example we see another detector that seems to be playing two roles: detecting red stitching on a baseball and a sharks’s white teeth and pink inner mouth. This detector also shows up in the [activation atlas at layer mixed5b filtered to “great white shark”](#focus-playground) and its attribution points towards all sorts of balls, the top one being “baseball”.

Let’s add a picture of a baseball to a picture of a “grey whale” from ImageNet and see how it effects the classification.

The results follow the pattern in previous examples pretty closely.
 Adding a small-sized baseball does change the top classification to “great white shark”, and as it gets bigger it overpowers the classification, so the top slot goes to “baseball”.

Let’s look at one more example: “frying pan” and “wok”.

One difference stands out here — the type of related foods present. On the right we can clearly see something resembling noodles (which have a strong attribution toward the class “carbonara”). Let’s take a picture from ImageNet labeled as “frying pan” and add an inset of some noodles.

Here the patch was not as effective at lowering the initial classification, which makes sense since the noodle-like icons were plotted more toward the center of the visualization thus having less of a difference in attribution. We suspect that the training set simply contained more images of woks with noodles than frying pans with noodles.

### Testing dozens of patches on thousands of images

So far we’ve only shown single examples of these patches. Below we show the result of ten sample patches (each set includes the one example we explored above), run on 1,000 images from the ImageNet training set for the class in question. While they aren’t effective in all cases, they do flip the image classification to the target class in about 2 in 5 images. The success rate reaches about 1 in 2 images if we also allow to position the patch in the best of the four corners of the image (top left, top right, bottom left, bottom right) at the most effective size. To ensure our attack isn’t just blocking out evidence for the original class, we also compare each attack to a random noise image patch.

 #adversarial-analysis {
 display: none;
 grid-column: text;
 margin-bottom: 0;
 }
 #adversarial-analysis table {
 width: 100%;
 text-align: right;
 }
 #adversarial-analysis table th {
 vertical-align: initial;
 }
 #adversarial-analysis table th .figcaption {
 display: block;
 font-weight: initial;
 }
 #adversarial-analysis table th img {
 width: initial;
 }
 #adversarial-analysis table .detail {
 /\* padding-left: 1em; \*/
 font-size: 85%;
 color: rgba(0, 0, 0, 0.6);
 }
 #adversarial-analysis table .synset {
 font-family: monospace;
 white-space: nowrap;
 }
 #adversarial-analysis table .from,
 #adversarial-analysis table .to {
 white-space: nowrap;
 }
 #adversarial-analysis table .overall {
 font-weight: bold;
 }
 @supports (font-variant-numeric: tabular-nums) {
 #adversarial-analysis table {
 font-variant-numeric: tabular-nums;
 }
 }

 Our “attacks” can be seen as part of a larger trend (eg. ) of researchers
 exploring input attacks on models other than the traditional epsilon ball adversarial examples .
 In many ways, our attacks are most similar to adversarial patches , which also add a small patch to the input image.
 From this perspective, adversarial patches are far more effective, working much more reliably. Instead, we see our attacks as interesting because they are synthesized by humans from their understanding of the model,
 and seem to be attacking the model at a higher level of abstraction.

 We also want to emphasize that not all class comparisons reveal these type of patches and not all icons in the visualization have the same (or any) effectiveness and we’ve only tested them on one model. If we wanted to find these patches more systematically, a different approach would most likely be more effective. However, the class activation atlas technique was what revealed the existence of these patches before we knew to look for them. If you’d like to explore your own comparisons and search for your own patches, we’ve provided a notebook to get you started:

Conclusion and Future Work
--------------------------

Activation atlases give us a new way to peer into convolutional vision networks. They give us a global, hierarchical, and human-interpretable overview of concepts within the hidden layers. Not only does this allow us to better see the inner workings of these complicated systems, but it’s possible that it could enable new interfaces for working with images.

### Surfacing Inner Properties of Models

The vast majority of neural network research focuses on quantitative evaluations of network behavior. How accurate is the model? What’s the precision-recall curve?

While these questions can describe how the network behaves in specific situations, it doesn’t give us a great understanding of *why* it behaves the way it does. To truly understand why a network behaves the way it does, we would need to fully understand the rich inner world of the network — it’s hidden layers. For example, understanding better how InceptionV1 builds up a classifier for a fireboat from component parts in mixed4d can help us build confidence in our models and can surface places where it isn’t doing what we want.

Engaging with this inner world also invites us to do deep learning research in a new way. Normally, each neural network experiment gives only a few bits of feedback — whether the loss went up or down — to inform the next round of experiments. We design architectures by almost blind trial and error, guided by vague intuitions that we build up over years. In the future, we hope that researchers will get rich feedback on what each layer in their model is doing in a way that will make our current approach seem like stumbling in the dark.

Activation atlases, as they presently stand, are inadequate to really help researchers iterate on models, in part because they aren’t comparable. If you look at atlases for two slightly different models, it’s hard to take away anything. In future work, we will explore how similar visualizations can compare models, showing similarities and differences beyond error rates.

### New interfaces

Machine learning models are usually deployed as black boxes that automate a specific task, executing it on their own. But there’s a growing sense that there might be an alternate way for us to relate to them: that instead of increasingly automating a task, they could be used more directly by a person. One vision of this augmentation that we find particularly compelling is the idea that the internal representations neural networks learn can be repurposed as tools . Already, we’ve seen exciting demonstrations of this in images

 and music

  .

We think of activation atlases as revealing a machine-learned alphabet for images — a collection of simple, atomic concepts that are combined and recombined to form much more complex visual ideas. In the same way that we use word processors to turn letters into words, and words into sentences, we can imagine a tool that would allow us to create images from a machine-learned language system for images. Similar to GAN painting , imagine using something like an activation atlas as a palette — one could dip a brush into a “tree” activation and paint with it. A palette of concepts rather than colors.

While classification models are not generally thought of as being used to generate images, techniques like deep dream has shown that this is entirely possible. In this particular instance, we imagine constructing a grid of activations by selecting them from an atlas (or some derivation), then optimizing an output image that would correspond to the user’s constructed activation matrix.

Such a tool would not necessarily be limited to targeting realistic images either. Techniques like style transfer  have shown us that we can use these vision networks to create nuanced visual expression outside the explicit distribution of visual data it was trained on. We speculate that activation atlases could be helpful in manipulating artistic styles without having to find an existing reference image, or they could help in guiding and modifying automated style transfer techniques.

We could also use these atlases to query large image datasets. In the same way that we probe large corpuses of text with words, we could, too, use activation atlases to find types of images in large image datasets. Using words to search for something like a “tree” is quite powerful, but as you get more specific, human language is often ill-suited to describe specific visual characteristics. In contrast, the hidden layers of neural networks are a language optimized for the sole purpose of representing visual concepts. Instead of using the proverbial thousand words to uniquely specify the image one is seeking, we can imagine someone using the language of the activation atlas.

And lastly, we can also liken activation atlases to histograms. In the same way that traditional histograms give us good summaries of large datasets, activation atlases can be used to summarize large numbers of images.

In the examples in this article we used the same dataset for training the model as we did for collecting the activations. But, if we use a different dataset to collect the activations, we could use the atlas as a way of inspecting an unknown dataset. An activation atlas could show us a histogram of *learned concepts* that exist within the images. Such a tool could show us the semantics of the data and not just visual similarities, like showing histograms of common pixel values.

While we are excited about the potential of activation atlases, we are even more excited at the possibility of developing similar techniques for other types of models. Imagine having an array of machine learned, but human interpretable, languages for images, audio and text.

Activation Atlas

Shan Carter
Zan Armstrong
Ludwig Schubert
Ian Johnson
Chris Olah

Summary: By using feature inversion to visualize millions of activations from an image classification network, we create an explorable activation atlas of features the network has learned and what concepts it typically represents.

Visualizing higher-layer features of a deep network

Feature Visualization

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Understanding deep image representations by inverting them

The Building Blocks of Interpretability

t-SNE visualization of CNN codes

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Imagenet: A large-scale hierarchical image database

Going deeper with convolutions

Visualizing data using t-SNE

Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization

Adversarial Patch

Intriguing properties of neural networks

Using Artificial Intelligence to Augment Human Intelligence

Visualizing Representations: Deep Learning and Human Beings

Machine Learning for Visualization

ML as Collaborator: Composing Melodic Palettes with Latent Loops

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

A neural algorithm of artistic style

Exploring Histograms