Visualizing the Impact of Feature Attribution Baselines

Pascal Sturmfels
Scott Lundberg
Su-In Lee

Abstract: Path attribution methods are a gradient-based way            of explaining deep models. These methods require choosing a            hyperparameter known as the baseline input.            What does this hyperparameter mean, and how important is it? In this article,            we investigate these questions using image classification networks            as a case study. We discuss several different ways to choose a baseline            input and the assumptions that are implicit in each baseline.            Although we focus here on path attribution methods, our discussion of baselines            is closely connected with the concept of missingness in the feature space -            a concept that is critical to interpretability research.

Path attribution methods are a gradient-based way

 of explaining deep models. These methods require choosing a

 hyperparameter known as the *baseline input*.

 What does this hyperparameter mean, and how important is it? In this article,

 we investigate these questions using image classification networks

 as a case study. We discuss several different ways to choose a baseline

 input and the assumptions that are implicit in each baseline.

 Although we focus here on path attribution methods, our discussion of baselines

 is closely connected with the concept of missingness in the feature space -

 a concept that is critical to interpretability research.

Introduction

------------

 If you are in the business of training neural networks,

 you might have heard of the integrated gradients method, which

 was introduced at

 .

 The method computes which features are important

 to a neural network when making a prediction on a

 particular data point. This helps users

 understand which features their network relies on.

 Since its introduction,

 integrated gradients has been used to interpret

 networks trained on a variety of data types,

 including retinal fundus images

 and electrocardiogram recordings .

 If you've ever used integrated gradients,

 you know that you need to define a baseline input x'x'x' before

 using the method. Although the original paper discusses the need for a baseline

 and even proposes several different baselines for image data - including

 the constant black image and an image of random noise - there is

 little existing research about the impact of this baseline.

 Is integrated gradients sensitive to the

 hyperparameter choice? Why is the constant black image

 a "natural baseline" for image data? Are there any alternative choices?

 In this article, we will delve into how this hyperparameter choice arises,

 and why understanding it is important when you are doing model interpretation.

 As a case-study, we will focus on image classification models in order

 to visualize the effects of the baseline input. We will explore several

 notions of missingness, including both constant baselines and baselines

 defined by distributions. Finally, we will discuss different ways to compare

 baseline choices and talk about why quantitative evaluation

 remains a difficult problem.

Image Classification

--------------------

 We focus on image classification as a task, as it will allow us to visually

 plot integrated gradients attributions, and compare them with our intuition

 , a convolutional

 neural network designed for the ImageNet dataset ,

 On the ImageNet validation set, Inception V4 has a top-1 accuracy of over 80%.

 We download weights from TensorFlow-Slim ,

 and visualize the predictions of the network on four different images from the

 validation set.

 Right: The predicted logits of the network on the original image. The

 network correctly classifies all images with high confidence.

 relative to the true class label.

 Although state of the art models perform well on unseen data,

 users may still be left wondering: *how* did the model figure

 out which object was in the image? There are a myriad of methods to

 interpret machine learning models, including methods to

 visualize and understand how the network represents inputs internally ,

 feature attribution methods that assign an importance score to each feature

 for a specific input ,

 and saliency methods that aim to highlight which regions of an image

 the model was looking at when making a decision

 .

 visualized as a saliency method, and a saliency method can assign importance

 scores to each individual pixel. In this article, we will focus

 on the feature attribution method integrated gradients.

 Formally, given a target input xxx and a network function fff,

 to the iiith feature value representing how much that feature

 indicates that feature strongly increases or decreases the network output

 the feature in question did not influence f(x)f(x)f(x).

 prediction using integrated gradients.

 The pixels in white indicate more important pixels. In order to plot

 attributions, we follow the same design choices as .

 That is, we plot the absolute value of the sum of feature attributions

 high-magnitude attributions dominating the color scheme.

A Better Understanding of Integrated Gradients

----------------------------------------------

 As you look through the attribution maps, you might find some of them

 To better understand this behavior, we need to explore how

 we generated feature attributions. Formally, integrated gradients

 defines the importance value for the iiith feature value as follows:

 \times \underbrace{\int\_{\alpha = 0}^ 1}\_{\text{From baseline to input…}}

 where xxx is the current input,

 "absence" of feature input. The subscript iii is used

 to denote indexing into the iiith feature.

 As the formula above states, integrated gradients gets importance scores

 But why would doing this make sense? Recall that the gradient of

 a function represents the direction of maximum increase. The gradient

 is telling us which pixels have the steepest local slope with respect

 to the output. For this reason, the gradient of a network at the input

 was one of the earliest saliency methods.

 Unfortunately, there are many problems with using gradients to interpret

 deep neural networks .

 One specific issue is that neural networks are prone to a problem

 sample even if the network depends heavily on those features. This can happen

 Intuitively, shifting the pixels in an image by a small amount typically

 doesn't change what the network sees in the image. We can illustrate

 saturation by plotting the network output at all

 images between the baseline x'x'x' and the current image. The figure

 below displays that the network

 output for the correct class increases initially, but then quickly flattens.

 A plot of network outputs at x'+α(x−x')x' + \alpha (x - x')x'+α(x−x').

 Notice that the network output saturates the correct class

 at small values of α\alphaα. By the time α=1\alpha = 1α=1,

 the network output barely changes.

 What we really want to know is how our network got from

 predicting essentially nothing at x'x'x' to being

 completely saturated towards the correct output class at xxx.

 Which pixels, when scaled along this path, most

 increased the network output for the correct class? This is

 exactly what the formula for integrated gradients gives us.

 By integrating over a path,

 integrated gradients avoids problems with local gradients being

 saturated. We can break the original equation

 down and visualize it in three separate parts: the interpolated image between

 the baseline image and the target image, the gradients at the interpolated

 image, and accumulating many such gradients over α\alphaα.

 \int\_{\alpha' = 0}^{\alpha} \underbrace{(x\_i - x'\_i) \times

 {\delta x\_i} d \alpha'}\_{\text{(2): Gradients at Interpolation}}

 approximation of the integral with 500 linearly-spaced points between 0 and 1.

 Integrated gradients, visualized. In the line chart, the red line refers to

 accumulate at small values of α\alphaα.

 We have casually omitted one part of the formula: the fact

 that we multiply by a difference from a baseline. Although

 we won't go into detail here, this term falls out because we

 care about the derivative of the network

 function fff with respect to the path we are integrating over.

 That is, if we integrate over the

 straight-line between x'x'x' and xxx, which

 we can represent as γ(α)=x'+α(x−x')\gamma(\alpha) =

 x' + \alpha(x - x')γ(α)=x'+α(x−x'), then:

 δf(γ(α))δα=δf(γ(α))δγ(α)×δγ(α)δα=δf(x'+α'(x−x'))δxi×(xi−x'i)

 \frac{\delta f(\gamma(\alpha))}{\delta \alpha} =

 \frac{\delta f(\gamma(\alpha))}{\delta \gamma(\alpha)} \times

 \frac{\delta \gamma(\alpha)}{\delta \alpha} =

 \frac{\delta f(x' + \alpha' (x - x'))}{\delta x\_i} \times (x\_i - x'\_i)

 δαδf(γ(α))=δγ(α)δf(γ(α))×δαδγ(α)=δxiδf(x'+α'(x−x'))×(xi−x'i)

 The difference from baseline term is the derivative of the

 path function γ\gammaγ with respect to α\alphaα.

  The theory behind integrated gradients is discussed

 in more detail in the original paper. In particular, the authors

 show that integrated gradients satisfies several desirable

 properties, including the completeness axiom:

 Axiom 1: Completeness∑iϕiIG(f,x,x')=f(x)−f(x')

 \textrm{Axiom 1: Completeness}\\

 \sum\_i \phi\_i^{IG}(f, x, x') = f(x) - f(x')

 Axiom 1: Completenessi∑ϕiIG(f,x,x')=f(x)−f(x')

 Note that this theorem holds for any baseline x'x'x'.

 Completeness is a desirable property because it states that the

 importance scores for each feature break down the output of the network:

 each importance score represents that feature's individual contribution to

 Although it's not essential to our discussion here, we can prove

 that integrated gradients satisfies this axiom using the

 [fundamental

 full discussion of all of the properties that integrated

 gradients satisfies to the original paper, since they hold

 independent of the choice of baseline.  The completeness

 axiom also provides a way to measure convergence.

 In practice, we can't compute the exact value of the integral. Instead,

 we use a discrete sum approximation with kkk linearly-spaced points between

 0 and 1 for some value of kkk. If we only chose 1 point to

 approximate the integral, that feels like too few. Is 10 enough? 100?

 Intuitively 1,000 may seem like enough, but can we be certain?

 As proposed in the original paper, we can use the completeness axiom

 as a sanity check on convergence: run integrated gradients with kkk

 and if the difference is large, re-run with a larger kkk

 Of course, this brings up a new question: what is "large" in this context?

 One heuristic is to compare the difference with the magnitude of the

 output itself.

 .

 The line chart above plots the following equation in red:

 ∑iϕiIG(f,x,x';α)⏟(4): Sum of Cumulative Gradients up to α

 (4): Sum of Cumulative Gradients up to αi∑ϕiIG(f,x,x';α)

 That is, it sums all of the pixel attributions in the saliency map.

 We can see that with 500 samples, we seem (at least intuitively) to

 have converged. But this article isn't about how

 to get good convergence - it's about baselines! In order

 to advance our understanding of the baseline, we will need a brief excursion

 into the world of game theory.

Game Theory and Missingness

---------------------------

 Integrated gradients is inspired by work

 from cooperative game theory, specifically the Aumann-Shapley value

 . In cooperative game theory,

 a non-atomic game is a construction used to model large-scale economic systems

 Aumann-Shapley values provide a theoretically grounded way to

 determine how much different groups of participants contribute to the system.

 In game theory, a notion of missingness is well-defined. Games are defined

 on coalitions - sets of participants - and for any specific coalition,

 a participant of the system can be in or out of that coalition. The fact

 that games can be evaluated on coalitions is the foundation of

 the Aumann-Shapley value. Intuitively, it computes how

 much value a group of participants adds to the game

 by computing how much the value of the game would increase

 if we added more of that group to any given coalition.

 Unfortunately, missingness is a more difficult notion when

 we are speaking about machine learning models. In order

 to evaluate how important the iiith feature is, we

 want to be able to compute how much the output of

 the network would increase if we successively increased

 the "presence" of the iiith feature. But what does this mean, exactly?

 In order to increase the presence of a feature, we would need to start

 with the feature being "missing" and have a way of interpolating

 between that missingness and its current, known value.

 Hopefully, this is sounding awfully familiar. Integrated gradients

 has a baseline input x'x'x' for exactly this reason: to model a

 feature being absent. But how should you choose

 x'x'x' in order to best represent this? It seems to be common practice

 to choose a baseline input x'x'x' to be the vector of

 all zeros. But consider the following scenario: you've learned a model

 on a healthcare dataset, and one of the features is blood sugar level.

 The model has correctly learned that excessively low levels of blood sugar,

 which correspond to hypoglycemia, is dangerous. Does

 a blood sugar level of 000 seem like a good choice to represent missingness?

 The point here is that fixed feature values may have unintended meaning.

 The problem compounds further when you consider the difference from

 baseline term xi−x'ix\_i - x'\_ixi−x'i.

 To understand why our machine learning model thinks this patient

 is at high risk, you run integrated gradients on this data point with a

 because xi−x'i=0x\_i - x'\_i = 0xi−x'i=0. This is despite the fact that

 a blood sugar level of 000 would be fatal!

 We find similar problems when we move to the image domain.

 If you use a constant black image as a baseline, integrated gradients will

 not highlight black pixels as important even if black pixels make up

 authors in , and is in fact

 central to the definition of a baseline: we wouldn't want integrated gradients

 to highlight missing features as important! But then how do we avoid

 giving zero importance to the baseline color?

 Mouse over the segmented image to choose a different color

 as a baseline input x'x'x'. Notice that pixels

 of the baseline color are not highlighted as important,

 even if they make up part of the main object in the image.

Alternative Baseline Choices

----------------------------

 It's clear that any constant color baseline will have this problem.

 Are there any alternatives? In this section, we

 compare four alternative choices for a baseline in the image domain.

 Before proceeding, it's important to note that this article isn't

 the first article to point out the difficulty of choosing a baselines.

 Several articles, including the original paper, discuss and compare

 several notions of "missingness", both in the

 context of integrated gradients and more generally

 .

 Nonetheless, choosing the right baseline remains a challenge. Here we will

 present several choices for baselines: some based on existing literature,

 others inspired by the problems discussed above. The figure at the end

 of the section visualizes the four baselines presented here.

### The Maximum Distance Baseline

 If we are worried about constant baselines that are blind to the baseline

 color, can we explicitly construct a baseline that doesn't suffer from this

 problem? One obvious way to construct such a baseline is to take the

 farthest image in L1 distance from the current image such that the

 baseline is still in the valid pixel range. This baseline, which

 we will refer to as the maximum distance baseline (denoted

 *max dist.* in the figure below),

 avoids the difference from baseline issue directly.

### The Blurred Baseline

 The issue with the maximum distance baseline is that it doesn't

 really represent *missingness*. It actually contains a lot of

 information about the original image, which means we are no longer

 explaining our prediction relative to a lack of information. To better

 preserve the notion of missingness, we take inspiration from

 . In their paper,

 Fong and Vedaldi use a blurred version of the image as a

 domain-specific way to represent missing information. This baseline

 is attractive because it captures the notion of missingness in images

 in a very human intuitive way. In the figure below, this baseline is

 denoted *blur*. The figure lets you play with the smoothing constant

 used to define the baseline.

### The Uniform Baseline

 One potential drawback with the blurred baseline is that it is biased

 to highlight high-frequency information. Pixels that are very similar

 to their neighbors may get less importance than pixels that are very

 different than their neighbors, because the baseline is defined as a weighted

 from both  and the original integrated

 gradients paper. Another way to define missingness is to simply sample a random

 uniform image in the valid pixel range and call that the baseline.

 We refer to this baseline as the *uniform* baseline in the figure below.

### The Gaussian Baseline

 Of course, the uniform distribution is not the only distribution we can

 touch on in the next section), Smilkov et al.

 variance σ\sigmaσ. We can use the same distribution as a baseline for

 range, which means that as σ\sigmaσ approaches ∞\infty∞, the gaussian

 baseline approaches the uniform baseline.

 Comparing alternative baseline choices. For the blur and gaussian

 baselines, you can vary the parameter σ\sigmaσ, which refers

 to the width of the smoothing kernel and the standard deviation of

 noise respectively.

Averaging Over Multiple Baselines

---------------------------------

 You may have nagging doubts about those last two baselines, and you

 would be right to have them. A randomly generated baseline

 can suffer from the same blindness problem that a constant image can. If

 we draw a uniform random image as a baseline, there is a small chance

 that a baseline pixel will be very close to its corresponding input pixel

 in value. Those pixels will not be highlighted as important. The resulting

 saliency map may have artifacts due to the randomly drawn baseline. Is there

 any way we can fix this problem?

 Perhaps the most natural way to do so is to average over multiple

 different baselines, as discussed in

 .

 Although doing so may not be particularly natural for constant color images

 (which colors do you choose to average over and why?), it is a

 very natural notion for baselines drawn from distributions. Simply

 draw more samples from the same distribution and average the

 importance scores from each sample.

### Assuming a Distribution

 At this point, it's worth connecting the idea of averaging over multiple

 baselines back to the original definition of integrated gradients. When

 we average over multiple baselines from the same distribution DDD,

 we are attempting to use the distribution itself as our baseline.

 We use the distribution to define the notion of missingness:

 if we don't know a pixel value, we don't assume its value to be 0 - instead

 we assume that it has some underlying distribution DDD. Formally, given

 a baseline distribution DDD, we integrate over all possible baselines

 x'∈Dx' \in Dx'∈D weighted by the density function pDp\_DpD:

 )}^{\text{integrated gradients

 with baseline } x'

 } \times \underbrace{p\_D(x') dx'}\_{\text{…and weight by the density}} \bigg)

 In terms of missingness, assuming a distribution might intuitively feel

 like a more reasonable assumption to make than assuming a constant value.

 But this doesn't quite solve the issue: instead of having to choose a baseline

 x'x'x', now we have to choose a baseline distribution DDD. Have we simply

 postponed the problem? We will discuss one theoretically motivated

 way to choose DDD in an upcoming section, but before we do, we'll take

 a brief aside to talk about how we compute the formula above in practice,

 and a connection to an existing method that arises as a result.

### Expectations, and Connections to SmoothGrad

 Now that we've introduced a second integral into our formula,

 we need to do a second discrete sum to approximate it, which

 requires an additional hyperparameter: the number of baselines to sample.

 In , Erion et al. make the

 observation that both integrals can be thought of as expectations:

 the first integral as an expectation over DDD, and the second integral

 as an expectation over the path between x'x'x' and xxx. This formulation,

 called *expected gradients*, is defined formally as:

 {\text{Expectation over \(D\) and the path…}}

 \bigg[ \overbrace{(x\_i - x'\_i) \times

 \frac{\delta f(x' + \alpha (x - x'))}{\delta x\_i}}^{\text{…of the

 importance of the } i\text{th pixel}} \bigg]

 Expected gradients and integrated gradients belong to a family of methods

 known as "path attribution methods" because they integrate gradients

 over one or more paths between two valid inputs.

 Both expected gradients and integrated gradients use straight-line paths,

 in more detail in the original paper.  To compute expected gradients in

 practice, we use the following formula:

 ϕ^iEG(f,x;D)=1k∑j=1k(xi−x'ij)×δf(x'j+αj(x−x'j))δxi

 \frac{\delta f(x'^j + \alpha^{j} (x - x'^j))}{\delta x\_i}

 ϕ^iEG(f,x;D)=k1j=1∑k(xi−x'ij)×δxiδf(x'j+αj(x−x'j))

 where x'jx'^jx'j is the jjjth sample from DDD and

 αj\alpha^jαj is the jjjth sample from the uniform distribution between

 0 and 1. Now suppose that we use the gaussian baseline with variance

 ϕ^iEG(f,x;N(x,σ2I))=1k∑j=1kϵσj×δf(x+(1−αj)ϵσj)δxi

 \hat{\phi}\_i^{EG}(f, x; N(x, \sigma^2 I))

 = \frac{1}{k} \sum\_{j=1}^k

 \epsilon\_{\sigma}^{j} \times

 \frac{\delta f(x + (1 - \alpha^j)\epsilon\_{\sigma}^{j})}{\delta x\_i}

 ϕ^iEG(f,x;N(x,σ2I))=k1j=1∑kϵσj×δxiδf(x+(1−αj)ϵσj)

 To see how we arrived

 at the above formula, first observe that

 x'∼N(x,σ2I)=x+ϵσx'−x=ϵσ

 \begin{aligned}

 x' \sim N(x, \sigma^2 I) &= x + \epsilon\_{\sigma} \\

 x'- x &= \epsilon\_{\sigma} \\

 \end{aligned}

 x'∼N(x,σ2I)x'−x=x+ϵσ=ϵσ

 by definition of the gaussian baseline. Now we have:

 x'+α(x−x')=x+ϵσ+α(x−(x+ϵσ))=x+(1−α)ϵσ

 \begin{aligned}

 x' + \alpha(x - x') &= \\

 x + \epsilon\_{\sigma} + \alpha(x - (x + \epsilon\_{\sigma})) &= \\

 x + (1 - \alpha)\epsilon\_{\sigma}

 \end{aligned}

 x'+α(x−x')x+ϵσ+α(x−(x+ϵσ))x+(1−α)ϵσ==

 The above formula simply substitues the last line

 of each equation block back into the formula.

 .

 This looks awfully familiar to an existing method called SmoothGrad

 . If we use the (gradients ×\times× input image)

 variant of SmoothGrad  SmoothGrad is

 was a method designed to sharpen saliency maps and was meant to be run

 on top of an existing saliency method. The idea is simple:

 instead of running a saliency method once on an image, first

 add some gaussian noise to an image, then run the saliency method.

 Do this several times with different draws of gaussian noise, then

 is discussed in more detail in the original SmoothGrad paper.,

 then we have the following formula:

 ϕiSG(f,x;N(0ˉ,σ2I))=1k∑j=1k(x+ϵσj)×δf(x+ϵσj)δxi

 \phi\_i^{SG}(f, x; N(\bar{0}, \sigma^2 I))

 = \frac{1}{k} \sum\_{j=1}^k

 (x + \epsilon\_{\sigma}^{j}) \times

 \frac{\delta f(x + \epsilon\_{\sigma}^{j})}{\delta x\_i}

 ϕiSG(f,x;N(0ˉ,σ2I))=k1j=1∑k(x+ϵσj)×δxiδf(x+ϵσj)

 We can see that SmoothGrad and expected gradients with a

 gaussian baseline are quite similar, with two key differences:

 gradients multiplies by just ϵσ\epsilon\_{\sigma}ϵσ, and while expected

 gradients samples uniformly along the path, SmoothGrad always

 samples the endpoint α=0\alpha = 0α=0.

 Can this connection help us understand why SmoothGrad creates

 assuming that each of our pixel values is drawn from a

 gaussian *independently* of the other pixel values. But we know

 this is far from true: in images, there is a rich correlation structure

 between nearby pixels. Once your network knows the value of a pixel,

 it doesn't really need to use its immediate neighbors because

 it's likely that those immediate neighbors have very similar intensities.

 Assuming each pixel is drawn from an independent gaussian

 breaks this correlation structure. It means that expected gradients

 tabulates the importance of each pixel independently of

 the other pixel values. The generated saliency maps

 will be less noisy and better highlight the object of interest

 because we are no longer allowing the network to rely

 on only pixel in a group of correlated pixels. This may be

 why SmoothGrad is smooth: because it is implicitly assuming

 independence among pixels. In the figure below, you can compare

 integrated gradients with a single randomly drawn baseline

 to expected gradients sampled over a distribution. For

 the gaussian baseline, you can also toggle the SmoothGrad

 option to use the SmoothGrad formula above. For all figures,

 k=500k=500k=500.

 The difference between a single baseline and multiple

 baselines from the same distribution. Use the

 "Multi-Reference" button to toggle between the two. For the gaussian

 baseline, you can also toggle the "Smooth Grad" button

 to toggle between expected gradients and SmoothGrad

 with gradients \* inputs.

### Using the Training Distribution

 Is it really reasonable to assume independence among

 pixels while generating saliency maps? In supervised learning,

 we make the assumption that the data is drawn

 share a common, underlying distribution is what allows us to

 do supervised learning and make claims about generalizability. Given

 this assumption, we don't need to

 model missingness using a gaussian or a uniform distribution:

 we can use DdataD\_{\text{data}}Ddata to model missingness directly.

 The only problem is that we do not have access to the underlying distribution.

 But because this is a supervised learning task, we do have access to many

 independent draws from the underlying distribution: the training data!

 We can simply use samples from the training data as random draws

 from DdataD\_{\text{data}}Ddata. This brings us to the variant

 of expected gradients used in ,

 which we again visualize in three parts:

 \frac{1}{k} \sum\_{j=1}^k

 \underbrace{(x\_i - x'^j\_i) \times

 \frac{\delta f(\text{ }

 \overbrace{x'^j + \alpha^{j} (x - x'^j)}^{\text{(1): Interpolated Image}}

 \text{ })}{\delta x\_i}}\_{\text{ (2): Gradients at Interpolation}}

 = \overbrace{\hat{\phi\_i}^{EG}(f, x, k; D\_{\text{data}})}

 ^{\text{(3): Cumulative Gradients up to }\alpha}

 A visual representation of expected gradients. Instead of taking contributions

 from a single path, expected gradients averages contributions from

 all paths defined by the underlying data distribution. Note that

 this figure only displays every 10th sample to avoid loading many images.

 In (4) we again plot the sum of the importance scores over pixels. As mentioned

 gradients, satisfy the completeness axiom. We can definitely see that

 completeness is harder to satisfy when we integrate over both a path

 and a distribution: that is, with the same number

 of samples, expected gradients doesn't converge as quickly as

 integrated gradients does. Whether or not this is an acceptable price to

 pay to avoid color-blindness in attributions seems subjective.

Comparing Saliency Methods

--------------------------

 So we now have many different choices for a baseline. How do we choose

 which one we should use? The different choices of distributions and constant

 baselines have different theoretical motivations and practical concerns.

 Do we have any way of comparing the different baselines? In this section,

 we will touch on several different ideas about how to compare

 of all of the existing evaluation metrics, but is instead meant to

 emphasize that evaluating interpretability methods remains a difficult problem.

### The Dangers of Qualitative Assessment

 One naive way to evaluate our baselines is to look at the saliency maps

 they produce and see which ones best highlight the object in the image.

 reasonable results, as does using a gaussian baseline or the blurred baseline.

 But is visual inspection really a good way judge our baselines? For one thing,

 we've only presented four images from the test set here. We would need to

 conduct user studies on a much larger scale with more images from the test

 set to be confident in our results. But even with large-scale user studies,

 qualitative assessment of saliency maps has other drawbacks.

 When we rely on qualitative assessment, we are assuming that humans

 know what an "accurate" saliency map is. When we look at saliency maps

 on data like ImageNet, we often check whether or not the saliency map

 highlights the object that we see as representing the true class in the image.

 We make an assumption between the data and the label, and then further assume

 that a good saliency map should reflect that assumption. But doing so

 has no real justification. Consider the figure below, which compares

 two saliency methods on a network that gets above 99% accuracy

 on (an altered version of) MNIST.

 The first saliency method is just an edge detector plus gaussian smoothing,

 while the second saliency method is expected gradients using the training

 data as a distribution. Edge detection better reflects what we humans

 think is the relationship between the image and the label.

Original Image:Edge Detection:Expected Gradients:

 Qualitative assessment can be dangerous because we rely

 on our human knowledge of the relationship between

 the data and the labels, and then we assume

 that an accurate model has learned that very relationship.

 Unfortunately, the edge detection method here does not highlight

 what the network has learned. This dataset is a variant of

 decoy MNIST, in which the top left corner of the image has

 been altered to directly encode the image's class

 . That is, the intensity

 of the top left corner of each image has been altered to

 be 255×y9255 \times \frac{y}{9} 255×9y where yyy is the class

 the image belongs to. We can verify by removing this

 patch in the test set that the network heavily relies on it to make

 predictions, which is what the expected gradients saliency maps show.

 This is obviously a contrived example. Nonetheless, the fact that

 visual assessment is not necessarily a useful way to evaluate

 saliency maps and attribution methods has been extensively

 discussed in recent literature, with many proposed qualitative

 tests as replacements

 .

 At the heart of the issue is that we don't have ground truth explanations:

 we are trying to evaluate which methods best explain our network without

 actually knowing what our networks are doing.

### Top K Ablation Tests

 One simple way to evaluate the importance scores that

 expected/integrated gradients produces is to see whether

 ablating the top k features as ranked by their importance

 decreases the predicted output logit. In the figure below, we

 ablate either by mean-imputation or by replacing each pixel

 for 1000 different correctly classified test-set images using each

 of the baselines proposed above

 For the blur baseline and the blur

 ablation test, we use σ=20\sigma = 20σ=20.

 For the gaussian baseline, we use σ=1\sigma = 1σ=1. These choices

 are somewhat arbitrary - a more comprehensive evaluation

 would compare across many values of σ\sigmaσ.

 . As a

 control, we also include ranking features randomly

 (*Random Noise* in the plot).

 We plot, as a fraction of the original logit, the output logit

 of the network at the true class. That is, suppose the original

 image is a goldfinch and the network predicts the goldfinch class correctly

 with 95% confidence. If the confidence of class goldfinch drops

 to 60% after ablating the top 10% of pixels as ranked by

 feature importance, then we plot a curve that goes through

 that best highlights which pixels the network

 should exhibit the fastest drop in logit magnitude, because

 it highlights the pixels that most increase the confidence of the network.

 That is, the lower the curve, the better the baseline.

### Mass Center Ablation Tests

 One problem with ablating the top k features in an image

 is related to an issue we already brought up: feature correlation.

 No matter how we ablate a pixel, that pixel's neighbors

 provide a lot of information about the pixel's original value.

 With this in mind, one could argue that progressively ablating

 pixels one by one is a rather meaningless thing to do. Can

 we instead perform ablations with feature correlation in mind?

 One straightforward way to do this is simply compute the

 center of mass

 of the saliency map, and ablate a boxed region centered on

 the center of mass. This tests whether or not the saliency map

 is generally highlighting an important region in the image. We plot

 replacing the boxed region around the saliency map using mean-imputation

 and blurring below as well (*Mean Center* and *Blur Center*, respectively).

 As a control, we compare against a saliency map generated from random gaussian

 noise (*Random Noise* in the plot).

 A variety of ablation tests on a variety of baselines.

 Using the training distribution and using the uniform distribution

 outperform most other methods on the top k ablation tests. The

 blur baseline inspired by

 does equally well on the blur top-k test. All methods

 perform similarly on the mass center ablation tests. Mouse

 over the legend to highlight a single curve.

 The ablation tests seem to indicate some interesting trends.

 All methods do similarly on the mass center ablation tests, and

 only slightly better than random noise. This may be because the

 object of interest generally lies in the center of the image - it

 isn't hard for random noise to be centered at the image. In contrast,

 using the training data or a uniform distribution seems to do quite well

 on the top-k ablation tests. Interestingly, the blur baseline

 inspired by  also

 does quite well on the top k baseline tests, especially when

 we ablate pixels by blurring them! Would the uniform

 baseline do better if you ablate the image with uniform random noise?

 by progressively replacing it with a different image. We leave

 these experiments as future work, as there is a more pressing question

 we need to discuss.

### The Pitfalls of Ablation Tests

 Constant baselines tend to not need as many samples

 comparing not only across baselines but also across number of samples drawn,

 and for the blur and gaussian baselines, the parameter σ\sigmaσ.

 As mentioned above, we have defined many notions of missingness other than

 mean-imputation or blurring: more extensive comparisons would also compare

 all of our baselines across all of the corresponding notions of missing data.

 But even with all of these added comparisons, do ablation

 tests really provide a well-founded metric to judge attribution methods?

 The authors of  argue

 against ablation tests. They point out that once we artificially ablate

 pixels an image, we have created inputs that do not come from

 the original data distribution. Our trained model has never seen such

 inputs. Why should we expect to extract any reasonable information

 from evaluating our model on them?

 On the other hand, integrated gradients and expected gradients

 rely on presenting interpolated images to your model, and unless

 you make some strange convexity assumption, those interpolated images

 don't belong to the original training distribution either.

 In general, whether or not users should present

 is a subject of ongoing debate

 . Nonetheless,

 the point raised in  is still an

 important one: "it is unclear whether the degradation in model

 performance comes from the distribution shift or because the

 features that were removed are truly informative."

### Alternative Evaluation Metrics

 So what about other evaluation metrics proposed in recent literature? In

 , Hooker et al. propose a variant of

 an ablation test where we first ablate pixels in the training and

 test sets. Then, we re-train a model on the ablated data and measure

 by how much the test-set performance degrades. This approach has the advantage

 of better capturing whether or not the saliency method

 highlights the pixels that are most important for predicting the output class.

 Unfortunately, it has the drawback of needing to re-train the model several

 times. This metric may also get confused by feature correlation.

 Consider the following scenario: our dataset has two features

 that are highly correlated. We train a model which learns to only

 use the first feature, and completely ignore the second feature.

 A feature attribution method might accurately reveal what the model is doing:

 re-train the model and get similar performance because similar information

 is stored in the second feature. We might conclude that our feature

 attribution method is lousy - is it? This problem fits into a larger discussion

 about whether or not your attribution method

 should be "true to the model" or "true to the data"

 which has been discussed in several recent articles

 .

 In , the authors propose several

 sanity checks that saliency methods should pass. One is the "Model Parameter

 Randomization Test". Essentially, it states that a feature attribution

 method should produce different attributions when evaluated on a trained

 model (assumedly a trained model that performs well) and a randomly initialized

 model. This metric is intuitive: if a feature attribution method produces

 similar attributions for random and trained models, is the feature

 attribution really using information from the model? It might just

 be relying entirely on information from the input image.

 But consider the following figure, which is another (modified) version

 of MNIST. We've generated expected gradients attributions using the training

 distribution as a baseline for two different networks. One of the networks

 is a trained model that gets over 99% accuracy on the test set. The other

 Should we now conclude that expected gradients is an unreliable method?

Original Image:Network 1 Saliency:Network 2 Saliency:

 A comparison of two network's saliency maps using expected gradients. One

 network has randomly initialized weights, the other gets >99% accuracy

 on the test set.

 you would run these kinds of saliency method sanity checks on un-modified data.

 But the truth is, even for natural images, we don't actually

 know what an accurate model's saliency maps should look like.

 Different architectures trained on ImageNet can all get good performance

 and have very different saliency maps. Can we really say that

 trained models should have saliency maps that don't look like

 saliency maps generated on randomly initialized models? That isn't

 to say that the model randomization test doesn't have merit: it

 does reveal interesting things about what saliency methods are are doing.

 It just doesn't tell the whole story.

 .

 Each proposed metric comes with their various pros and cons.

 we don't know what our model is doing and have no ground truth to compare

 against.

Conclusion

----------

 So what should be done? We have many baselines and

 no conclusion about which one is the "best." Although

 we don't provide extensive quantitative results

 comparing each baseline, we do provide a foundation

 for understanding them further. At the heart of

 each baseline is an assumption about missingness

 in our model and the distribution of our data. In this article,

 we shed light on some of those assumptions, and their impact

 on the corresponding path attribution. We lay

 groundwork for future discussion about baselines in the

 context of path attributions, and more generally about

 the relationship between representations of missingness

 and how we explain machine learning models.

 A side-by-side comparison of integrated gradients

 using a black baseline

 and expected gradients using the training data

 as a baseline.

Related Methods

---------------

 This work focuses on a specific interpretability method: integrated gradients

 and its extension, expected gradients. We refer to these

 methods as path attribution methods because they integrate

 importances over a path. However, path attribution methods

 represent only a tiny fraction of existing interpretability methods. We focus

 on them here both because they are amenable to interesting visualizations,

 and because they provide a springboard for talking about missingness.

 We briefly cited several other methods at the beginning of this article.

 Many of those methods use some notion of baseline and have contributed to

 the discussion surrounding baseline choices.

 In , Fong and Vedaldi propose

 a model-agnostic method to explain neural networks that is based

 on learning the minimal deletion to an image that changes the model

 prediction. In section 4, their work contains an extended discussion on

 how to represent deletions: that is, how to represent missing pixels. They

 argue that one natural way to delete pixels in an image is to blur them.

 This discussion inspired the blurred baseline that we presented in our article.

 They also discuss how noise can be used to represent missingness, which

 was part of the inspiration for our uniform and gaussian noise baselines.

 In , Shrikumar et al.

 propose a feature attribution method called deepLIFT. It assigns

 importance scores to features by propagating scores from the output

 of the model back to the input. Similar to integrated gradients,

 deepLIFT also defines importance scores relative to a baseline, which

 they call the "reference". Their paper has an extended discussion on

 why explaining relative to a baseline is meaningful. They also discuss

 a few different baselines, including "using a blurred version of the original

 image".

 The list of other related methods that we didn't discuss

 in this article goes on: SHAP and DeepSHAP

 ,

 layer-wise relevance propagation ,

 LIME ,

 RISE  and

 Grad-CAM

 among others. Many methods for explaining machine learning models

 define some notion of baseline or missingness,

 because missingness and explanations are closely related. When we explain

 a model, we often want to know which features, when missing, would most

 change model output. But in order to do so, we need to define

 what missing means because most machine learning models cannot

 handle arbitrary patterns of missing inputs. This article

 does not discuss all of the nuances presented alongside

 each existing method, but it is important to note that these methods were

 points of inspiration for a larger discussion about missingness.

Axiomatic attribution for deep networks

Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy

Ensembling convolutional and long short-term memory networks for electrocardiogram arrhythmia detection

Inception-v4, inception-resnet and the impact of residual connections on learning

Imagenet: A large-scale hierarchical image database

Tensorflow-slim image classification model library

The Building Blocks of Interpretability

Feature Visualization

Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

Visualizing and understanding convolutional networks

Visualizing higher-layer features of a deep network

Understanding deep image representations by inverting them

Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks

"Why Should I Trust You?": Explaining the Predictions of Any Classifier

A unified approach to interpreting model predictions

Layer-wise relevance propagation for neural networks with local renormalization layers

Learning important features through propagating activation differences

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Interpretable explanations of black boxes by meaningful perturbation

Learning deep features for discriminative localization

Grad-cam: Visual explanations from deep networks via gradient-based localization

Smoothgrad: removing noise by adding noise

Rise: Randomized input sampling for explanation of black-box models

Understanding the difficulty of training deep feedforward neural networks

Gradients of counterfactuals

Values of non-atomic games

A note about: Local explanation methods for deep neural networks lack sensitivity to parameter values

The (Un)reliability of saliency methods

Towards better understanding of gradient-based attribution methods for Deep Neural Networks

Learning Explainable Models Using Attribution Priors

XRAI: Better Attributions Through Regions

Right for the right reasons: Training differentiable models by constraining their explanations

A Benchmark for Interpretability Methods in Deep Neural Networks

On the (In)fidelity and Sensitivity for Explanations

Sanity Checks for Saliency Maps

Benchmarking Attribution Methods with Relative Feature Importance

Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation

Interpretation of neural networks is fragile

The many Shapley values for model explanation

Feature relevance quantification in explainable AI: A causality problem

Explaining Models by Propagating Shapley Values of Local Components


Feature Visualization

Chris Olah
Alexander Mordvintsev
Ludwig Schubert

Abstract: There is a growing sense that neural networks need to be interpretable to humans.   The field of neural network interpretability has formed in response to these concerns.   As it matures, two major threads of research have begun to coalesce: feature visualization and attribution.

![](Feature%20Visualization_files/neuron.png)

![](Feature%20Visualization_files/channel.png)

![](Feature%20Visualization_files/attribution-1.png)

![](Feature%20Visualization_files/attribution-2.jpg)

**Attribution**

 As a young field, neural network interpretability does not yet

have standardized terminology.

 Attribution has gone under many different names in the

literature — including "feature visualization"! — but recent work seems

to prefer terms like "attribution" and "saliency maps".

 This article focuses on feature visualization.

 While feature visualization is a powerful tool, actually getting it to

 work involves a number of details.

 In this article, we examine the major issues and explore common

approaches to solving them.

 We find that remarkably simple methods can produce high-quality

visualizations. Along the way we introduce a few tricks for exploring

variation in what neurons react to, how they interact, and how to

improve the optimization process.

---

Feature Visualization by Optimization

-------------------------------------

 Neural networks are, generally speaking, differentiable with respect

to their inputs.

 If we want to find out what kind of input would cause a certain

behavior — whether that's an internal neuron firing or the final output

behavior — we can use derivatives to iteratively tweak the input

 towards that goal .

Step 0

→

Step 4

→

Step 48

→

Step 2048

 While conceptually simple, there are subtle challenges in getting the

optimization to work. We will explore them, as well as common approaches

### Optimization Objectives

 What do we want examples of?

 This is the core question in working with examples, regardless of

whether we're searching through a dataset to find the examples, or

optimizing images to create them from scratch.

 We have a wide variety of options in what we search for:

 #optimization-objectives .objectives {

 grid-column: text;

 grid-template-columns: repeat(1, 1fr);

 }

 #optimization-objectives .objective {

 display: grid;

 grid-template-columns: repeat(2, 1fr) 1.1fr;

 }

 #optimization-objectives .objectives figcaption {

 padding: 4px 8px;

 word-wrap: break-word;

 word-break: break-word;

 }

 #optimization-objectives .objective .objective-icon {

 padding: 8px;

 }

 @media (min-width: 512px) {

 #optimization-objectives .objectives {

 grid-template-columns: repeat(5, 1fr);

 grid-column-gap: 16px;

 }

 #optimization-objectives .objective {

 display: flex;

 flex-flow: column;

 }

 #optimization-objectives .objectives figcaption {

 padding: 0;

 padding-top: 4px;

 word-wrap: break-word;

 word-break: break-word;

 }

 }

`**n**` layer index

`**x,y**` spatial position

`**z**` channel index

`**k**` class index

![](Feature%20Visualization_files/neuron.png)

**Neuron**

`layern[x,y,z]`

![](Feature%20Visualization_files/channel.png)

**Channel**

`layern[:,:,z]`

![](Feature%20Visualization_files/layer.png)

**Layer**/DeepDream

`layern[:,:,:]2`

softmax

![](Feature%20Visualization_files/logits.png)

**Class Logits**

`pre_softmax[k]`

softmax

![](Feature%20Visualization_files/logits_post.png)

**Class Probability**

`softmax[k]`

 We used the channel objective to create most of the images in this article.

 after the softmax.

 One can see the logits as the evidence for each class, and the

probabilities as the likelihood of each class given the evidence.

 Unfortunately, the easiest way to increase the probability softmax

gives to a class is often to make the alternatives unlikely rather than

to make the class of interest likely .

 While the standard explanation is that maximizing probability

doesn't work very well because you can just push down evidence for other

 classes, an alternate hypothesis is that it's just harder to optimize

through the softmax function. We understand this has sometimes been an

issue in adversarial examples, and the solution is to optimize the

LogSumExp of the logits instead. This is equivalent to optimizing

softmax but generally more tractable. Our experience was that the

LogSumExp trick doesn't seem better than dealing with the raw

probabilities.

 Regardless of why that happens, it can be fixed by very strong

regularization with generative models. In this case the probabilities

can be a very principled thing to optimize.

 and objectives used in optimization-based model inversion ,

 which help us understand what information a model keeps and what it

throws away.

 We are only at the beginning of understanding which objectives are

interesting, and there is a lot of room for more work in this area.

### Why visualize by optimization?

**Dataset Examples** show us what neurons respond to in practice

**Optimization** isolates the causes of behavior from mere correlations.

A neuron may not be detecting what you initially thought.

Baseball—or stripes?

*mixed4a, Unit 6*

Animal faces—or snouts?

*mixed4a, Unit 240*

Clouds—or fluffiness?

*mixed4a, Unit 453*

Buildings—or sky?

*mixed4a, Unit 492*

 Optimization also has the advantage of flexibility.

 For example, if we want to study how neurons jointly represent

information,

 we can easily ask how a particular example would need to be different

for an additional neuron to activate.

 This flexibility can also be helpful in visualizing how features

evolve as the network trains.

 If we were limited to understanding the model on the fixed examples in

 our dataset, topics like these ones would be much harder to explore.

 On the other hand, there are also significant challenges to

visualizing features with optimization.

 In the following sections we'll examine techniques to get diverse

visualizations, understand how neurons interact, and avoid high

frequency artifacts.

---

Diversity

---------

 Do our examples show us the full picture?

 Dataset examples have a big advantage here.

 By looking through our dataset, we can find diverse examples.

 It doesn't just give us ones activating a neuron intensely:

**Negative** optimized

**Minimum** activation examples

Slightly negative activation examples

**Positive** optimized

Slightly positive activation examples

**Maximum** activation examples

**Layer mixed 4a, unit 492**

[Reproduce in a

 In contrast, optimization generally gives us just one extremely

positive example — and if we're creative, a very negative example as

well.

 Is there some way that optimization could also give us this diversity?

### Achieving Diversity with Optimization

 A given feature of a network may respond to a wide range of inputs.

 On the class level, for example, a classifier that has been trained to

 recognize dogs should recognize both closeups of their faces as well as

 wider profile images — even though those have quite different visual

appearances.

 Early work by Wei *et al.*

attempts to demonstrate this "intra-class" diversity by recording

activations over the entire training set, clustering them and optimizing

 for the cluster centroids, revealing the different facets of a class

that were learned.

 A different approach by Nguyen, Yosinski, and collaborators was to

search through the dataset for diverse examples and use those as

starting points for the optimization process .

 The idea is that this initiates optimization in different facets of

the feature so that the resulting example from optimization will

demonstrate that facet.

 In more recent work, they combine visualizing classes with a

generative model, which they can sample for diverse examples .

 Their first approach had limited success, and while the generative

model approach works very well — we'll discuss it more in the section on

 For this article we use an approach based on ideas from artistic

style transfer. Following that work, we begin by computing the Gram

 Gi,j=∑x,ylayern[x, y, i]⋅layern[x, y, j]

 Gi,j=x,y∑layern[x, y, i]⋅layern[x, y, j]

 Cdiversity=−∑a∑b≠a vec(Ga)⋅vec(Gb)∣∣vec(Ga)∣∣ ∣∣vec(Gb)∣∣

 C\_{	ext{diversity}} = - \sum\_{a} \sum\_{b
eq a} ~

rac{	ext{vec}(G\_a) \cdot

	ext{vec}(G\_b)}{||	ext{vec}(G\_a)||~||	ext{vec}(G\_b)||}

 Cdiversity=−a∑b≠a∑ ∣∣vec(Ga)∣∣ ∣∣vec(Gb)∣∣vec(Ga)⋅vec(Gb)

 One possibility is to penalize the cosine similarity of different examples.

![](Feature%20Visualization_files/mixed4a_97_optimized.png)

Simple Optimization

![](Feature%20Visualization_files/mixed4a_97_diversity.png)

[Reproduce in a

![](Feature%20Visualization_files/mixed4a_97_examples.jpg)

Dataset examples

 Diverse feature visualizations allow us to more closely pinpoint what

activates a neuron, to the degree that we can make, and — by looking at

 For example, let's examine this simple optimization result.

![](Feature%20Visualization_files/mixed4a_143_optimized.png)

Simple optimization

 Looking at it in isolation one might infer that this neuron activates

on the top of dog heads, as the optimization shows both eyes and only

downward curved edges.

 Looking at the optimization with diversity however, we see

optimization results which don't include eyes, and also one which

includes upward curved edges. We thus have to broaden our expectation of

 what this neuron activates on to be mostly about the fur texture.

Checking this hypothesis against dataset examples shows that is broadly

correct. Note the spoon with a texture and color similar enough to dog

fur for the neuron to activate.

![](Feature%20Visualization_files/mixed4a_143_diversity.png)

 Optimization with diversity. *Layer mixed4a, Unit 143*

![](Feature%20Visualization_files/mixed4a_143_examples.jpg)

Dataset examples

 The effect of diversity can be even more striking in higher level

neurons, where it can show us different types of objects that stimulate a

 neuron.

 For example, one neuron responds to different kinds of balls, even

though they have a variety of appearances.

![](Feature%20Visualization_files/mixed5a_9_optimized.png)

Simple Optimization

![](Feature%20Visualization_files/mixed5a_9_diversity.png)

![](Feature%20Visualization_files/mixed5a_9_examples.jpg)

Dataset examples

 This simpler approach has a number of shortcomings:

 For one, the pressure to make examples different can cause unrelated

artifacts (such as eyes) to appear.

 Additionally, the optimization may make examples be different in an

unnatural way.

 For example, in the above example one might want to see examples of

soccer balls clearly separated from other types of balls like golf or

tennis balls.

 Dataset based approaches such as Wei *et al.*

 can split features apart more naturally — however they may not be as

helpful in understanding how the model will behave on different data.

 Diversity also starts to brush on a more fundamental issue: while the

examples above represent a mostly coherent idea, there are also neurons

that represent strange mixtures of ideas.

 Below, a neuron responds to two types of animal faces, and also to car

 bodies.

![](Feature%20Visualization_files/mixed4e_55_optimized.png)

Simple Optimization

![](Feature%20Visualization_files/mixed4e_55_diversity.png)

![](Feature%20Visualization_files/mixed4e_55_examples.jpg)

Dataset examples

---

Interaction between Neurons

---------------------------

 If neurons are not the right way to understand neural nets, what is?

 This framing unifies the concepts "neurons" and "combinations of

neurons" as "vectors in activation space". It allows us to ask: Should

we expect the directions of the basis vectors to be any more

interpretable than the directions of other vectors in this space?

 More recently Bau, Zhou *et al.*

 found the directions of the basis vectors to be interpretable more

often than random directions.

 Our experience is broadly consistent with both results; we find that

random directions often seem interpretable, but at a lower rate than

basis directions.

[Reproduce in a

*mixed3a, random direction*

*mixed4c, random direction*

*mixed4d, random direction*

*mixed5a, random direction*

By jointly optimizing two neurons we can get a sense of how they interact.

[Reproduce in a

Neuron 1

Neuron 2

Jointly optimized

 These examples show us how neurons jointly represent images.

 The optimization objective is a linear

interpolation between the individual channel objectives. To get the

interpolations to look better, we also add a small alignment objective

that encourages lower layer activations to be similar. We additionally

use a combination of separate and shared image parameterizations to make

 it easier for the optimization algorithm to cause objects to line up,

while still giving it the freedom to create any image it needs to.

 This is similar to interpolating in the latent space of generative models.

[Reproduce in a

Layer 4a, Unit 476

Layer 4a, Unit 460

 This is only starting to scratch the surface of how neurons interact.

 The truth is that we have almost no clue how to select meaningful

directions, or whether there even exist particularly meaningful

directions.

 Independent of finding directions, there are also questions on how

directions interact — for example, interpolation can show us how a small

 number of directions interact, but in reality there are hundreds of

directions interacting.

---

The Enemy of Feature Visualization

----------------------------------

 If you want to visualize features, you might just optimize an image to

 make neurons fire.

 Unfortunately, this doesn't really work.

 Instead, you end up with a kind of neural network optical

illusion — an image full of noise and nonsensical high-frequency

patterns that the network responds strongly to.

Even if you carefully tune learning rate, you'll get noise.

Optimization results are enlarged to show detail and artifacts.

[Reproduce in a

Learning Rate

(0.05)

Step 1

Step 32

Step 128

Step 256

Step 2048

 but the image is dominated by these high frequency patterns.

 We don't fully understand why these high frequency patterns form,

 but an important part seems to be strided convolutions and pooling

operations, which create high-frequency patterns in the gradient .

input

conv2d0

conv2d1

conv2d2

mixed3a

mixed3b

mixed4a

mixed4b

mixed4c

mixed4d

mixed4e

mixed5a

 These high-frequency patterns show us that, while optimization based

visualization's freedom from constraints is appealing, it's a

double-edged sword.

 Without any constraints on images, we end up with adversarial

examples.

 These are certainly interesting, but if we want to understand how

these models work in real life, we need to somehow move past them…

### The Spectrum of Regularization

 Dealing with this high frequency noise has been one of the primary

challenges and overarching threads of feature visualization research.

 If you want to get useful visualizations, you need to impose a more

natural structure using some kind of prior, regularizer, or constraint.

 In fact, if you look at most notable papers on feature visualization,

one of their main points will usually be an approach to regularization.

 Researchers have tried a lot of different things!

 In the middle we have three main families of regularization options.

 #feature-vis-history .row {

 }

 #feature-vis-history .line {

 width: 900px;

 }

 #feature-vis-history .row .info {

 display: inline-block;

 width: 320px;

 height: 56px;

 vertical-align: top;

 }

 #feature-vis-history .row .info img {

 width: 56px;

 height: 56px;

 border-radius: 5px;

 background: #EEE;

 }

 #feature-vis-history figcaption {

 line-height: 16px;

 vertical-align: top;

 }

 #feature-vis-history .row .info figcaption {

 width: 250px;

 margin-left: 8px;

 display: inline-block;

 }

 #feature-vis-history .row.header-row .info figcaption {

 margin-left: 0;

 }

 #feature-vis-history .header-row .sub-headers {

 display: inline-block;

 margin-top: 16px;

 margin-bottom: 10px;

 vertical-align: top;

 }

 #feature-vis-history .header-row .sub-headers figcaption {

 width: 100px; /\*104px;\*/

 display: inline-block;

 word-wrap: break-word;

 word-break: keep-all;

 }

 #feature-vis-history .row .spacer {

 display: inline-block;

 height: 48px;

 width: 45px;

 }

 #feature-vis-history .info p {

 margin-bottom: 4px;

 }

 #feature-vis-history .info .paper-title {

 font-weight: bold;

 }

 #feature-vis-history .info .paper-text {

 }

 @media (max-width: 1100px) {

 #feature-vis-history .row .spacer {

 display: inline-block;

 height: 48px;

 width: 24px;

 }

 #feature-vis-history .line {

 width: 750px;

 }

 }

 #feature-vis-history .row .category-check-container {

 /\*vertical-align: middle;\*/

 display: inline-block;

 margin: 8px;

 margin-left: 0px;

 margin-right: 70px;

 width: 32px;

 height: 32px;

 border-radius: 5px;

 border: 1px solid #CCC;

 }

 #feature-vis-history .row .category-check-container .category-check {

 margin: 6px;

 width: 20px;

 height: 20px;

 border-radius: 6px;

 }

 #feature-vis-history .row .category-check-container .set {

 background: #909092;

 }

 #feature-vis-history .line {

 margin-top: 4px;

 margin-bottom: 4px;

 margin-left: 67px;

 height: 1px;

 background: #EAEAEA;

 }

**Weak Regularization**

 avoids misleading correlations, but is less connected to real use.

**Unregularized**

**FrequencyPenalization**

**TransformationRobustness**

**Strong Regularization**

 gives more realistic examples at risk of misleading correlations.

**LearnedPrior**

**DatasetExamples**

![](Feature%20Visualization_files/Erhan2009.png)

Erhan, *et al.*, 2009

Introduced core idea. Minimal regularization.

![](Feature%20Visualization_files/Szegedy2013.png)

Szegedy, *et al.*, 2013

Adversarial examples. Visualizes with dataset examples.

![](Feature%20Visualization_files/Mahendran2015.png)

Mahendran & Vedaldi, 2015

Introduces total variation regularizer. Reconstructs input from representation.

![](Feature%20Visualization_files/Nguyen2015.png)

Nguyen, *et al.*, 2015

Explores counterexamples. Introduces image blurring.

![](Feature%20Visualization_files/Mordvintsev2015.png)

Mordvintsev, *et al.*, 2015

Introduced jitter & multi-scale. Explored GMM priors for classes.

![](Feature%20Visualization_files/Oygard2015.png)

Øygard, *et al.*, 2015

Introduces gradient blurring.

(Also uses jitter.)

![](Feature%20Visualization_files/Tyka2016.png)

Tyka, *et al.*, 2016

Regularizes with bilateral filters.

(Also uses jitter.)

![](Feature%20Visualization_files/Mordvintsev2016.png)

Mordvintsev, *et al.*, 2016

Normalizes gradient frequencies.

(Also uses jitter.)

![](Feature%20Visualization_files/Nguyen2016a.png)

Nguyen, *et al.*, 2016

Paramaterizes images with GAN generator.

![](Feature%20Visualization_files/Nguyen2016b.png)

Nguyen, *et al.*, 2016

Uses denoising autoencoder prior to make a generative model.

### Three Families of Regularization

 If we think about blurring in Fourier space, it is equivalent to

adding a scaled L2 penalty to the objective, penalizing each

Fourier-component based on its frequency.

 These techniques are in some ways very similar to the above and in

Frequency penalization directly targets high frequency noise

[Reproduce in a

L1

(-0.05)

Total Variation

(-0.25)

Blur

(-1)

Step 1

Step 32

Step 128

Step 256

Step 2048

 Even a small amount seems to be very effective in the case of images ,

 especially when combined with a more general regularizer for high-frequencies .

[Reproduce in a

Jitter

(1px)

Rotate

(5°)

Scale

(1.1×)

Step 1

Step 32

Step 128

Step 256

Step 2048

**Learned priors.**

 Our previous regularizers use very simple heuristics to keep examples

reasonable.

 A natural next step is to actually learn a model of the real data and

try to enforce that.

 With a strong model, this becomes similar to searching over the

dataset.

 This approach produces the most photorealistic visualizations, but it

may be unclear what came from the model being visualized and what came

from the prior.

 such as a GAN or VAE,

 and optimize within that latent space .

 this allows you to jointly optimize for the prior along with your objective .

 When one optimizes for the prior and the probability of a class, one

recovers a generative model of the data conditioned on that particular

class.

 Finally, Wei *et al.*

approximate a generative model prior, at least for the color

distribution, by penalizing distance between patches of the output and

the nearest patches retrieved from a database of image patches collected

 from the training data.

---

Preconditioning and Parameterization

------------------------------------

 It's not clear this is really a regularizer:

 If it isn't a regularizer, what does transforming the gradient like this do?

 You can think of it as doing steepest descent to optimize the same objective,

 Gradient blurring is equivalent

 to gradient descent in a different parameterization of image space,

where high frequency dimensions are stretched to make moving in those

 This changes which direction of descent will be steepest, and how fast

 the optimization moves in each direction, but it does not change what

the minimums are.

 If there are many local minima, it can stretch and shrink their basins

 of attraction, changing which ones the optimization process falls into.

 As a result, using the right preconditioner can make an optimization

problem radically easier.

 How can we choose a preconditioner that will give us these benefits?

 A good first guess is one that makes your data decorrelated and whitened.

 In the case of images this means doing gradient descent in the Fourier basis,

 This points to a profound fact about the Fourier transform.

 As long as a correlation is consistent across spatial

positions — such as the correlation between a pixel and its left

neighbor being the same across all positions of an image — the Fourier

coefficients will be independent variables.

 To see this, note that such a spatially consistent correlation can

be expressed as a convolution, and by the convolution theorem becomes

pointwise multiplication after the Fourier transform.

 with frequencies scaled so that they all have equal energy.

 Note that we have to be careful to get the colors to be

decorrelated, too. The Fourier transforms decorrelates spatially, but a

correlation will still exist between colors.

 To address this, we explicitly measure the correlation between

colors in the training set and use a Cholesky decomposition to

decorrelate them. Compare the directions of steepest decent before and

after decorrelating colors:

![](Feature%20Visualization_files/correlated_colors.jpeg)

 Correlated Colors

![](Feature%20Visualization_files/decorrelated_colors.jpeg)

 Decorrelated Colors

 Three directions of steepest descent under different notions of distance

Image

**L∞ metric**

used in adverserial work

**L2 metric**

regular gradient

**decorrelated space**

 All of these directions are valid descent directions for the same objective,

 but we can see they're radically different.

 Notice that optimizing in the decorrelated space reduces high frequencies,

 while using L∞ increases them.

 It's hard to do really fair comparisons because of hyperparameters, but the

 resulting visualizations seem a lot better — and develop faster, too.

[Reproduce in a

Learning Rate

(0.05)

Decorrelated Space

Transformation Robustness

Step 1

Step 32

Step 128

Step 256

Step 2048

 (Unless otherwise noted, the images in this article were optimizing in

 the decorrelated space and a suite of transformation robustness

techniques.

 Images were optimized for 2560 steps in a color-decorrelated

fourier-transformed space, using Adam at a learning rate of 0.05.

 We used each of following transformations in the given order at each

 step of the optimization:

 • Padding the input by 16 pixels to avoid edge artifacts

 • Jittering by up to 16 pixels

 • Jittering a second time by up to 8 pixels

 • Cropping the padding

)

 Is the preconditioner merely accelerating descent, bringing us to the

same place

 normal gradient descent would have brought us if we were patient

enough?

 Or is it also regularizing, changing which local minima we get

attracted to?

 It's hard to tell for sure.

 On the one hand, gradient descent seems to continue improving as you

exponentially increase the number of optimization steps — it hasn't

converged, it's just moving very slowly.

 On the other hand, if you turn off all other regularizers, the

preconditioner seems to reduce high-frequency patterns.

---

Conclusion

----------

 Neural feature visualization has made great progress over the last few years.

 In the quest to make neural networks interpretable, feature visualization

 stands out as one of the most promising and developed research directions.

 By itself, feature visualization will never give a completely satisfactory

 understanding. We see it as one of the fundamental building blocks that,

 There remains still a lot of important work to be done in improving

feature visualization.

 Some issues that stand out include understanding neuron interaction,

finding which units are most meaningful for understanding neural net

activations, and giving a holistic view of the facets of a feature.

Going deeper with convolutions

Imagenet: A large-scale hierarchical image database

Visualizing higher-layer features of a deep network

Inceptionism: Going deeper into neural networks

Deep inside convolutional networks: Visualising image classification models and saliency maps

A neural algorithm of artistic style

Understanding deep image representations by inverting them

Understanding Intra-Class Knowledge Inside {CNN}

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Intriguing properties of neural networks

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Deconvolution and checkerboard artifacts

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Visualizing GoogLeNet Classes

Class visualization with bilateral filters

DeepDreaming with TensorFlow

Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

Tensorflow: Large-scale machine learning on heterogeneous distributed systems