Feature-wise transformations

Vincent Dumoulin
Ethan Perez
Nathan Schucher
Florian Strub
Harm de Vries
Aaron Courville
Yoshua Bengio

 Many real-world problems require integrating multiple sources of information.

 Sometimes these problems involve multiple, distinct modalities of

 information — vision, language, audio, etc. — as is required

 to understand a scene in a movie or answer a question about an image.

 Other times, these problems involve multiple sources of the same

 kind of input, i.e. when summarizing several documents or drawing one

 image in the style of another.

 When approaching such problems, it often makes sense to process one source

 of information *in the context of* another; for instance, in the

 right example above, one can extract meaning from the image in the context

 of the question. In machine learning, we often refer to this context-based

 processing as *conditioning*: the computation carried out by a model

 is conditioned or *modulated* by information extracted from an

 auxiliary input.

 Finding an effective way to condition on or fuse sources of information

 is an open research problem, and

 in this article, we concentrate on a specific family of approaches we call

 *feature-wise transformations*.

 We will examine the use of feature-wise transformations in many neural network

 architectures to solve a surprisingly large and diverse set of problems;

 their success, we will argue, is due to being flexible enough to learn an

 effective representation of the conditioning input in varied settings.

 In the language of multi-task learning, where the conditioning signal is

 taken to be a task description, feature-wise transformations

 learn a task representation which allows them to capture and leverage the

 relationship between multiple sources of information, even in remarkably

 different problem settings.

---

Feature-wise transformations

----------------------------

 To motivate feature-wise transformations, we start with a basic example,

 where the two inputs are images and category labels, respectively. For the

 purpose of this example, we are interested in building a generative model of

 images of various classes (puppy, boat, airplane, etc.). The model takes as

 input a class and a source of random noise (e.g., a vector sampled from a

 normal distribution) and outputs an image sample for the requested class.

 Our first instinct might be to build a separate model for each

 class. For a small number of classes this approach is not too bad a solution,

 but for thousands of classes, we quickly run into scaling issues, as the number

 of parameters to store and train grows with the number of classes.

 We are also missing out on the opportunity to leverage commonalities between

 classes; for instance, different types of dogs (puppy, terrier, dalmatian,

 etc.) share visual traits and are likely to share computation when

 mapping from the abstract noise vector to the output image.

 Now let’s imagine that, in addition to the various classes, we also need to

 model attributes like size or color. In this case, we can’t

 reasonably expect to train a separate network for *each* possible

 conditioning combination! Let’s examine a few simple options.

 A quick fix would be to concatenate a representation of the conditioning

 information to the noise vector and treat the result as the model’s input.

 This solution is quite parameter-efficient, as we only need to increase

 Maybe this assumption is correct, or maybe it’s not; perhaps the

 model does not need to incorporate the conditioning information until late

 into the generation process (e.g., right before generating the final pixel

 carry this information around unaltered for many layers.

 Because this operation is cheap, we might as well avoid making any such

 assumptions and concatenate the conditioning representation to the input of

 *all* layers in the network. Let’s call this approach

 *concatenation-based conditioning*.

 Another efficient way to integrate conditioning information into the network

 is via *conditional biasing*, namely, by adding a *bias* to

 the hidden layers based on the conditioning representation.

 Interestingly, conditional biasing can be thought of as another way to

 implement concatenation-based conditioning. Consider a fully-connected

 linear layer applied to the concatenation of an input

 x\mathbf{x}x and a conditioning representation

 z\mathbf{z}z:

 The same argument applies to convolutional networks, provided we ignore

 the border effects due to zero-padding.

 Yet another efficient way to integrate class information into the network is

 via *conditional scaling*, i.e., scaling hidden layers

 based on the conditioning representation.

 A special instance of conditional scaling is feature-wise sigmoidal gating:

 we scale each feature by a value between 000 and

 111 (enforced by applying the logistic function), as a

 function of the conditioning representation. Intuitively, this gating allows

 the conditioning information to select which features are passed forward

 and which are zeroed out.

 Given that both additive and multiplicative interactions seem natural and

 intuitive, which approach should we pick? One argument in favor of

 *multiplicative* interactions is that they are useful in learning

 relationships between inputs, as these interactions naturally identify

 “matches”: multiplying elements that agree in sign yields larger values than

 multiplying elements that disagree. This property is why dot products are

 often used to determine how similar two vectors are.

 Multiplicative interactions alone have had a history of success in various

 domains — see [Bibliographic Notes](#bibliographic-notes).

 One argument in favor of *additive* interactions is that they are

 more natural for applications that are less strongly dependent on the

 joint values of two inputs, like feature aggregation or feature detection

 (i.e., checking if a feature is present in either of two inputs).

 In the spirit of making as few assumptions about the problem as possible,

 we may as well combine *both* into a

 conditional *affine transformation*.

 An affine transformation is a transformation of the form

 y=m∗x+by = m \* x + by=m∗x+b.

 All methods outlined above share the common trait that they act at the

 *feature* level; in other words, they leverage *feature-wise*

 interactions between the conditioning representation and the conditioned

 network. It is certainly possible to use more complex interactions,

 but feature-wise interactions often strike a happy compromise between

 effectiveness and efficiency: the number of scaling and/or shifting

 coefficients to predict scales linearly with the number of features in the

 network. Also, in practice, feature-wise transformations (often compounded

 across multiple layers) frequently have enough capacity to model complex

 phenomenon in various settings.

 Lastly, these transformations only enforce a limited inductive bias and

 remain domain-agnostic. This quality can be a downside, as some problems may

 be easier to solve with a stronger inductive bias. However, it is this

 characteristic which also enables these transformations to be so widely

 effective across problem domains, as we will later review.

### Nomenclature

 To continue the discussion on feature-wise transformations we need to

 abstract away the distinction between multiplicative and additive

 interactions. Without losing generality, let’s focus on feature-wise affine

 transformations, and let’s adopt the nomenclature of Perez et al.

 , which formalizes conditional affine

 transformations under the acronym *FiLM*, for Feature-wise Linear

 Modulation.

 Strictly speaking, *linear* is a misnomer, as we allow biasing, but

 we hope the more rigorous-minded reader will forgive us for the sake of a

 better-sounding acronym.

 We say that a neural network is modulated using FiLM, or *FiLM-ed*,

 after inserting *FiLM layers* into its architecture. These layers are

 parametrized by some form of conditioning information, and the mapping from

 conditioning information to FiLM parameters (i.e., the shifting and scaling

 coefficients) is called the *FiLM generator*.

 In other words, the FiLM generator predicts the parameters of the FiLM

 layers based on some auxiliary input.

 Note that the FiLM parameters are parameters in one network but predictions

 from another network, so they aren’t learnable parameters with fixed

 weights as in the fully traditional sense.

 For simplicity, you can assume that the FiLM generator outputs the

 concatenation of all FiLM parameters for the network architecture.

 As the name implies, a FiLM layer applies a feature-wise affine

 transformation to its input. By *feature-wise*, we mean that scaling

 and shifting are applied element-wise, or in the case of convolutional

 networks, feature map -wise.

 To expand a little more on the convolutional case, feature maps can be

 thought of as the same feature detector being evaluated at different

 spatial locations, in which case it makes sense to apply the same affine

 transformation to all spatial locations.

 In other words, assuming x\mathbf{x}x is a FiLM layer’s

 input, z\mathbf{z}z is a conditioning input, and

 γ\gammaγ and β\betaβ are

 z\mathbf{z}z-dependent scaling and shifting vectors,

 FiLM(x)=γ(z)⊙x+β(z).

 \textrm{FiLM}(\mathbf{x}) = \gamma(\mathbf{z}) \odot \mathbf{x}

 + \beta(\mathbf{z}).

 FiLM(x)=γ(z)⊙x+β(z).

 You can interact with the following fully-connected and convolutional FiLM

 layers to get an intuition of the sort of modulation they allow:

 In addition to being a good abstraction of conditional feature-wise

 transformations, the FiLM nomenclature lends itself well to the notion of a

 *task representation*. From the perspective of multi-task learning,

 we can view the conditioning signal as the task description. More

 specifically, we can view the concatenation of all FiLM scaling and shifting

 coefﬁcients as both an instruction on *how to modulate* the

 conditioned network and a *representation* of the task at hand. We

 will explore and illustrate this idea later on.

---

Feature-wise transformations in the literature

----------------------------------------------

 Feature-wise transformations find their way into methods applied to many

 problem settings, but because of their simplicity, their effectiveness is

 seldom highlighted in lieu of other novel research contributions. Below are

 a few notable examples of feature-wise transformations in the literature,

 grouped by application domain. The diversity of these applications

 underscores the flexible, general-purpose ability of feature-wise

 interactions to learn effective task representations.

expand all

Visual question-answering+

 Perez et al.  use

 FiLM layers to build a visual reasoning model

 trained on the CLEVR dataset  to

 answer multi-step, compositional questions about synthetic images.

 The model’s linguistic pipeline is a FiLM generator which

 extracts a question representation that is linearly mapped to

 FiLM parameter values. Using these values, FiLM layers inserted within each

 residual block condition the visual pipeline. The model is trained

 end-to-end on image-question-answer triples. Strub et al.

  later on improved on the model by

 using an attention mechanism to alternate between attending to the language

 input and generating FiLM parameters layer by layer. This approach was

 better able to scale to settings with longer input sequences such as

 dialogue and was evaluated on the GuessWhat?!

 and ReferIt  datasets.

 de Vries et al.  leverage FiLM

 to condition a pre-trained network. Their model’s linguistic pipeline

 modulates the visual pipeline via conditional batch normalization,

 real-world images on the GuessWhat?!

 and VQAv1  datasets.

 The visual pipeline consists of a pre-trained residual network that is

 fixed throughout training. The linguistic pipeline manipulates the visual

 pipeline by perturbing the residual network’s batch normalization

 parameters, which re-scale and re-shift feature maps after activations

 have been normalized to have zero mean and unit variance. As hinted

 earlier, conditional batch normalization can be viewed as an instance of

 FiLM where the post-normalization feature-wise affine transformation is

 replaced with a FiLM layer.

Style transfer+

 Dumoulin et al.  use

 feature-wise affine transformations — in the form of conditional

 instance normalization layers — to condition a style transfer

 network on a chosen style image. Like conditional batch normalization

 discussed in the previous subsection,

 conditional instance normalization can be seen as an instance of FiLM

 where a FiLM layer replaces the post-normalization feature-wise affine

 instance normalization parameters, and it applies normalization with these

 style-specific parameters.

 Dumoulin et al.  use a simple

 embedding lookup to produce instance normalization parameters, while

 Ghiasi et al.  further

 introduce a *style prediction network*, trained jointly with the

 style transfer network to predict the conditioning parameters directly from

 a given style image. In this article we opt to use the FiLM nomenclature

 because it is decoupled from normalization operations, but the FiLM

 layers used by Perez et al.  were

 themselves heavily inspired by the conditional normalization layers used

 by Dumoulin et al. .

 Yang et al.  use a related

 architecture for video object segmentation — the task of segmenting a

 particular object throughout a video given that object’s segmentation in the

 first frame. Their model conditions an image segmentation network over a

 video frame on the provided first frame segmentation using feature-wise

 scaling factors, as well as on the previous frame using position-wise

 biases.

 So far, the models we covered have two sub-networks: a primary

 network in which feature-wise transformations are applied and a secondary

 network which outputs parameters for these transformations. However, this

 distinction between *FiLM-ed network* and *FiLM generator*

 is not strictly necessary. As an example, Huang and Belongie

  propose an alternative

 style transfer network that uses adaptive instance normalization layers,

 which compute normalization parameters using a simple heuristic.

 Adaptive instance normalization can be interpreted as inserting a FiLM

 layer midway through the model. However, rather than relying

 on a secondary network to predict the FiLM parameters from the style

 image, the main network itself is used to extract the style features

 used to compute FiLM parameters. Therefore, the model can be seen as

 *both* the FiLM-ed network and the FiLM generator.

Image recognition+

 neural network’s activations *themselves* as conditioning

 information. This idea gives rise to

 *self-conditioned* models.

 Highway Networks  are a prime

 example of applying this self-conditioning principle. They take inspiration

 from the LSTMs’  heavy use of

 feature-wise sigmoidal gating in their input, forget, and output gates to

 regulate information flow:

inputsub-networksigmoidal layer1 - xoutput

 The ImageNet 2017 winning model  also

 employs feature-wise sigmoidal gating in a self-conditioning manner, as a

 way to “recalibrate” a layer’s activations conditioned on themselves.

Natural language processing+

 For statistical language modeling (i.e., predicting the next word

 in a sentence), the LSTM

 constitutes a popular class of recurrent network architectures. The LSTM

 relies heavily on feature-wise sigmoidal gating to control the

 information flow in and out of the memory or context cell

 c\mathbf{c}c, based on the hidden states

 h\mathbf{h}h and inputs x\mathbf{x}x at

 every timestep t\mathbf{t}t.

ct-1tanhcthtsigmoidsigmoidtanhsigmoidlinearlinearlinearlinearht-1xtconcatenate

 Also in the domain of language modeling, Dauphin et al.  use sigmoidal

 gating in their proposed *gated linear unit*, which uses half of the

 input features to apply feature-wise sigmoidal gating to the other half.

 Gehring et al.  adopt this

 architectural feature, introducing a fast, parallelizable model for machine

 translation in the form of a fully convolutional network.

 The Gated-Attention Reader

 uses feature-wise scaling, extracting information

 from text by conditioning a document-reading network on a query. Its

 architecture consists of multiple Gated-Attention modules, which involve

 element-wise multiplications between document representation tokens and

 token-specific query representations extracted via soft attention on the

 query representation tokens.

Reinforcement learning+

 The Gated-Attention architecture

 uses feature-wise sigmoidal gating to fuse linguistic and visual

 information in an agent trained to follow simple “go-to” language

 instructions in the VizDoom  3D

 environment.

 Bahdanau et al. use FiLM

 layers to condition Neural Module Network

 and LSTM -based policies to follow

 basic, compositional language instructions (arranging objects and going

 to particular locations) in a 2D grid world. They train this policy

 in an adversarial manner using rewards from another FiLM-based network,

 trained to discriminate between ground-truth examples of achieved

 instruction states and failed policy trajectories states.

 Outside instruction-following, Kirkpatrick et al.

  also use

 game-specific scaling and biasing to condition a shared policy network

 trained to play 10 different Atari games.

Generative modeling+

 The conditional variant of DCGAN ,

 a well-recognized network architecture for generative adversarial networks

 , uses concatenation-based

 conditioning. The class label is broadcasted as a feature map and then

 concatenated to the input of convolutional and transposed convolutional

 layers in the discriminator and generator networks.

 For convolutional layers, concatenation-based conditioning requires the

 network to learn redundant convolutional parameters to interpret each

 constant, conditioning feature map; as a result, directly applying a

 conditional bias is more parameter efficient, but the two approaches are

 still mathematically equivalent.

 PixelCNN

 and WaveNet  — two recent

 advances in autoregressive, generative modeling of images and audio,

 respectively — use conditional biasing. The simplest form of

 conditioning in PixelCNN adds feature-wise biases to all convolutional layer

 outputs. In FiLM parlance, this operation is equivalent to inserting FiLM

 layers after each convolutional layer and setting the scaling coefficients

 to a constant value of 1.

 The authors also describe a location-dependent biasing scheme which

 cannot be expressed in terms of FiLM layers due to the absence of the

 feature-wise property.

 WaveNet describes two ways in which conditional biasing allows external

 information to modulate the audio or speech generation process based on

 conditioning input:

1. **Global conditioning** applies the same conditional bias

 to the whole generated sequence and is used e.g. to condition on speaker

 identity.

2. **Local conditioning** applies a conditional bias which

 varies across time steps of the generated sequence and is used e.g. to

 let linguistic features in a text-to-speech model influence which sounds

 are produced.

 As in PixelCNN, conditioning in WaveNet can be viewed as inserting FiLM

 layers after each convolutional layer. The main difference lies in how

 the FiLM-generating network is defined: global conditioning

 expresses the FiLM-generating network as an embedding lookup which is

 broadcasted to the whole time series, whereas local conditioning expresses

 it as a mapping from an input sequence of conditioning information to an

 output sequence of FiLM parameters.

Speech recognition+

 Kim et al. modulate a deep

 bidirectional LSTM using a form

 of conditional normalization. As discussed in the

 *Visual question-answering* and *Style transfer* subsections,

 conditional normalization can be seen as an instance of FiLM where

 the post-normalization feature-wise affine transformation is replaced

 with a FiLM layer.

 The key difference here is that the conditioning signal does not come from

 an external source but rather from utterance

 summarization feature vectors extracted in each layer to adapt the model.

Domain adaptation and few-shot learning+

 For domain adaptation, Li et al.

 find it effective to update the per-channel batch normalization

 statistics (mean and variance) of a network trained on one domain with that

 network’s statistics in a new, target domain. As discussed in the

 *Style transfer* subsection, this operation is akin to using the network as

 both the FiLM generator and the FiLM-ed network. Notably, this approach,

 along with Adaptive Instance Normalization, has the particular advantage of

 not requiring any extra trainable parameters.

 For few-shot learning, Oreshkin et al.

  explore the use of FiLM layers to

 provide more robustness to variations in the input distribution across

 few-shot learning episodes. The training set for a given episode is used to

 produce FiLM parameters which modulate the feature extractor used in a

 Prototypical Networks

 meta-training procedure.

---

Related ideas

-------------

 Aside from methods which make direct use of feature-wise transformations,

 the FiLM framework connects more broadly with the following methods and

 concepts.

expand all

Zero-shot learning+

 The idea of learning a task representation shares a strong connection with

 zero-shot learning approaches. In zero-shot learning, semantic task

 embeddings may be learned from external information and then leveraged to

 make predictions about classes without training examples. For instance, to

 generalize to unseen object categories for image classification, one may

 construct semantic task embeddings from text-only descriptions and exploit

 objects’ text-based relationships to make predictions for unseen image

 categories. Frome et al. , Socher et

 al. , and Norouzi et al.

  are a few notable exemplars

 of this idea.

HyperNetworks+

 The notion of a secondary network predicting the parameters of a primary

 (e.g., a recurrent neural network layer). From this perspective, the FiLM

 generator is a specialized HyperNetwork that predicts the FiLM parameters of

 the FiLM-ed network. The main distinction between the two resides in the

 number and specificity of predicted parameters: FiLM requires predicting far

 fewer parameters than Hypernetworks, but also has less modulation potential.

 The ideal trade-off between a conditioning mechanism’s capacity,

 regularization, and computational complexity is still an ongoing area of

 investigation, and many proposed approaches lie on the spectrum between FiLM

 and HyperNetworks (see [Bibliographic Notes](#bibliographic-notes)).

Attention+

 Some parallels can be drawn between attention and FiLM, but the two operate

 in different ways which are important to disambiguate.

 This difference stems from distinct intuitions underlying attention and

 FiLM: the former assumes that specific spatial locations or time steps

 contain the most useful information, whereas the latter assumes that

 specific features or feature maps contain the most useful information.

Bilinear transformations+

 With a little bit of stretching, FiLM can be seen as a special case of a

 bilinear transformation

  with low-rank weight

 matrices. A bilinear transformation defines the relationship between two

 inputs x\mathbf{x}x and z\mathbf{z}z and the

 kthk^{th}kth output feature yky\_kyk as

 yk=xTWkz.

 y\_k = \mathbf{x}^T W\_k \mathbf{z}.

 yk=xTWkz.

 Note that for each output feature yky\_kyk we have a separate

 matrix WkW\_kWk, so the full set of weights forms a

 multi-dimensional array.

 If we view z\mathbf{z}z as the concatenation of the scaling

 and shifting vectors γ\gammaγ and β\betaβ and

 if we augment the input x\mathbf{x}x with a 1-valued feature,

 As is commonly done to turn a linear transformation into an affine

 transformation.

 we can represent FiLM using a bilinear transformation by zeroing out the

 appropriate weight matrix entries:

 For some applications of bilinear transformations,

 see the [Bibliographic Notes](#bibliographic-notes).

---

Properties of the learned task representation

---------------------------------------------

 As hinted earlier, in adopting the FiLM perspective we implicitly introduce

 a notion of *task representation*: each task — be it a question

 about an image or a painting style to imitate — elicits a different

 set of FiLM parameters via the FiLM generator which can be understood as its

 representation in terms of how to modulate the FiLM-ed network. To help

 better understand the properties of this representation, let’s focus on two

 FiLM-ed models used in fairly different problem settings:

* The visual reasoning model of Perez et al.

 , which uses FiLM

 to modulate a visual processing pipeline based off an input question.

* The artistic style transfer model of Ghiasi et al.

 , which uses FiLM to modulate a

 feed-forward style transfer network based off an input style image.

 As a starting point, can we discern any pattern in the FiLM parameters as a

 function of the task description? One way to visualize the FiLM parameter

 space is to plot γ\gammaγ against β\betaβ,

 with each point corresponding to a specific task description and a specific

 feature map. If we color-code each point according to the feature map it

 belongs to we observe the following:

 The plots above allow us to make several interesting observations. First,

 FiLM parameters cluster by feature map in parameter space, and the cluster

 locations are not uniform across feature maps. The orientation of these

 clusters is also not uniform across feature maps: the main axis of variation

 can be γ\gammaγ-aligned, β\betaβ-aligned, or

 diagonal at varying angles. These findings suggest that the affine

 transformation in FiLM layers is not modulated in a single, consistent way,

 i.e., using γ\gammaγ only, β\betaβ only, or

 γ\gammaγ and β\betaβ together in some specific

 way. Maybe this is due to the affine transformation being overspecified, or

 maybe this shows that FiLM layers can be used to perform modulation

 operations in several distinct ways.

 Nevertheless, the fact that these parameter clusters are often somewhat

 “dense” may help explain why the style transfer model of Ghiasi et al.

  is able to perform style

 interpolations: any convex combination of FiLM parameters is likely to

 correspond to a meaningful parametrization of the FiLM-ed network.

Style 1Style 2 w InterpolationContent Image

 To some extent, the notion of interpolating between tasks using FiLM

 parameters can be applied even in the visual question-answering setting.

 Using the model trained in Perez et al. ,

 we interpolated between the model’s FiLM parameters for two pairs of CLEVR

 questions. Here we visualize the input locations responsible for

 The network seems to be softly switching where in the image it is looking,

 based on the task description. It is quite interesting that these semantically

 meaningful interpolation behaviors emerge, as the network has not been

 trained to act this way.

 Despite these similarities across problem settings, we also observe

 qualitative differences in the way in which FiLM parameters cluster as a

 function of the task description. Unlike the style transfer model, the

 visual reasoning model sometimes exhibits several FiLM parameter

 sub-clusters for a given feature map.

 At the very least, this may indicate that FiLM learns to operate in ways

 that are problem-specific, and that we should not expect to find a unified

 and problem-independent explanation for FiLM’s success in modulating FiLM-ed

 networks. Perhaps the compositional or discrete nature of visual reasoning

 requires the model to implement several well-defined modes of operation

 which are less necessary for style transfer.

 Focusing on individual feature maps which exhibit sub-clusters, we can try

 to infer how questions regroup by color-coding the scatter plots by question

 type.

 Sometimes a clear pattern emerges, as in the right plot, where color-related

 questions concentrate in the top-right cluster — we observe that

 questions either are of type *Query color* or *Equal color*,

 or contain concepts related to color. Sometimes it is harder to draw a

 conclusion, as in the left plot, where question types are scattered across

 the three clusters.

 In cases where question types alone cannot explain the clustering of the

 FiLM parameters, we can turn to the conditioning content itself to gain

 an understanding of the mechanism at play. Let’s take a look at two more

 plots: one for feature map 26 as in the previous figure, and another

 for a different feature map, also exhibiting several subclusters. This time

 we regroup points by the words which appear in their associated question.

 In the left plot, the left subcluster corresponds to questions involving

 objects positioned *in front* of other objects, while the right

 subcluster corresponds to questions involving objects positioned

 *behind* other objects. In the right plot we see some evidence of

 separation based on object material: the left subcluster corresponds to

 questions involving *matte* and *rubber* objects, while the

 right subcluster contains questions about *shiny* or

 *metallic* objects.

 The presence of sub-clusters in the visual reasoning model also suggests

 that question interpolations may not always work reliably, but these

 sub-clusters don’t preclude one from performing arithmetic on the question

 representations, as Perez et al.

 report.

 Perez et al.  report that this sort of

 task analogy is not always successful in correcting the model’s answer, but

 it does point to an interesting fact about FiLM-ed networks: sometimes the

 model makes a mistake not because it is incapable of computing the correct

 output, but because it fails to produce the correct FiLM parameters for a

 given task description. The reverse can also be true: if the set of tasks

 the model was trained on is insufficiently rich, the computational

 primitives learned by the FiLM-ed network may be insufficient to ensure good

 generalization. For instance, a style transfer model may lack the ability to

 produce zebra-like patterns if there are no stripes in the styles it was

 trained on. This could explain why Ghiasi et al.

  report that their style transfer

 model’s ability to produce pastiches for new styles degrades if it has been

 trained on an insufficiently large number of styles. Note however that in

 that case the FiLM generator’s failure to generalize could also play a role,

 and further analysis would be needed to draw a definitive conclusion.

 This points to a separation between the various computational

 primitives learned by the FiLM-ed network and the “numerical recipes”

 learned by the FiLM generator: the model’s ability to generalize depends

 both on its ability to parse new forms of task descriptions and on it having

 learned the required computational primitives to solve those tasks. We note

 that this multi-faceted notion of generalization is inherited directly from

 the multi-task point of view adopted by the FiLM framework.

 Let’s now turn our attention back to the overal structural properties of FiLM

 parameters observed thus far. The existence of this structure has already

 been explored, albeit more indirectly, by Ghiasi et al.

  as well as Perez et al.

 , who applied t-SNE

  on the FiLM parameter values.

 t-SNE projection of FiLM parameters for many task descriptions.

 The projection on the left is inspired by a similar projection done by Perez

 et al.  for their visual reasoning

 model trained on CLEVR and shows how questions group by question type.

 The projection on the right is inspired by a similar projection done by

 Ghiasi et al.  for their style

 transfer network. The projection does not cluster artists as neatly as the

 projection on the left, but this is to be expected, given that an artist’s

 style may vary widely over time. However, we can still detect interesting

 patterns in the projection: note for instance the isolated cluster (circled

 in the figure) in which paintings by Ivan Shishkin and Rembrandt are

 aggregated. While these two painters exhibit fairly different styles, the

 cluster is a grouping of their sketches.

 To summarize, the way neural networks learn to use FiLM layers seems to

 vary from problem to problem, input to input, and even from feature to

 feature; there does not seem to be a single mechanism by which the

 network uses FiLM to condition computation. This flexibility may

 explain why FiLM-related methods have been successful across such a

 wide variety of domains.

---

Discussion

----------

 Looking forward, there are still many unanswered questions.

 Do these experimental observations on FiLM-based architectures generalize to

 other related conditioning mechanisms, such as conditional biasing, sigmoidal

 gating, HyperNetworks, and bilinear transformations? When do feature-wise

 transformations outperform methods with stronger inductive biases and vice

 versa? Recent work combines feature-wise transformations with stronger

 inductive bias methods

 ,

 which could be an optimal middle ground. Also, to what extent are FiLM’s

 task representation properties

 inherent to FiLM, and to what extent do they emerge from other features

 of neural networks (i.e. non-linearities, FiLM generator

 depth, etc.)? If you are interested in exploring these or other

 questions about FiLM, we recommend looking into the code bases for

 FiLM models for [visual reasoning](https://github.com/ethanjperez/film)

 which we used as a

 starting point for our experiments here.

 Finally, the fact that changes on the feature level alone are able to

 compound into large and meaningful modulations of the FiLM-ed network is

 still very surprising to us, and hopefully future work will uncover deeper

 explanations. For now, though, it is a question that

 evokes the even grander mystery of how neural networks in general compound

 simple operations like matrix multiplications and element-wise

 non-linearities into semantically meaningful transformations.

FiLM: Visual Reasoning with a General Conditioning Layer

Learning visual reasoning without strong priors

CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning

Visual Reasoning with Multi-hop Feature Modulation

GuessWhat?! Visual object discovery through multi-modal dialogue

ReferItGame: Referring to objects in photographs of natural scenes

Modulating early visual processing by language

VQA: visual question answering

A learned representation for artistic style

Exploring the structure of a real-time, arbitrary neural artistic stylization network

Efficient video object segmentation via network modulation

Arbitrary style transfer in real-time with adaptive instance normalization

Highway networks

Long short-term memory

Squeeze-and-Excitation networks

On the state of the art of evaluation in neural language models

Language modeling with gated convolutional networks

Convolution sequence-to-sequence learning

Gated-attention readers for text comprehension

Gated-attention architectures for task-oriented language grounding

Vizdoom: A doom-based AI research platform for visual reinforcement learning

Learning to follow language instructions with adversarial reward induction

Neural module networks

Overcoming catastrophic forgetting in neural networks

Unsupervised representation learning with deep convolutional generative adversarial networks

Generative adversarial nets

Conditional image generation with PixelCNN decoders

WaveNet: A generative model for raw audio

Dynamic layer normalization for adaptive neural acoustic modeling in speech recognition

Adaptive batch normalization for practical domain adaptation

TADAM: Task dependent adaptive metric for improved few-shot learning

Prototypical networks for few-shot learning

Devise: A deep visual-semantic embedding model

Zero-shot learning through cross-modal transfer

Zero-shot learning by convex combination of semantic embeddings

HyperNetworks

Separating style and content with bilinear models

Visualizing data using t-SNE

A dataset and architecture for visual reasoning with a working memory

A parallel computation that assigns canonical object-based frames of reference

The correlation theory of brain function

Generating text with recurrent neural networks

Robust boltzmann machines for recognition and denoising

Factored conditional restricted Boltzmann machines for modeling motion style

Combining discriminative features to infer complex trajectories

Learning where to attend with deep architectures for image tracking

Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis

Convolutional learning of spatio-temporal features

Learning to relate images

Incorporating side information by adaptive convolution

Learning multiple visual domains with residual adapters

Predicting deep zero-shot convolutional neural networks using textual descriptions

Zero-shot task generalization with multi-task deep reinforcement learning

Separating style and content

Facial expression space learning

Personalized recommendation on dynamic content using predictive bilinear models

Like like alike: joint friendship and interest propagation in social networks

Matrix factorization techniques for recommender systems

Bilinear CNN models for fine-grained visual recognition

Convolutional two-stream network fusion for video action recognition

Multimodal compact bilinear pooling for visual question answering and visual grounding


Pascal Sturmfels
Scott Lundberg
Su-In Lee

Abstract: Path attribution methods are a gradient-based way            of explaining deep models. These methods require choosing a            hyperparameter known as the baseline input.            What does this hyperparameter mean, and how important is it? In this article,            we investigate these questions using image classification networks            as a case study. We discuss several different ways to choose a baseline            input and the assumptions that are implicit in each baseline.            Although we focus here on path attribution methods, our discussion of baselines            is closely connected with the concept of missingness in the feature space -            a concept that is critical to interpretability research.

Path attribution methods are a gradient-based way

 of explaining deep models. These methods require choosing a

 hyperparameter known as the *baseline input*.

 What does this hyperparameter mean, and how important is it? In this article,

 we investigate these questions using image classification networks

 as a case study. We discuss several different ways to choose a baseline

 input and the assumptions that are implicit in each baseline.

 Although we focus here on path attribution methods, our discussion of baselines

 is closely connected with the concept of missingness in the feature space -

 a concept that is critical to interpretability research.

Introduction

------------

 If you are in the business of training neural networks,

 you might have heard of the integrated gradients method, which

 was introduced at

 .

 The method computes which features are important

 to a neural network when making a prediction on a

 particular data point. This helps users

 understand which features their network relies on.

 Since its introduction,

 integrated gradients has been used to interpret

 networks trained on a variety of data types,

 including retinal fundus images

 and electrocardiogram recordings .

 If you’ve ever used integrated gradients,

 you know that you need to define a baseline input x’x’x’ before

 using the method. Although the original paper discusses the need for a baseline

 and even proposes several different baselines for image data - including

 the constant black image and an image of random noise - there is

 little existing research about the impact of this baseline.

 Is integrated gradients sensitive to the

 hyperparameter choice? Why is the constant black image

 a “natural baseline” for image data? Are there any alternative choices?

 In this article, we will delve into how this hyperparameter choice arises,

 and why understanding it is important when you are doing model interpretation.

 As a case-study, we will focus on image classification models in order

 to visualize the effects of the baseline input. We will explore several

 notions of missingness, including both constant baselines and baselines

 defined by distributions. Finally, we will discuss different ways to compare

 baseline choices and talk about why quantitative evaluation

 remains a difficult problem.

Image Classification

--------------------

 We focus on image classification as a task, as it will allow us to visually

 plot integrated gradients attributions, and compare them with our intuition

 , a convolutional

 neural network designed for the ImageNet dataset ,

 On the ImageNet validation set, Inception V4 has a top-1 accuracy of over 80%.

 We download weights from TensorFlow-Slim ,

 and visualize the predictions of the network on four different images from the

 validation set.

 Right: The predicted logits of the network on the original image. The

 network correctly classifies all images with high confidence.

 relative to the true class label.

 Although state of the art models perform well on unseen data,

 users may still be left wondering: *how* did the model figure

 out which object was in the image? There are a myriad of methods to

 interpret machine learning models, including methods to

 visualize and understand how the network represents inputs internally ,

 feature attribution methods that assign an importance score to each feature

 for a specific input ,

 and saliency methods that aim to highlight which regions of an image

 the model was looking at when making a decision

 .

 visualized as a saliency method, and a saliency method can assign importance

 scores to each individual pixel. In this article, we will focus

 on the feature attribution method integrated gradients.

 Formally, given a target input xxx and a network function fff,

 to the iiith feature value representing how much that feature

 indicates that feature strongly increases or decreases the network output

 the feature in question did not influence f(x)f(x)f(x).

 prediction using integrated gradients.

 The pixels in white indicate more important pixels. In order to plot

 attributions, we follow the same design choices as .

 That is, we plot the absolute value of the sum of feature attributions

 high-magnitude attributions dominating the color scheme.

A Better Understanding of Integrated Gradients

----------------------------------------------

 As you look through the attribution maps, you might find some of them

 To better understand this behavior, we need to explore how

 we generated feature attributions. Formally, integrated gradients

 defines the importance value for the iiith feature value as follows:

 \times \underbrace{\int\_{\alpha = 0}^ 1}\_{\text{From baseline to input…}}

 where xxx is the current input,

 “absence” of feature input. The subscript iii is used

 to denote indexing into the iiith feature.

 As the formula above states, integrated gradients gets importance scores

 But why would doing this make sense? Recall that the gradient of

 a function represents the direction of maximum increase. The gradient

 is telling us which pixels have the steepest local slope with respect

 to the output. For this reason, the gradient of a network at the input

 was one of the earliest saliency methods.

 Unfortunately, there are many problems with using gradients to interpret

 deep neural networks .

 One specific issue is that neural networks are prone to a problem

 sample even if the network depends heavily on those features. This can happen

 Intuitively, shifting the pixels in an image by a small amount typically

 doesn’t change what the network sees in the image. We can illustrate

 saturation by plotting the network output at all

 images between the baseline x’x’x’ and the current image. The figure

 below displays that the network

 output for the correct class increases initially, but then quickly flattens.

 A plot of network outputs at x’+α(x−x’)x’ + \alpha (x - x’)x’+α(x−x’).

 Notice that the network output saturates the correct class

 at small values of α\alphaα. By the time α=1\alpha = 1α=1,

 the network output barely changes.

 What we really want to know is how our network got from

 predicting essentially nothing at x’x’x’ to being

 completely saturated towards the correct output class at xxx.

 Which pixels, when scaled along this path, most

 increased the network output for the correct class? This is

 exactly what the formula for integrated gradients gives us.

 By integrating over a path,

 integrated gradients avoids problems with local gradients being

 saturated. We can break the original equation

 down and visualize it in three separate parts: the interpolated image between

 the baseline image and the target image, the gradients at the interpolated

 image, and accumulating many such gradients over α\alphaα.

 \int\_{\alpha’ = 0}^{\alpha} \underbrace{(x\_i - x’\_i) \times

 {\delta x\_i} d \alpha’}\_{\text{(2): Gradients at Interpolation}}

 approximation of the integral with 500 linearly-spaced points between 0 and 1.

 Integrated gradients, visualized. In the line chart, the red line refers to

 accumulate at small values of α\alphaα.

 We have casually omitted one part of the formula: the fact

 that we multiply by a difference from a baseline. Although

 we won’t go into detail here, this term falls out because we

 care about the derivative of the network

 function fff with respect to the path we are integrating over.

 That is, if we integrate over the

 straight-line between x’x’x’ and xxx, which

 we can represent as γ(α)=x’+α(x−x’)\gamma(\alpha) =

 x’ + \alpha(x - x’)γ(α)=x’+α(x−x’), then:

 δf(γ(α))δα=δf(γ(α))δγ(α)×δγ(α)δα=δf(x’+α’(x−x’))δxi×(xi−x’i)

 \frac{\delta f(\gamma(\alpha))}{\delta \alpha} =

 \frac{\delta f(\gamma(\alpha))}{\delta \gamma(\alpha)} \times

 \frac{\delta \gamma(\alpha)}{\delta \alpha} =

 \frac{\delta f(x’ + \alpha’ (x - x’))}{\delta x\_i} \times (x\_i - x’\_i)

 δαδf(γ(α))=δγ(α)δf(γ(α))×δαδγ(α)=δxiδf(x’+α’(x−x’))×(xi−x’i)

 The difference from baseline term is the derivative of the

 path function γ\gammaγ with respect to α\alphaα.

  The theory behind integrated gradients is discussed

 in more detail in the original paper. In particular, the authors

 show that integrated gradients satisfies several desirable

 properties, including the completeness axiom:

 Axiom 1: Completeness∑iϕiIG(f,x,x’)=f(x)−f(x’)

 \textrm{Axiom 1: Completeness}\\

 \sum\_i \phi\_i^{IG}(f, x, x’) = f(x) - f(x’)

 Axiom 1: Completenessi∑ϕiIG(f,x,x’)=f(x)−f(x’)

 Note that this theorem holds for any baseline x’x’x’.

 Completeness is a desirable property because it states that the

 importance scores for each feature break down the output of the network:

 each importance score represents that feature’s individual contribution to

 Although it’s not essential to our discussion here, we can prove

 that integrated gradients satisfies this axiom using the

 [fundamental

 full discussion of all of the properties that integrated

 gradients satisfies to the original paper, since they hold

 independent of the choice of baseline.  The completeness

 axiom also provides a way to measure convergence.

 In practice, we can’t compute the exact value of the integral. Instead,

 we use a discrete sum approximation with kkk linearly-spaced points between

 0 and 1 for some value of kkk. If we only chose 1 point to

 approximate the integral, that feels like too few. Is 10 enough? 100?

 Intuitively 1,000 may seem like enough, but can we be certain?

 As proposed in the original paper, we can use the completeness axiom

 as a sanity check on convergence: run integrated gradients with kkk

 and if the difference is large, re-run with a larger kkk

 Of course, this brings up a new question: what is “large” in this context?

 One heuristic is to compare the difference with the magnitude of the

 output itself.

 .

 The line chart above plots the following equation in red:

 ∑iϕiIG(f,x,x’;α)⏟(4): Sum of Cumulative Gradients up to α

 (4): Sum of Cumulative Gradients up to αi∑ϕiIG(f,x,x’;α)

 That is, it sums all of the pixel attributions in the saliency map.

 We can see that with 500 samples, we seem (at least intuitively) to

 have converged. But this article isn’t about how

 to get good convergence - it’s about baselines! In order

 to advance our understanding of the baseline, we will need a brief excursion

 into the world of game theory.

Game Theory and Missingness

---------------------------

 Integrated gradients is inspired by work

 from cooperative game theory, specifically the Aumann-Shapley value

 . In cooperative game theory,

 a non-atomic game is a construction used to model large-scale economic systems

 Aumann-Shapley values provide a theoretically grounded way to

 determine how much different groups of participants contribute to the system.

 In game theory, a notion of missingness is well-defined. Games are defined

 on coalitions - sets of participants - and for any specific coalition,

 a participant of the system can be in or out of that coalition. The fact

 that games can be evaluated on coalitions is the foundation of

 the Aumann-Shapley value. Intuitively, it computes how

 much value a group of participants adds to the game

 by computing how much the value of the game would increase

 if we added more of that group to any given coalition.

 Unfortunately, missingness is a more difficult notion when

 we are speaking about machine learning models. In order

 to evaluate how important the iiith feature is, we

 want to be able to compute how much the output of

 the network would increase if we successively increased

 the “presence” of the iiith feature. But what does this mean, exactly?

 In order to increase the presence of a feature, we would need to start

 with the feature being “missing” and have a way of interpolating

 between that missingness and its current, known value.

 Hopefully, this is sounding awfully familiar. Integrated gradients

 has a baseline input x’x’x’ for exactly this reason: to model a

 feature being absent. But how should you choose

 x’x’x’ in order to best represent this? It seems to be common practice

 to choose a baseline input x’x’x’ to be the vector of

 all zeros. But consider the following scenario: you’ve learned a model

 on a healthcare dataset, and one of the features is blood sugar level.

 The model has correctly learned that excessively low levels of blood sugar,

 which correspond to hypoglycemia, is dangerous. Does

 a blood sugar level of 000 seem like a good choice to represent missingness?

 The point here is that fixed feature values may have unintended meaning.

 The problem compounds further when you consider the difference from

 baseline term xi−x’ix\_i - x’\_ixi−x’i.

 To understand why our machine learning model thinks this patient

 is at high risk, you run integrated gradients on this data point with a

 because xi−x’i=0x\_i - x’\_i = 0xi−x’i=0. This is despite the fact that

 a blood sugar level of 000 would be fatal!

 We find similar problems when we move to the image domain.

 If you use a constant black image as a baseline, integrated gradients will

 not highlight black pixels as important even if black pixels make up

 authors in , and is in fact

 central to the definition of a baseline: we wouldn’t want integrated gradients

 to highlight missing features as important! But then how do we avoid

 giving zero importance to the baseline color?

 Mouse over the segmented image to choose a different color

 as a baseline input x’x’x’. Notice that pixels

 of the baseline color are not highlighted as important,

 even if they make up part of the main object in the image.

Alternative Baseline Choices

----------------------------

 It’s clear that any constant color baseline will have this problem.

 Are there any alternatives? In this section, we

 compare four alternative choices for a baseline in the image domain.

 Before proceeding, it’s important to note that this article isn’t

 the first article to point out the difficulty of choosing a baselines.

 Several articles, including the original paper, discuss and compare

 several notions of “missingness”, both in the

 context of integrated gradients and more generally

 .

 Nonetheless, choosing the right baseline remains a challenge. Here we will

 present several choices for baselines: some based on existing literature,

 others inspired by the problems discussed above. The figure at the end

 of the section visualizes the four baselines presented here.

### The Maximum Distance Baseline

 If we are worried about constant baselines that are blind to the baseline

 color, can we explicitly construct a baseline that doesn’t suffer from this

 problem? One obvious way to construct such a baseline is to take the

 farthest image in L1 distance from the current image such that the

 baseline is still in the valid pixel range. This baseline, which

 we will refer to as the maximum distance baseline (denoted

 *max dist.* in the figure below),

 avoids the difference from baseline issue directly.

### The Blurred Baseline

 The issue with the maximum distance baseline is that it doesn’t

 really represent *missingness*. It actually contains a lot of

 information about the original image, which means we are no longer

 explaining our prediction relative to a lack of information. To better

 preserve the notion of missingness, we take inspiration from

 . In their paper,

 Fong and Vedaldi use a blurred version of the image as a

 domain-specific way to represent missing information. This baseline

 is attractive because it captures the notion of missingness in images

 in a very human intuitive way. In the figure below, this baseline is

 denoted *blur*. The figure lets you play with the smoothing constant

 used to define the baseline.

### The Uniform Baseline

 One potential drawback with the blurred baseline is that it is biased

 to highlight high-frequency information. Pixels that are very similar

 to their neighbors may get less importance than pixels that are very

 different than their neighbors, because the baseline is defined as a weighted

 from both  and the original integrated

 gradients paper. Another way to define missingness is to simply sample a random

 uniform image in the valid pixel range and call that the baseline.

 We refer to this baseline as the *uniform* baseline in the figure below.

### The Gaussian Baseline

 Of course, the uniform distribution is not the only distribution we can

 touch on in the next section), Smilkov et al.

 variance σ\sigmaσ. We can use the same distribution as a baseline for

 range, which means that as σ\sigmaσ approaches ∞\infty∞, the gaussian

 baseline approaches the uniform baseline.

 Comparing alternative baseline choices. For the blur and gaussian

 baselines, you can vary the parameter σ\sigmaσ, which refers

 to the width of the smoothing kernel and the standard deviation of

 noise respectively.

Averaging Over Multiple Baselines

---------------------------------

 You may have nagging doubts about those last two baselines, and you

 would be right to have them. A randomly generated baseline

 can suffer from the same blindness problem that a constant image can. If

 we draw a uniform random image as a baseline, there is a small chance

 that a baseline pixel will be very close to its corresponding input pixel

 in value. Those pixels will not be highlighted as important. The resulting

 saliency map may have artifacts due to the randomly drawn baseline. Is there

 any way we can fix this problem?

 Perhaps the most natural way to do so is to average over multiple

 different baselines, as discussed in

 .

 Although doing so may not be particularly natural for constant color images

 (which colors do you choose to average over and why?), it is a

 very natural notion for baselines drawn from distributions. Simply

 draw more samples from the same distribution and average the

 importance scores from each sample.

### Assuming a Distribution

 At this point, it’s worth connecting the idea of averaging over multiple

 baselines back to the original definition of integrated gradients. When

 we average over multiple baselines from the same distribution DDD,

 we are attempting to use the distribution itself as our baseline.

 We use the distribution to define the notion of missingness:

 if we don’t know a pixel value, we don’t assume its value to be 0 - instead

 we assume that it has some underlying distribution DDD. Formally, given

 a baseline distribution DDD, we integrate over all possible baselines

 x’∈Dx’ \in Dx’∈D weighted by the density function pDp\_DpD:

 )}^{\text{integrated gradients

 with baseline } x’

 } \times \underbrace{p\_D(x’) dx’}\_{\text{…and weight by the density}} \bigg)

 In terms of missingness, assuming a distribution might intuitively feel

 like a more reasonable assumption to make than assuming a constant value.

 But this doesn’t quite solve the issue: instead of having to choose a baseline

 x’x’x’, now we have to choose a baseline distribution DDD. Have we simply

 postponed the problem? We will discuss one theoretically motivated

 way to choose DDD in an upcoming section, but before we do, we’ll take

 a brief aside to talk about how we compute the formula above in practice,

 and a connection to an existing method that arises as a result.

### Expectations, and Connections to SmoothGrad

 Now that we’ve introduced a second integral into our formula,

 we need to do a second discrete sum to approximate it, which

 requires an additional hyperparameter: the number of baselines to sample.

 In , Erion et al. make the

 observation that both integrals can be thought of as expectations:

 the first integral as an expectation over DDD, and the second integral

 as an expectation over the path between x’x’x’ and xxx. This formulation,

 called *expected gradients*, is defined formally as:

 {\text{Expectation over \(D\) and the path…}}

 \bigg[ \overbrace{(x\_i - x’\_i) \times

 \frac{\delta f(x’ + \alpha (x - x’))}{\delta x\_i}}^{\text{…of the

 importance of the } i\text{th pixel}} \bigg]

 Expected gradients and integrated gradients belong to a family of methods

 known as “path attribution methods” because they integrate gradients

 over one or more paths between two valid inputs.

 Both expected gradients and integrated gradients use straight-line paths,

 in more detail in the original paper.  To compute expected gradients in

 practice, we use the following formula:

 ϕ^iEG(f,x;D)=1k∑j=1k(xi−x’ij)×δf(x’j+αj(x−x’j))δxi

 \frac{\delta f(x’^j + \alpha^{j} (x - x’^j))}{\delta x\_i}

 ϕ^iEG(f,x;D)=k1j=1∑k(xi−x’ij)×δxiδf(x’j+αj(x−x’j))

 where x’jx’^jx’j is the jjjth sample from DDD and

 αj\alpha^jαj is the jjjth sample from the uniform distribution between

 0 and 1. Now suppose that we use the gaussian baseline with variance

 ϕ^iEG(f,x;N(x,σ2I))=1k∑j=1kϵσj×δf(x+(1−αj)ϵσj)δxi

 \hat{\phi}\_i^{EG}(f, x; N(x, \sigma^2 I))

 = \frac{1}{k} \sum\_{j=1}^k

 \epsilon\_{\sigma}^{j} \times

 \frac{\delta f(x + (1 - \alpha^j)\epsilon\_{\sigma}^{j})}{\delta x\_i}

 ϕ^iEG(f,x;N(x,σ2I))=k1j=1∑kϵσj×δxiδf(x+(1−αj)ϵσj)

 To see how we arrived

 at the above formula, first observe that

 x’∼N(x,σ2I)=x+ϵσx’−x=ϵσ

 \begin{aligned}

 x’ \sim N(x, \sigma^2 I) &= x + \epsilon\_{\sigma} \\

 x’- x &= \epsilon\_{\sigma} \\

 \end{aligned}

 x’∼N(x,σ2I)x’−x=x+ϵσ=ϵσ

 by definition of the gaussian baseline. Now we have:

 x’+α(x−x’)=x+ϵσ+α(x−(x+ϵσ))=x+(1−α)ϵσ

 \begin{aligned}

 x’ + \alpha(x - x’) &= \\

 x + \epsilon\_{\sigma} + \alpha(x - (x + \epsilon\_{\sigma})) &= \\

 x + (1 - \alpha)\epsilon\_{\sigma}

 \end{aligned}

 x’+α(x−x’)x+ϵσ+α(x−(x+ϵσ))x+(1−α)ϵσ==

 The above formula simply substitues the last line

 of each equation block back into the formula.

 .

 This looks awfully familiar to an existing method called SmoothGrad

 . If we use the (gradients ×\times× input image)

 variant of SmoothGrad  SmoothGrad is

 was a method designed to sharpen saliency maps and was meant to be run

 on top of an existing saliency method. The idea is simple:

 instead of running a saliency method once on an image, first

 add some gaussian noise to an image, then run the saliency method.

 Do this several times with different draws of gaussian noise, then

 is discussed in more detail in the original SmoothGrad paper.,

 then we have the following formula:

 ϕiSG(f,x;N(0ˉ,σ2I))=1k∑j=1k(x+ϵσj)×δf(x+ϵσj)δxi

 \phi\_i^{SG}(f, x; N(\bar{0}, \sigma^2 I))

 = \frac{1}{k} \sum\_{j=1}^k

 (x + \epsilon\_{\sigma}^{j}) \times

 \frac{\delta f(x + \epsilon\_{\sigma}^{j})}{\delta x\_i}

 ϕiSG(f,x;N(0ˉ,σ2I))=k1j=1∑k(x+ϵσj)×δxiδf(x+ϵσj)

 We can see that SmoothGrad and expected gradients with a

 gaussian baseline are quite similar, with two key differences:

 gradients multiplies by just ϵσ\epsilon\_{\sigma}ϵσ, and while expected

 gradients samples uniformly along the path, SmoothGrad always

 samples the endpoint α=0\alpha = 0α=0.

 Can this connection help us understand why SmoothGrad creates

 assuming that each of our pixel values is drawn from a

 gaussian *independently* of the other pixel values. But we know

 this is far from true: in images, there is a rich correlation structure

 between nearby pixels. Once your network knows the value of a pixel,

 it doesn’t really need to use its immediate neighbors because

 it’s likely that those immediate neighbors have very similar intensities.

 Assuming each pixel is drawn from an independent gaussian

 breaks this correlation structure. It means that expected gradients

 tabulates the importance of each pixel independently of

 the other pixel values. The generated saliency maps

 will be less noisy and better highlight the object of interest

 because we are no longer allowing the network to rely

 on only pixel in a group of correlated pixels. This may be

 why SmoothGrad is smooth: because it is implicitly assuming

 independence among pixels. In the figure below, you can compare

 integrated gradients with a single randomly drawn baseline

 to expected gradients sampled over a distribution. For

 the gaussian baseline, you can also toggle the SmoothGrad

 option to use the SmoothGrad formula above. For all figures,

 k=500k=500k=500.

 The difference between a single baseline and multiple

 baselines from the same distribution. Use the

 “Multi-Reference” button to toggle between the two. For the gaussian

 baseline, you can also toggle the “Smooth Grad” button

 to toggle between expected gradients and SmoothGrad

 with gradients \* inputs.

### Using the Training Distribution

 Is it really reasonable to assume independence among

 pixels while generating saliency maps? In supervised learning,

 we make the assumption that the data is drawn

 share a common, underlying distribution is what allows us to

 do supervised learning and make claims about generalizability. Given

 this assumption, we don’t need to

 model missingness using a gaussian or a uniform distribution:

 we can use DdataD\_{\text{data}}Ddata to model missingness directly.

 The only problem is that we do not have access to the underlying distribution.

 But because this is a supervised learning task, we do have access to many

 independent draws from the underlying distribution: the training data!

 We can simply use samples from the training data as random draws

 from DdataD\_{\text{data}}Ddata. This brings us to the variant

 of expected gradients used in ,

 which we again visualize in three parts:

 \frac{1}{k} \sum\_{j=1}^k

 \underbrace{(x\_i - x’^j\_i) \times

 \frac{\delta f(\text{ }

 \overbrace{x’^j + \alpha^{j} (x - x’^j)}^{\text{(1): Interpolated Image}}

 \text{ })}{\delta x\_i}}\_{\text{ (2): Gradients at Interpolation}}

 = \overbrace{\hat{\phi\_i}^{EG}(f, x, k; D\_{\text{data}})}

 ^{\text{(3): Cumulative Gradients up to }\alpha}

 A visual representation of expected gradients. Instead of taking contributions

 from a single path, expected gradients averages contributions from

 all paths defined by the underlying data distribution. Note that

 this figure only displays every 10th sample to avoid loading many images.

 In (4) we again plot the sum of the importance scores over pixels. As mentioned

 gradients, satisfy the completeness axiom. We can definitely see that

 completeness is harder to satisfy when we integrate over both a path

 and a distribution: that is, with the same number

 of samples, expected gradients doesn’t converge as quickly as

 integrated gradients does. Whether or not this is an acceptable price to

 pay to avoid color-blindness in attributions seems subjective.

Comparing Saliency Methods

--------------------------

 So we now have many different choices for a baseline. How do we choose

 which one we should use? The different choices of distributions and constant

 baselines have different theoretical motivations and practical concerns.

 Do we have any way of comparing the different baselines? In this section,

 we will touch on several different ideas about how to compare

 of all of the existing evaluation metrics, but is instead meant to

 emphasize that evaluating interpretability methods remains a difficult problem.

### The Dangers of Qualitative Assessment

 One naive way to evaluate our baselines is to look at the saliency maps

 they produce and see which ones best highlight the object in the image.

 reasonable results, as does using a gaussian baseline or the blurred baseline.

 But is visual inspection really a good way judge our baselines? For one thing,

 we’ve only presented four images from the test set here. We would need to

 conduct user studies on a much larger scale with more images from the test

 set to be confident in our results. But even with large-scale user studies,

 qualitative assessment of saliency maps has other drawbacks.

 When we rely on qualitative assessment, we are assuming that humans

 know what an “accurate” saliency map is. When we look at saliency maps

 on data like ImageNet, we often check whether or not the saliency map

 highlights the object that we see as representing the true class in the image.

 We make an assumption between the data and the label, and then further assume

 that a good saliency map should reflect that assumption. But doing so

 has no real justification. Consider the figure below, which compares

 two saliency methods on a network that gets above 99% accuracy

 on (an altered version of) MNIST.

 The first saliency method is just an edge detector plus gaussian smoothing,

 while the second saliency method is expected gradients using the training

 data as a distribution. Edge detection better reflects what we humans

 think is the relationship between the image and the label.

Original Image:Edge Detection:Expected Gradients:

 Qualitative assessment can be dangerous because we rely

 on our human knowledge of the relationship between

 the data and the labels, and then we assume

 that an accurate model has learned that very relationship.

 Unfortunately, the edge detection method here does not highlight

 what the network has learned. This dataset is a variant of

 decoy MNIST, in which the top left corner of the image has

 been altered to directly encode the image’s class

 . That is, the intensity

 of the top left corner of each image has been altered to

 be 255×y9255 \times \frac{y}{9} 255×9y where yyy is the class

 the image belongs to. We can verify by removing this

 patch in the test set that the network heavily relies on it to make

 predictions, which is what the expected gradients saliency maps show.

 This is obviously a contrived example. Nonetheless, the fact that

 visual assessment is not necessarily a useful way to evaluate

 saliency maps and attribution methods has been extensively

 discussed in recent literature, with many proposed qualitative

 tests as replacements

 .

 At the heart of the issue is that we don’t have ground truth explanations:

 we are trying to evaluate which methods best explain our network without

 actually knowing what our networks are doing.

### Top K Ablation Tests

 One simple way to evaluate the importance scores that

 expected/integrated gradients produces is to see whether

 ablating the top k features as ranked by their importance

 decreases the predicted output logit. In the figure below, we

 ablate either by mean-imputation or by replacing each pixel

 for 1000 different correctly classified test-set images using each

 of the baselines proposed above

 For the blur baseline and the blur

 ablation test, we use σ=20\sigma = 20σ=20.

 For the gaussian baseline, we use σ=1\sigma = 1σ=1. These choices

 are somewhat arbitrary - a more comprehensive evaluation

 would compare across many values of σ\sigmaσ.

 . As a

 control, we also include ranking features randomly

 (*Random Noise* in the plot).

 We plot, as a fraction of the original logit, the output logit

 of the network at the true class. That is, suppose the original

 image is a goldfinch and the network predicts the goldfinch class correctly

 with 95% confidence. If the confidence of class goldfinch drops

 to 60% after ablating the top 10% of pixels as ranked by

 feature importance, then we plot a curve that goes through

 that best highlights which pixels the network

 should exhibit the fastest drop in logit magnitude, because

 it highlights the pixels that most increase the confidence of the network.

 That is, the lower the curve, the better the baseline.

### Mass Center Ablation Tests

 One problem with ablating the top k features in an image

 is related to an issue we already brought up: feature correlation.

 No matter how we ablate a pixel, that pixel’s neighbors

 provide a lot of information about the pixel’s original value.

 With this in mind, one could argue that progressively ablating

 pixels one by one is a rather meaningless thing to do. Can

 we instead perform ablations with feature correlation in mind?

 One straightforward way to do this is simply compute the

 center of mass

 of the saliency map, and ablate a boxed region centered on

 the center of mass. This tests whether or not the saliency map

 is generally highlighting an important region in the image. We plot

 replacing the boxed region around the saliency map using mean-imputation

 and blurring below as well (*Mean Center* and *Blur Center*, respectively).

 As a control, we compare against a saliency map generated from random gaussian

 noise (*Random Noise* in the plot).

 A variety of ablation tests on a variety of baselines.

 Using the training distribution and using the uniform distribution

 outperform most other methods on the top k ablation tests. The

 blur baseline inspired by

 does equally well on the blur top-k test. All methods

 perform similarly on the mass center ablation tests. Mouse

 over the legend to highlight a single curve.

 The ablation tests seem to indicate some interesting trends.

 All methods do similarly on the mass center ablation tests, and

 only slightly better than random noise. This may be because the

 object of interest generally lies in the center of the image - it

 isn’t hard for random noise to be centered at the image. In contrast,

 using the training data or a uniform distribution seems to do quite well

 on the top-k ablation tests. Interestingly, the blur baseline

 inspired by  also

 does quite well on the top k baseline tests, especially when

 we ablate pixels by blurring them! Would the uniform

 baseline do better if you ablate the image with uniform random noise?

 by progressively replacing it with a different image. We leave

 these experiments as future work, as there is a more pressing question

 we need to discuss.

### The Pitfalls of Ablation Tests

 Constant baselines tend to not need as many samples

 comparing not only across baselines but also across number of samples drawn,

 and for the blur and gaussian baselines, the parameter σ\sigmaσ.

 As mentioned above, we have defined many notions of missingness other than

 mean-imputation or blurring: more extensive comparisons would also compare

 all of our baselines across all of the corresponding notions of missing data.

 But even with all of these added comparisons, do ablation

 tests really provide a well-founded metric to judge attribution methods?

 The authors of  argue

 against ablation tests. They point out that once we artificially ablate

 pixels an image, we have created inputs that do not come from

 the original data distribution. Our trained model has never seen such

 inputs. Why should we expect to extract any reasonable information

 from evaluating our model on them?

 On the other hand, integrated gradients and expected gradients

 rely on presenting interpolated images to your model, and unless

 you make some strange convexity assumption, those interpolated images

 don’t belong to the original training distribution either.

 In general, whether or not users should present

 is a subject of ongoing debate

 . Nonetheless,

 the point raised in  is still an

 important one: “it is unclear whether the degradation in model

 performance comes from the distribution shift or because the

 features that were removed are truly informative.”

### Alternative Evaluation Metrics

 So what about other evaluation metrics proposed in recent literature? In

 , Hooker et al. propose a variant of

 an ablation test where we first ablate pixels in the training and

 test sets. Then, we re-train a model on the ablated data and measure

 by how much the test-set performance degrades. This approach has the advantage

 of better capturing whether or not the saliency method

 highlights the pixels that are most important for predicting the output class.

 Unfortunately, it has the drawback of needing to re-train the model several

 times. This metric may also get confused by feature correlation.

 Consider the following scenario: our dataset has two features

 that are highly correlated. We train a model which learns to only

 use the first feature, and completely ignore the second feature.

 A feature attribution method might accurately reveal what the model is doing:

 re-train the model and get similar performance because similar information

 is stored in the second feature. We might conclude that our feature

 attribution method is lousy - is it? This problem fits into a larger discussion

 about whether or not your attribution method

 should be “true to the model” or “true to the data”

 which has been discussed in several recent articles

 .

 In , the authors propose several

 sanity checks that saliency methods should pass. One is the “Model Parameter

 Randomization Test”. Essentially, it states that a feature attribution

 method should produce different attributions when evaluated on a trained

 model (assumedly a trained model that performs well) and a randomly initialized

 model. This metric is intuitive: if a feature attribution method produces

 similar attributions for random and trained models, is the feature

 attribution really using information from the model? It might just

 be relying entirely on information from the input image.

 But consider the following figure, which is another (modified) version

 of MNIST. We’ve generated expected gradients attributions using the training

 distribution as a baseline for two different networks. One of the networks

 is a trained model that gets over 99% accuracy on the test set. The other

 Should we now conclude that expected gradients is an unreliable method?

Original Image:Network 1 Saliency:Network 2 Saliency:

 A comparison of two network’s saliency maps using expected gradients. One

 network has randomly initialized weights, the other gets >99% accuracy

 on the test set.

 you would run these kinds of saliency method sanity checks on un-modified data.

 But the truth is, even for natural images, we don’t actually

 know what an accurate model’s saliency maps should look like.

 Different architectures trained on ImageNet can all get good performance

 and have very different saliency maps. Can we really say that

 trained models should have saliency maps that don’t look like

 saliency maps generated on randomly initialized models? That isn’t

 to say that the model randomization test doesn’t have merit: it

 does reveal interesting things about what saliency methods are are doing.

 It just doesn’t tell the whole story.

 .

 Each proposed metric comes with their various pros and cons.

 we don’t know what our model is doing and have no ground truth to compare

 against.

Conclusion

----------

 So what should be done? We have many baselines and

 no conclusion about which one is the “best.” Although

 we don’t provide extensive quantitative results

 comparing each baseline, we do provide a foundation

 for understanding them further. At the heart of

 each baseline is an assumption about missingness

 in our model and the distribution of our data. In this article,

 we shed light on some of those assumptions, and their impact

 on the corresponding path attribution. We lay

 groundwork for future discussion about baselines in the

 context of path attributions, and more generally about

 the relationship between representations of missingness

 and how we explain machine learning models.

 A side-by-side comparison of integrated gradients

 using a black baseline

 and expected gradients using the training data

 as a baseline.

Related Methods

---------------

 This work focuses on a specific interpretability method: integrated gradients

 and its extension, expected gradients. We refer to these

 methods as path attribution methods because they integrate

 importances over a path. However, path attribution methods

 represent only a tiny fraction of existing interpretability methods. We focus

 on them here both because they are amenable to interesting visualizations,

 and because they provide a springboard for talking about missingness.

 We briefly cited several other methods at the beginning of this article.

 Many of those methods use some notion of baseline and have contributed to

 the discussion surrounding baseline choices.

 In , Fong and Vedaldi propose

 a model-agnostic method to explain neural networks that is based

 on learning the minimal deletion to an image that changes the model

 prediction. In section 4, their work contains an extended discussion on

 how to represent deletions: that is, how to represent missing pixels. They

 argue that one natural way to delete pixels in an image is to blur them.

 This discussion inspired the blurred baseline that we presented in our article.

 They also discuss how noise can be used to represent missingness, which

 was part of the inspiration for our uniform and gaussian noise baselines.

 In , Shrikumar et al.

 propose a feature attribution method called deepLIFT. It assigns

 importance scores to features by propagating scores from the output

 of the model back to the input. Similar to integrated gradients,

 deepLIFT also defines importance scores relative to a baseline, which

 they call the “reference”. Their paper has an extended discussion on

 why explaining relative to a baseline is meaningful. They also discuss

 a few different baselines, including “using a blurred version of the original

 image”.

 The list of other related methods that we didn’t discuss

 in this article goes on: SHAP and DeepSHAP

 ,

 layer-wise relevance propagation ,

 LIME ,

 RISE  and

 Grad-CAM

 among others. Many methods for explaining machine learning models

 define some notion of baseline or missingness,

 because missingness and explanations are closely related. When we explain

 a model, we often want to know which features, when missing, would most

 change model output. But in order to do so, we need to define

 what missing means because most machine learning models cannot

 handle arbitrary patterns of missing inputs. This article

 does not discuss all of the nuances presented alongside

 each existing method, but it is important to note that these methods were

 points of inspiration for a larger discussion about missingness.

Axiomatic attribution for deep networks

Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy

Ensembling convolutional and long short-term memory networks for electrocardiogram arrhythmia detection

Inception-v4, inception-resnet and the impact of residual connections on learning

Imagenet: A large-scale hierarchical image database

Tensorflow-slim image classification model library

The Building Blocks of Interpretability

Feature Visualization

Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

Visualizing and understanding convolutional networks

Visualizing higher-layer features of a deep network

Understanding deep image representations by inverting them

Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks

"Why Should I Trust You?": Explaining the Predictions of Any Classifier

A unified approach to interpreting model predictions

Layer-wise relevance propagation for neural networks with local renormalization layers

Learning important features through propagating activation differences

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Interpretable explanations of black boxes by meaningful perturbation

Learning deep features for discriminative localization

Grad-cam: Visual explanations from deep networks via gradient-based localization

Smoothgrad: removing noise by adding noise

Rise: Randomized input sampling for explanation of black-box models

Understanding the difficulty of training deep feedforward neural networks

Gradients of counterfactuals

Values of non-atomic games

A note about: Local explanation methods for deep neural networks lack sensitivity to parameter values

The (Un)reliability of saliency methods

Towards better understanding of gradient-based attribution methods for Deep Neural Networks

Learning Explainable Models Using Attribution Priors

XRAI: Better Attributions Through Regions

Right for the right reasons: Training differentiable models by constraining their explanations

A Benchmark for Interpretability Methods in Deep Neural Networks

On the (In)fidelity and Sensitivity for Explanations

Sanity Checks for Saliency Maps

Benchmarking Attribution Methods with Relative Feature Importance

Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation

Interpretation of neural networks is fragile

The many Shapley values for model explanation

Feature relevance quantification in explainable AI: A causality problem

Explaining Models by Propagating Shapley Values of Local Components

Visualizing the Impact of Feature Attribution Baselines.html