Feature-wise transformations

Vincent Dumoulin
Ethan Perez
Nathan Schucher
Florian Strub
Harm de Vries
Aaron Courville
Yoshua Bengio

 Many real-world problems require integrating multiple sources of information.

 Sometimes these problems involve multiple, distinct modalities of

 information — vision, language, audio, etc. — as is required

 to understand a scene in a movie or answer a question about an image.

 Other times, these problems involve multiple sources of the same

 kind of input, i.e. when summarizing several documents or drawing one

 image in the style of another.

 When approaching such problems, it often makes sense to process one source

 of information *in the context of* another; for instance, in the

 right example above, one can extract meaning from the image in the context

 of the question. In machine learning, we often refer to this context-based

 processing as *conditioning*: the computation carried out by a model

 is conditioned or *modulated* by information extracted from an

 auxiliary input.

 Finding an effective way to condition on or fuse sources of information

 is an open research problem, and

 in this article, we concentrate on a specific family of approaches we call

 *feature-wise transformations*.

 We will examine the use of feature-wise transformations in many neural network

 architectures to solve a surprisingly large and diverse set of problems;

 their success, we will argue, is due to being flexible enough to learn an

 effective representation of the conditioning input in varied settings.

 In the language of multi-task learning, where the conditioning signal is

 taken to be a task description, feature-wise transformations

 learn a task representation which allows them to capture and leverage the

 relationship between multiple sources of information, even in remarkably

 different problem settings.

---

Feature-wise transformations

----------------------------

 To motivate feature-wise transformations, we start with a basic example,

 where the two inputs are images and category labels, respectively. For the

 purpose of this example, we are interested in building a generative model of

 images of various classes (puppy, boat, airplane, etc.). The model takes as

 input a class and a source of random noise (e.g., a vector sampled from a

 normal distribution) and outputs an image sample for the requested class.

 Our first instinct might be to build a separate model for each

 class. For a small number of classes this approach is not too bad a solution,

 but for thousands of classes, we quickly run into scaling issues, as the number

 of parameters to store and train grows with the number of classes.

 We are also missing out on the opportunity to leverage commonalities between

 classes; for instance, different types of dogs (puppy, terrier, dalmatian,

 etc.) share visual traits and are likely to share computation when

 mapping from the abstract noise vector to the output image.

 Now let's imagine that, in addition to the various classes, we also need to

 model attributes like size or color. In this case, we can't

 reasonably expect to train a separate network for *each* possible

 conditioning combination! Let's examine a few simple options.

 A quick fix would be to concatenate a representation of the conditioning

 information to the noise vector and treat the result as the model's input.

 This solution is quite parameter-efficient, as we only need to increase

 Maybe this assumption is correct, or maybe it's not; perhaps the

 model does not need to incorporate the conditioning information until late

 into the generation process (e.g., right before generating the final pixel

 carry this information around unaltered for many layers.

 Because this operation is cheap, we might as well avoid making any such

 assumptions and concatenate the conditioning representation to the input of

 *all* layers in the network. Let's call this approach

 *concatenation-based conditioning*.

 Another efficient way to integrate conditioning information into the network

 is via *conditional biasing*, namely, by adding a *bias* to

 the hidden layers based on the conditioning representation.

 Interestingly, conditional biasing can be thought of as another way to

 implement concatenation-based conditioning. Consider a fully-connected

 linear layer applied to the concatenation of an input

 x\mathbf{x}x and a conditioning representation

 z\mathbf{z}z:

 The same argument applies to convolutional networks, provided we ignore

 the border effects due to zero-padding.

 Yet another efficient way to integrate class information into the network is

 via *conditional scaling*, i.e., scaling hidden layers

 based on the conditioning representation.

 A special instance of conditional scaling is feature-wise sigmoidal gating:

 we scale each feature by a value between 000 and

 111 (enforced by applying the logistic function), as a

 function of the conditioning representation. Intuitively, this gating allows

 the conditioning information to select which features are passed forward

 and which are zeroed out.

 Given that both additive and multiplicative interactions seem natural and

 intuitive, which approach should we pick? One argument in favor of

 *multiplicative* interactions is that they are useful in learning

 relationships between inputs, as these interactions naturally identify

 "matches": multiplying elements that agree in sign yields larger values than

 multiplying elements that disagree. This property is why dot products are

 often used to determine how similar two vectors are.

 Multiplicative interactions alone have had a history of success in various

 domains — see [Bibliographic Notes](#bibliographic-notes).

 One argument in favor of *additive* interactions is that they are

 more natural for applications that are less strongly dependent on the

 joint values of two inputs, like feature aggregation or feature detection

 (i.e., checking if a feature is present in either of two inputs).

 In the spirit of making as few assumptions about the problem as possible,

 we may as well combine *both* into a

 conditional *affine transformation*.

 An affine transformation is a transformation of the form

 y=m∗x+by = m \* x + by=m∗x+b.

 All methods outlined above share the common trait that they act at the

 *feature* level; in other words, they leverage *feature-wise*

 interactions between the conditioning representation and the conditioned

 network. It is certainly possible to use more complex interactions,

 but feature-wise interactions often strike a happy compromise between

 effectiveness and efficiency: the number of scaling and/or shifting

 coefficients to predict scales linearly with the number of features in the

 network. Also, in practice, feature-wise transformations (often compounded

 across multiple layers) frequently have enough capacity to model complex

 phenomenon in various settings.

 Lastly, these transformations only enforce a limited inductive bias and

 remain domain-agnostic. This quality can be a downside, as some problems may

 be easier to solve with a stronger inductive bias. However, it is this

 characteristic which also enables these transformations to be so widely

 effective across problem domains, as we will later review.

### Nomenclature

 To continue the discussion on feature-wise transformations we need to

 abstract away the distinction between multiplicative and additive

 interactions. Without losing generality, let's focus on feature-wise affine

 transformations, and let's adopt the nomenclature of Perez et al.

 , which formalizes conditional affine

 transformations under the acronym *FiLM*, for Feature-wise Linear

 Modulation.

 Strictly speaking, *linear* is a misnomer, as we allow biasing, but

 we hope the more rigorous-minded reader will forgive us for the sake of a

 better-sounding acronym.

 We say that a neural network is modulated using FiLM, or *FiLM-ed*,

 after inserting *FiLM layers* into its architecture. These layers are

 parametrized by some form of conditioning information, and the mapping from

 conditioning information to FiLM parameters (i.e., the shifting and scaling

 coefficients) is called the *FiLM generator*.

 In other words, the FiLM generator predicts the parameters of the FiLM

 layers based on some auxiliary input.

 Note that the FiLM parameters are parameters in one network but predictions

 from another network, so they aren't learnable parameters with fixed

 weights as in the fully traditional sense.

 For simplicity, you can assume that the FiLM generator outputs the

 concatenation of all FiLM parameters for the network architecture.

 As the name implies, a FiLM layer applies a feature-wise affine

 transformation to its input. By *feature-wise*, we mean that scaling

 and shifting are applied element-wise, or in the case of convolutional

 networks, feature map -wise.

 To expand a little more on the convolutional case, feature maps can be

 thought of as the same feature detector being evaluated at different

 spatial locations, in which case it makes sense to apply the same affine

 transformation to all spatial locations.

 In other words, assuming x\mathbf{x}x is a FiLM layer's

 input, z\mathbf{z}z is a conditioning input, and

 γ\gammaγ and βetaβ are

 z\mathbf{z}z-dependent scaling and shifting vectors,

 FiLM(x)=γ(z)⊙x+β(z).

 	extrm{FiLM}(\mathbf{x}) = \gamma(\mathbf{z}) \odot \mathbf{x}

 + eta(\mathbf{z}).

 FiLM(x)=γ(z)⊙x+β(z).

 You can interact with the following fully-connected and convolutional FiLM

 layers to get an intuition of the sort of modulation they allow:

 In addition to being a good abstraction of conditional feature-wise

 transformations, the FiLM nomenclature lends itself well to the notion of a

 *task representation*. From the perspective of multi-task learning,

 we can view the conditioning signal as the task description. More

 specifically, we can view the concatenation of all FiLM scaling and shifting

 coefficients as both an instruction on *how to modulate* the

 conditioned network and a *representation* of the task at hand. We

 will explore and illustrate this idea later on.

---

Feature-wise transformations in the literature

----------------------------------------------

 Feature-wise transformations find their way into methods applied to many

 problem settings, but because of their simplicity, their effectiveness is

 seldom highlighted in lieu of other novel research contributions. Below are

 a few notable examples of feature-wise transformations in the literature,

 grouped by application domain. The diversity of these applications

 underscores the flexible, general-purpose ability of feature-wise

 interactions to learn effective task representations.

expand all

Visual question-answering+

 Perez et al.  use

 FiLM layers to build a visual reasoning model

 trained on the CLEVR dataset  to

 answer multi-step, compositional questions about synthetic images.

 The model's linguistic pipeline is a FiLM generator which

 extracts a question representation that is linearly mapped to

 FiLM parameter values. Using these values, FiLM layers inserted within each

 residual block condition the visual pipeline. The model is trained

 end-to-end on image-question-answer triples. Strub et al.

  later on improved on the model by

 using an attention mechanism to alternate between attending to the language

 input and generating FiLM parameters layer by layer. This approach was

 better able to scale to settings with longer input sequences such as

 dialogue and was evaluated on the GuessWhat?!

 and ReferIt  datasets.

 de Vries et al.  leverage FiLM

 to condition a pre-trained network. Their model's linguistic pipeline

 modulates the visual pipeline via conditional batch normalization,

 real-world images on the GuessWhat?!

 and VQAv1  datasets.

 The visual pipeline consists of a pre-trained residual network that is

 fixed throughout training. The linguistic pipeline manipulates the visual

 pipeline by perturbing the residual network's batch normalization

 parameters, which re-scale and re-shift feature maps after activations

 have been normalized to have zero mean and unit variance. As hinted

 earlier, conditional batch normalization can be viewed as an instance of

 FiLM where the post-normalization feature-wise affine transformation is

 replaced with a FiLM layer.

Style transfer+

 Dumoulin et al.  use

 feature-wise affine transformations — in the form of conditional

 instance normalization layers — to condition a style transfer

 network on a chosen style image. Like conditional batch normalization

 discussed in the previous subsection,

 conditional instance normalization can be seen as an instance of FiLM

 where a FiLM layer replaces the post-normalization feature-wise affine

 instance normalization parameters, and it applies normalization with these

 style-specific parameters.

 Dumoulin et al.  use a simple

 embedding lookup to produce instance normalization parameters, while

 Ghiasi et al.  further

 introduce a *style prediction network*, trained jointly with the

 style transfer network to predict the conditioning parameters directly from

 a given style image. In this article we opt to use the FiLM nomenclature

 because it is decoupled from normalization operations, but the FiLM

 layers used by Perez et al.  were

 themselves heavily inspired by the conditional normalization layers used

 by Dumoulin et al. .

 Yang et al.  use a related

 architecture for video object segmentation — the task of segmenting a

 particular object throughout a video given that object's segmentation in the

 first frame. Their model conditions an image segmentation network over a

 video frame on the provided first frame segmentation using feature-wise

 scaling factors, as well as on the previous frame using position-wise

 biases.

 So far, the models we covered have two sub-networks: a primary

 network in which feature-wise transformations are applied and a secondary

 network which outputs parameters for these transformations. However, this

 distinction between *FiLM-ed network* and *FiLM generator*

 is not strictly necessary. As an example, Huang and Belongie

  propose an alternative

 style transfer network that uses adaptive instance normalization layers,

 which compute normalization parameters using a simple heuristic.

 Adaptive instance normalization can be interpreted as inserting a FiLM

 layer midway through the model. However, rather than relying

 on a secondary network to predict the FiLM parameters from the style

 image, the main network itself is used to extract the style features

 used to compute FiLM parameters. Therefore, the model can be seen as

 *both* the FiLM-ed network and the FiLM generator.

Image recognition+

 neural network's activations *themselves* as conditioning

 information. This idea gives rise to

 *self-conditioned* models.

 Highway Networks  are a prime

 example of applying this self-conditioning principle. They take inspiration

 from the LSTMs'  heavy use of

 feature-wise sigmoidal gating in their input, forget, and output gates to

 regulate information flow:

inputsub-networksigmoidal layer1 - xoutput

 The ImageNet 2017 winning model  also

 employs feature-wise sigmoidal gating in a self-conditioning manner, as a

 way to "recalibrate" a layer's activations conditioned on themselves.

Natural language processing+

 For statistical language modeling (i.e., predicting the next word

 in a sentence), the LSTM

 constitutes a popular class of recurrent network architectures. The LSTM

 relies heavily on feature-wise sigmoidal gating to control the

 information flow in and out of the memory or context cell

 c\mathbf{c}c, based on the hidden states

 h\mathbf{h}h and inputs x\mathbf{x}x at

 every timestep t\mathbf{t}t.

ct-1tanhcthtsigmoidsigmoidtanhsigmoidlinearlinearlinearlinearht-1xtconcatenate

 Also in the domain of language modeling, Dauphin et al.  use sigmoidal

 gating in their proposed *gated linear unit*, which uses half of the

 input features to apply feature-wise sigmoidal gating to the other half.

 Gehring et al.  adopt this

 architectural feature, introducing a fast, parallelizable model for machine

 translation in the form of a fully convolutional network.

 The Gated-Attention Reader

 uses feature-wise scaling, extracting information

 from text by conditioning a document-reading network on a query. Its

 architecture consists of multiple Gated-Attention modules, which involve

 element-wise multiplications between document representation tokens and

 token-specific query representations extracted via soft attention on the

 query representation tokens.

Reinforcement learning+

 The Gated-Attention architecture

 uses feature-wise sigmoidal gating to fuse linguistic and visual

 information in an agent trained to follow simple "go-to" language

 instructions in the VizDoom  3D

 environment.

 Bahdanau et al. use FiLM

 layers to condition Neural Module Network

 and LSTM -based policies to follow

 basic, compositional language instructions (arranging objects and going

 to particular locations) in a 2D grid world. They train this policy

 in an adversarial manner using rewards from another FiLM-based network,

 trained to discriminate between ground-truth examples of achieved

 instruction states and failed policy trajectories states.

 Outside instruction-following, Kirkpatrick et al.

  also use

 game-specific scaling and biasing to condition a shared policy network

 trained to play 10 different Atari games.

Generative modeling+

 The conditional variant of DCGAN ,

 a well-recognized network architecture for generative adversarial networks

 , uses concatenation-based

 conditioning. The class label is broadcasted as a feature map and then

 concatenated to the input of convolutional and transposed convolutional

 layers in the discriminator and generator networks.

 For convolutional layers, concatenation-based conditioning requires the

 network to learn redundant convolutional parameters to interpret each

 constant, conditioning feature map; as a result, directly applying a

 conditional bias is more parameter efficient, but the two approaches are

 still mathematically equivalent.

 PixelCNN

 and WaveNet  — two recent

 advances in autoregressive, generative modeling of images and audio,

 respectively — use conditional biasing. The simplest form of

 conditioning in PixelCNN adds feature-wise biases to all convolutional layer

 outputs. In FiLM parlance, this operation is equivalent to inserting FiLM

 layers after each convolutional layer and setting the scaling coefficients

 to a constant value of 1.

 The authors also describe a location-dependent biasing scheme which

 cannot be expressed in terms of FiLM layers due to the absence of the

 feature-wise property.

 WaveNet describes two ways in which conditional biasing allows external

 information to modulate the audio or speech generation process based on

 conditioning input:

1. **Global conditioning** applies the same conditional bias

 to the whole generated sequence and is used e.g. to condition on speaker

 identity.

2. **Local conditioning** applies a conditional bias which

 varies across time steps of the generated sequence and is used e.g. to

 let linguistic features in a text-to-speech model influence which sounds

 are produced.

 As in PixelCNN, conditioning in WaveNet can be viewed as inserting FiLM

 layers after each convolutional layer. The main difference lies in how

 the FiLM-generating network is defined: global conditioning

 expresses the FiLM-generating network as an embedding lookup which is

 broadcasted to the whole time series, whereas local conditioning expresses

 it as a mapping from an input sequence of conditioning information to an

 output sequence of FiLM parameters.

Speech recognition+

 Kim et al. modulate a deep

 bidirectional LSTM using a form

 of conditional normalization. As discussed in the

 *Visual question-answering* and *Style transfer* subsections,

 conditional normalization can be seen as an instance of FiLM where

 the post-normalization feature-wise affine transformation is replaced

 with a FiLM layer.

 The key difference here is that the conditioning signal does not come from

 an external source but rather from utterance

 summarization feature vectors extracted in each layer to adapt the model.

Domain adaptation and few-shot learning+

 For domain adaptation, Li et al.

 find it effective to update the per-channel batch normalization

 statistics (mean and variance) of a network trained on one domain with that

 network's statistics in a new, target domain. As discussed in the

 *Style transfer* subsection, this operation is akin to using the network as

 both the FiLM generator and the FiLM-ed network. Notably, this approach,

 along with Adaptive Instance Normalization, has the particular advantage of

 not requiring any extra trainable parameters.

 For few-shot learning, Oreshkin et al.

  explore the use of FiLM layers to

 provide more robustness to variations in the input distribution across

 few-shot learning episodes. The training set for a given episode is used to

 produce FiLM parameters which modulate the feature extractor used in a

 Prototypical Networks

 meta-training procedure.

---

Related ideas

-------------

 Aside from methods which make direct use of feature-wise transformations,

 the FiLM framework connects more broadly with the following methods and

 concepts.

expand all

Zero-shot learning+

 The idea of learning a task representation shares a strong connection with

 zero-shot learning approaches. In zero-shot learning, semantic task

 embeddings may be learned from external information and then leveraged to

 make predictions about classes without training examples. For instance, to

 generalize to unseen object categories for image classification, one may

 construct semantic task embeddings from text-only descriptions and exploit

 objects' text-based relationships to make predictions for unseen image

 categories. Frome et al. , Socher et

 al. , and Norouzi et al.

  are a few notable exemplars

 of this idea.

HyperNetworks+

 The notion of a secondary network predicting the parameters of a primary

 (e.g., a recurrent neural network layer). From this perspective, the FiLM

 generator is a specialized HyperNetwork that predicts the FiLM parameters of

 the FiLM-ed network. The main distinction between the two resides in the

 number and specificity of predicted parameters: FiLM requires predicting far

 fewer parameters than Hypernetworks, but also has less modulation potential.

 The ideal trade-off between a conditioning mechanism's capacity,

 regularization, and computational complexity is still an ongoing area of

 investigation, and many proposed approaches lie on the spectrum between FiLM

 and HyperNetworks (see [Bibliographic Notes](#bibliographic-notes)).

Attention+

 Some parallels can be drawn between attention and FiLM, but the two operate

 in different ways which are important to disambiguate.

 This difference stems from distinct intuitions underlying attention and

 FiLM: the former assumes that specific spatial locations or time steps

 contain the most useful information, whereas the latter assumes that

 specific features or feature maps contain the most useful information.

Bilinear transformations+

 With a little bit of stretching, FiLM can be seen as a special case of a

 bilinear transformation

  with low-rank weight

 matrices. A bilinear transformation defines the relationship between two

 inputs x\mathbf{x}x and z\mathbf{z}z and the

 kthk^{th}kth output feature yky\_kyk as

 yk=xTWkz.

 y\_k = \mathbf{x}^T W\_k \mathbf{z}.

 yk=xTWkz.

 Note that for each output feature yky\_kyk we have a separate

 matrix WkW\_kWk, so the full set of weights forms a

 multi-dimensional array.

 If we view z\mathbf{z}z as the concatenation of the scaling

 and shifting vectors γ\gammaγ and βetaβ and

 if we augment the input x\mathbf{x}x with a 1-valued feature,

 As is commonly done to turn a linear transformation into an affine

 transformation.

 we can represent FiLM using a bilinear transformation by zeroing out the

 appropriate weight matrix entries:

 For some applications of bilinear transformations,

 see the [Bibliographic Notes](#bibliographic-notes).

---

Properties of the learned task representation

---------------------------------------------

 As hinted earlier, in adopting the FiLM perspective we implicitly introduce

 a notion of *task representation*: each task — be it a question

 about an image or a painting style to imitate — elicits a different

 set of FiLM parameters via the FiLM generator which can be understood as its

 representation in terms of how to modulate the FiLM-ed network. To help

 better understand the properties of this representation, let's focus on two

 FiLM-ed models used in fairly different problem settings:

* The visual reasoning model of Perez et al.

 , which uses FiLM

 to modulate a visual processing pipeline based off an input question.

* The artistic style transfer model of Ghiasi et al.

 , which uses FiLM to modulate a

 feed-forward style transfer network based off an input style image.

 As a starting point, can we discern any pattern in the FiLM parameters as a

 function of the task description? One way to visualize the FiLM parameter

 space is to plot γ\gammaγ against βetaβ,

 with each point corresponding to a specific task description and a specific

 feature map. If we color-code each point according to the feature map it

 belongs to we observe the following:

 The plots above allow us to make several interesting observations. First,

 FiLM parameters cluster by feature map in parameter space, and the cluster

 locations are not uniform across feature maps. The orientation of these

 clusters is also not uniform across feature maps: the main axis of variation

 can be γ\gammaγ-aligned, βetaβ-aligned, or

 diagonal at varying angles. These findings suggest that the affine

 transformation in FiLM layers is not modulated in a single, consistent way,

 i.e., using γ\gammaγ only, βetaβ only, or

 γ\gammaγ and βetaβ together in some specific

 way. Maybe this is due to the affine transformation being overspecified, or

 maybe this shows that FiLM layers can be used to perform modulation

 operations in several distinct ways.

 Nevertheless, the fact that these parameter clusters are often somewhat

 "dense" may help explain why the style transfer model of Ghiasi et al.

  is able to perform style

 interpolations: any convex combination of FiLM parameters is likely to

 correspond to a meaningful parametrization of the FiLM-ed network.

Style 1Style 2 w InterpolationContent Image

 To some extent, the notion of interpolating between tasks using FiLM

 parameters can be applied even in the visual question-answering setting.

 Using the model trained in Perez et al. ,

 we interpolated between the model's FiLM parameters for two pairs of CLEVR

 questions. Here we visualize the input locations responsible for

 The network seems to be softly switching where in the image it is looking,

 based on the task description. It is quite interesting that these semantically

 meaningful interpolation behaviors emerge, as the network has not been

 trained to act this way.

 Despite these similarities across problem settings, we also observe

 qualitative differences in the way in which FiLM parameters cluster as a

 function of the task description. Unlike the style transfer model, the

 visual reasoning model sometimes exhibits several FiLM parameter

 sub-clusters for a given feature map.

 At the very least, this may indicate that FiLM learns to operate in ways

 that are problem-specific, and that we should not expect to find a unified

 and problem-independent explanation for FiLM's success in modulating FiLM-ed

 networks. Perhaps the compositional or discrete nature of visual reasoning

 requires the model to implement several well-defined modes of operation

 which are less necessary for style transfer.

 Focusing on individual feature maps which exhibit sub-clusters, we can try

 to infer how questions regroup by color-coding the scatter plots by question

 type.

 Sometimes a clear pattern emerges, as in the right plot, where color-related

 questions concentrate in the top-right cluster — we observe that

 questions either are of type *Query color* or *Equal color*,

 or contain concepts related to color. Sometimes it is harder to draw a

 conclusion, as in the left plot, where question types are scattered across

 the three clusters.

 In cases where question types alone cannot explain the clustering of the

 FiLM parameters, we can turn to the conditioning content itself to gain

 an understanding of the mechanism at play. Let's take a look at two more

 plots: one for feature map 26 as in the previous figure, and another

 for a different feature map, also exhibiting several subclusters. This time

 we regroup points by the words which appear in their associated question.

 In the left plot, the left subcluster corresponds to questions involving

 objects positioned *in front* of other objects, while the right

 subcluster corresponds to questions involving objects positioned

 *behind* other objects. In the right plot we see some evidence of

 separation based on object material: the left subcluster corresponds to

 questions involving *matte* and *rubber* objects, while the

 right subcluster contains questions about *shiny* or

 *metallic* objects.

 The presence of sub-clusters in the visual reasoning model also suggests

 that question interpolations may not always work reliably, but these

 sub-clusters don't preclude one from performing arithmetic on the question

 representations, as Perez et al.

 report.

 Perez et al.  report that this sort of

 task analogy is not always successful in correcting the model's answer, but

 it does point to an interesting fact about FiLM-ed networks: sometimes the

 model makes a mistake not because it is incapable of computing the correct

 output, but because it fails to produce the correct FiLM parameters for a

 given task description. The reverse can also be true: if the set of tasks

 the model was trained on is insufficiently rich, the computational

 primitives learned by the FiLM-ed network may be insufficient to ensure good

 generalization. For instance, a style transfer model may lack the ability to

 produce zebra-like patterns if there are no stripes in the styles it was

 trained on. This could explain why Ghiasi et al.

  report that their style transfer

 model's ability to produce pastiches for new styles degrades if it has been

 trained on an insufficiently large number of styles. Note however that in

 that case the FiLM generator's failure to generalize could also play a role,

 and further analysis would be needed to draw a definitive conclusion.

 This points to a separation between the various computational

 primitives learned by the FiLM-ed network and the "numerical recipes"

 learned by the FiLM generator: the model's ability to generalize depends

 both on its ability to parse new forms of task descriptions and on it having

 learned the required computational primitives to solve those tasks. We note

 that this multi-faceted notion of generalization is inherited directly from

 the multi-task point of view adopted by the FiLM framework.

 Let's now turn our attention back to the overal structural properties of FiLM

 parameters observed thus far. The existence of this structure has already

 been explored, albeit more indirectly, by Ghiasi et al.

  as well as Perez et al.

 , who applied t-SNE

  on the FiLM parameter values.

 t-SNE projection of FiLM parameters for many task descriptions.

 The projection on the left is inspired by a similar projection done by Perez

 et al.  for their visual reasoning

 model trained on CLEVR and shows how questions group by question type.

 The projection on the right is inspired by a similar projection done by

 Ghiasi et al.  for their style

 transfer network. The projection does not cluster artists as neatly as the

 projection on the left, but this is to be expected, given that an artist's

 style may vary widely over time. However, we can still detect interesting

 patterns in the projection: note for instance the isolated cluster (circled

 in the figure) in which paintings by Ivan Shishkin and Rembrandt are

 aggregated. While these two painters exhibit fairly different styles, the

 cluster is a grouping of their sketches.

 To summarize, the way neural networks learn to use FiLM layers seems to

 vary from problem to problem, input to input, and even from feature to

 feature; there does not seem to be a single mechanism by which the

 network uses FiLM to condition computation. This flexibility may

 explain why FiLM-related methods have been successful across such a

 wide variety of domains.

---

Discussion

----------

 Looking forward, there are still many unanswered questions.

 Do these experimental observations on FiLM-based architectures generalize to

 other related conditioning mechanisms, such as conditional biasing, sigmoidal

 gating, HyperNetworks, and bilinear transformations? When do feature-wise

 transformations outperform methods with stronger inductive biases and vice

 versa? Recent work combines feature-wise transformations with stronger

 inductive bias methods

 ,

 which could be an optimal middle ground. Also, to what extent are FiLM's

 task representation properties

 inherent to FiLM, and to what extent do they emerge from other features

 of neural networks (i.e. non-linearities, FiLM generator

 depth, etc.)? If you are interested in exploring these or other

 questions about FiLM, we recommend looking into the code bases for

 FiLM models for [visual reasoning](https://github.com/ethanjperez/film)

 which we used as a

 starting point for our experiments here.

 Finally, the fact that changes on the feature level alone are able to

 compound into large and meaningful modulations of the FiLM-ed network is

 still very surprising to us, and hopefully future work will uncover deeper

 explanations. For now, though, it is a question that

 evokes the even grander mystery of how neural networks in general compound

 simple operations like matrix multiplications and element-wise

 non-linearities into semantically meaningful transformations.

FiLM: Visual Reasoning with a General Conditioning Layer

Learning visual reasoning without strong priors

CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning

Visual Reasoning with Multi-hop Feature Modulation

GuessWhat?! Visual object discovery through multi-modal dialogue

ReferItGame: Referring to objects in photographs of natural scenes

Modulating early visual processing by language

VQA: visual question answering

A learned representation for artistic style

Exploring the structure of a real-time, arbitrary neural artistic stylization network

Efficient video object segmentation via network modulation

Arbitrary style transfer in real-time with adaptive instance normalization

Highway networks

Long short-term memory

Squeeze-and-Excitation networks

On the state of the art of evaluation in neural language models

Language modeling with gated convolutional networks

Convolution sequence-to-sequence learning

Gated-attention readers for text comprehension

Gated-attention architectures for task-oriented language grounding

Vizdoom: A doom-based AI research platform for visual reinforcement learning

Learning to follow language instructions with adversarial reward induction

Neural module networks

Overcoming catastrophic forgetting in neural networks

Unsupervised representation learning with deep convolutional generative adversarial networks

Generative adversarial nets

Conditional image generation with PixelCNN decoders

WaveNet: A generative model for raw audio

Dynamic layer normalization for adaptive neural acoustic modeling in speech recognition

Adaptive batch normalization for practical domain adaptation

TADAM: Task dependent adaptive metric for improved few-shot learning

Prototypical networks for few-shot learning

Devise: A deep visual-semantic embedding model

Zero-shot learning through cross-modal transfer

Zero-shot learning by convex combination of semantic embeddings

HyperNetworks

Separating style and content with bilinear models

Visualizing data using t-SNE

A dataset and architecture for visual reasoning with a working memory

A parallel computation that assigns canonical object-based frames of reference

The correlation theory of brain function

Generating text with recurrent neural networks

Robust boltzmann machines for recognition and denoising

Factored conditional restricted Boltzmann machines for modeling motion style

Combining discriminative features to infer complex trajectories

Learning where to attend with deep architectures for image tracking

Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis

Convolutional learning of spatio-temporal features

Learning to relate images

Incorporating side information by adaptive convolution

Learning multiple visual domains with residual adapters

Predicting deep zero-shot convolutional neural networks using textual descriptions

Zero-shot task generalization with multi-task deep reinforcement learning

Separating style and content

Facial expression space learning

Personalized recommendation on dynamic content using predictive bilinear models

Like like alike: joint friendship and interest propagation in social networks

Matrix factorization techniques for recommender systems

Bilinear CNN models for fine-grained visual recognition

Convolutional two-stream network fusion for video action recognition

Multimodal compact bilinear pooling for visual question answering and visual grounding


Visualizing memorization in RNNs

Abstract: Memorization in Recurrent Neural Networks (RNNs) continues to pose a challenge     in many applications. We’d like RNNs to be able to store information over many     timesteps and retrieve it when it becomes relevant — but vanilla RNNs often struggle to do this.

 Memorization in Recurrent Neural Networks (RNNs) continues to pose a challenge

 in many applications. We'd like RNNs to be able to store information over many

 as Long-Short-Term Memory (LSTM)

 units and Gated Recurrent Units (GRU).

 However, the practical problem of memorization still poses a challenge.

 As such, developing new recurrent units that are better at memorization

 continues to be an active field of research.

 To compare a recurrent unit against its alternatives, both past and recent

 papers, such as the Nested LSTM paper by Monzi et al.

 , heavily rely on quantitative

 comparisons. These comparisons often measure accuracy or

 cross entropy loss on standard problems such as Penn Treebank, Chinese

 Poetry Generation, or text8, where the task is to predict the

 next character given existing input.

 While quantitative comparisons are useful, they only provide partial

 insight into the how a recurrent unit memorizes. A model can, for

example,

 achieve high accuracy and cross entropy loss by just providing

highly accurate

 predictions in cases that only require short-term memorization,

while

 being inaccurate at predictions that require long-term

 memorization.

 For example, when autocompleting words in a sentence, a model with

only short-term understanding could still exhibit high accuracy

completing the ends of words once most of the characters are present.

 However, without longer term contextual understanding it won't be

able to predict words when only a few characters are known.

 This article presents a qualitative visualization method for comparing

 recurrent units with regards to memorization and contextual understanding.

Recurrent Units

---------------

 The networks that will be analyzed all use a simple RNN structure:

hℓth\_{\ell}^{t}hℓt

 Output for layer ℓ\ellℓ at time ttt.

===

Unit\mathrm{Unit}Unit

 Recurrent unit of choice.

yty^tyt

===

Softmax\mathrm{Softmax}Softmax

(hLt)(h\_L^t)(hLt)

 In theory, the time dependency allows it in each iteration to know

 about every part of the sequence that came before. However, this time

 dependency typically causes a vanishing gradient problem that results in

 long-term dependencies being ignored during training

 .

**Vanishing Gradient:**  where the contribution from the

 earlier steps becomes insignificant in the gradient for the vanilla RNN

 unit.

Softmax LayerRecurrent LayerRecurrent LayerInput Layer

 Several solutions to the vanishing gradient problem have been proposed over

 the years. The most popular are the aforementioned LSTM and GRU units, but this

 is still an area of active research. Both LSTM and GRU are well known

  — an explanation of Nested LSTMs

 can be found [in the appendix](#appendix-nestedlstm).

* Nested LSTM

* LSTM

* GRU

**Recurrent Unit, Nested LSTM:** makes the cell update depend on another

 LSTM unit, supposedly this allows more long-term memory compared to

 stacking LSTM layers.

**Recurrent Unit, LSTM:** allows for long-term

 memorization by gateing its update, thereby solving the vanishing gradient

 problem.

**Recurrent Unit, GRU:** solves the vanishing gradient

 problem without depending on an internal memory state.

 It is not entirely clear why one recurrent unit performs better than another

 in some applications, while in other applications it is another type of

 recurrent unit that performs better. Theoretically they all solve the vanishing

 gradient problem, but in practice their performance is highly application

 dependent.

 Understanding why these differences occur is likely an opaque and

 challenging problem. The purpose of this article is to demonstrate a

 visualization technique that can better highlight what these differences

 are. Hopefully, such an understanding can lead to a deeper understanding.

Comparing Recurrent Units

-------------------------

 loss. Differences in these high-level quantitative measures

 can have many explanations and may only be because of some small improvement

 in predictions that only requires short-term contextual understanding,

 while it is often the long-term contextual understanding that is of interest.

### A problem for qualitative analysis

 Therefore a good problem for qualitatively analyzing contextual

 understanding should have a human-interpretable output and depend both on

 long-term and short-term contextual understanding. The typical problems

 that are often used, such as Penn Treebank, Chinese Poetry Generation, or

 text8  generation do not have outputs that are easy to reason about, as they

 require an extensive understanding of either grammar, Chinese poetry, or

 only output a single letter.

 The autocomplete problem is quite similar to the text8 generation

 problem: the only difference is that instead of predicting the next letter,

 the model predicts an entire word. This makes the output much more

 interpretable. Finally, because of its close relation to text8 generation,

 existing literature on text8 generation is relevant and comparable,

 in the sense that models that work well on text8 generation should work

 well on the autocomplete problem.

User types input sequence.

Recurrent neural network processes the sequence.

The output for the last character is used.

The most likely suggestions are extracted.

 parts of north af

custom input, loading ...

africa(85.30%)africans(1.40%)african(8.90%)

**Autocomplete:** An application that has a humanly

 interpretable output, while depending on both short and long-term

 contextual understanding. In this case, the network uses past information

 and understands the next word should be a country.

 The output in this figure was produced by the GRU model;

 all model setups are [described in the appendix](#appendix-autocomplete).

 Try [removing the last letters](javascript:arDemoShort();) to see

 that the network continues to give meaningful suggestions.

You can also type in your own text.

 ([reset](javascript:arDemoReset();)).

 The autocomplete dataset is constructed from the full

 [text8](http://mattmahoney.net/dc/textdata.html) dataset. The

 recurrent neural networks used to solve the problem have two layers, each

 with 600 units. There are three models, using GRU, LSTM, and Nested LSTM.

 See [the appendix](#appendix-autocomplete) for more details.

### Connectivity in the Autocomplete Problem

 In the recently published Nested LSTM paper

 , they qualitatively compared their

 Nested LSTM unit to other recurrent units, to show how it memorizes in

 comparison, by visualizing individual cell activations.

 This visualization was inspired by Karpathy et al.

  where they identify cells

 that capture a specific feature. To identify a specific

 feature, this visualization approach works well. However, it is not a useful

 argument for memorization in general as the output is entirely dependent

 on what feature the specific cell captures.

 Instead, to get a better idea of how well each model memorizes and uses

 memory for contextual understanding, the connectivity between the desired

 output and the input is analyzed. This is calculated as:

connectivity(	extrm{connectivity}(connectivity(

ttt

 Input time index.

,,,

t~	ilde{t}t~

 Output time index.

)=) =)=

 xtx^txt.

 Exploring the connectivity gives a surprising amount of insight into the

 different models' ability for long-term contextual understanding. Try and

 interact with the figure below yourself to see what information the

 different models use for their predictions.

**Connectivity:** the connection strength between

 ([reset](javascript:connectivitySetIndex(null);)).

 *Hover over or tap the text to change the selected character.*

 Let's highlight three specific situations:

1

 Observe how the models predict the word "learning" with [only the first two

 information and thus only suggests common words starting with the letter "l".

 In contrast, the LSTM and GRU models both suggests the word "learning".

 The GRU model shows stronger connectivity with the word "advanced",

2

 Examine how the models predict the word "grammar".

 Thus, no model suggests "grammar" until it has

 [seen at least 4 characters](javascript:connectivitySetIndex(32);).

 When "grammar" appears for the second time, the models have more context.

 need [at least 4 characters](javascript:connectivitySetIndex(162);).

3

 Finally, let's look at predicting the word "schools"

 the GRU model seems better at using past information for

 contextual understanding.

 What makes this case noteworthy is how the LSTM model appears to

 use words from almost the entire sentence as context. However,

 its suggestions are far from correct and have little to do

 with the previous words it seems to use in its prediction.

 This suggests that the LSTM model in this setup is capable of

 long-term memorization, but not long-term contextual understanding.

1

2

3

 These observations show that the connectivity visualization is a powerful tool

 However, it is only possible to compare models on the same dataset, and

 on a specific example. As such, while these observations may show that

 these observations may not generalize to other datasets or hyperparameters.

### Future work; quantitative metric

 From the above observations it appears that short-term contextual understanding

 using previously seen letters from the word itself, as more letters become

 GRU network — use previously seen words as context for the prediction.

 This observation suggests a quantitative metric: measure the accuracy given

 how many letters from the word being predicted are already known.

**Accuracy Graph**: shows the accuracy

 given a fixed number of characters in a word that the RNN has seen.

 0 characters mean that the RNN has only seen the space leading up

 to the word, including all the previous text which should provide context.

 The different line styles, indicates if the correct word should appear

 among the top 1, 2, or 3 suggestions.

 These results suggest that the GRU model is better at long-term contextual

 understanding, while the LSTM model is better at short-term contextual

 understanding. These observations are valuable, as it justifies why the

 the GRU model is far better at long-term contextual understanding.

 While more detailed quantitative metrics like this provides new insight,

 qualitative analysis like the connectivity figure presented

 intuitive understanding of how the model works, which a quantitative metric

 cannot. It also shows that a wrong prediction can still be considered a

 useful prediction, such as a synonym or a contextually reasonable

 prediction.

Conclusion

----------

 Looking at overall accuracy and cross entropy loss in itself is not that

 interesting. Different models may prioritize either long-term or

 short-term contextual understanding, while both models can have similar

 accuracy and cross entropy.

 A qualitative analysis, where one looks at how previous input is used in

 the prediction is therefore also important when judging models. In this

 case, the connectivity visualization together with the autocomplete

 predictions, reveals that the GRU model is much more capable of long-term

 contextual understanding, compared to LSTM and Nested LSTM. In the case of

 LSTM, the difference is much higher than one would guess from just looking

 at the overall accuracy and cross entropy loss alone. This observation is

 not that interesting in itself as it is likely very dependent on the

 hyperparameters, and the specific application.

 Much more valuable is that this visualization method makes it possible

 to intuitively understand how the models are different, to a much higher

 degree than just looking at accuracy and cross entropy. For this application,

 it is clear that the GRU model uses repeating words and semantic meaning

 of past words to make its prediction, to a much higher degree than the LSTM

 and Nested LSTM models. This is both a valuable insight when choosing the

 final model, but also essential knowledge when developing better models

 in the future.

Long short-term memory

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Nested LSTMs

The Penn Treebank: Annotating Predicate Argument Structure

text8 Dataset

On the difficulty of training recurrent neural networks

Visualizing and Understanding Recurrent Networks