Visualizing Neural Networks with the Grand Tour

Mingwei Li
Zhenge Zhao
Carlos Scheidegger

Abstract: The Grand Tour is a classic visualization technique for high-dimensional point clouds that projects  a high-dimensional dataset into two dimensions.    Over time, the Grand Tour smoothly animates its projection so that  every possible view of the dataset is (eventually) presented to the  viewer.    Unlike modern nonlinear projection methods such as t-SNE and UMAP, the Grand Tour is fundamentally a linear  method.    In this article, we show how to leverage the linearity of the Grand  Tour to enable a number of capabilities that are uniquely useful to  visualize the behavior of neural networks.      Concretely, we present three use cases of interest: visualizing the  training process as the network weights change, visualizing the  layer-to-layer behavior as the data goes through the network and  visualizing both how adversarial examples are crafted and how they fool a neural network.

 a high-dimensional dataset into two dimensions.

 Over time, the Grand Tour smoothly animates its projection so that

every possible view of the dataset is (eventually) presented to the

viewer.

 method.

 In this article, we show how to leverage the linearity of the Grand

Tour to enable a number of capabilities that are uniquely useful to

visualize the behavior of neural networks.

 Concretely, we present three use cases of interest: visualizing the

training process as the network weights change, visualizing the

layer-to-layer behavior as the data goes through the network and

Introduction

------------

 Deep neural networks often achieve best-in-class performance in

supervised learning contests such as the ImageNet Large Scale Visual

Recognition Challenge (ILSVRC).

 In this article, we present a method to visualize the responses of a

neural network which leverages properties of deep neural networks and

properties of the *Grand Tour*.

 As we will show, this data-visual correspondence is central to the

method we present, especially when compared to other non-linear

projection methods like UMAP and t-SNE.

 These kinds of visualizations are useful to elucidate the activation

patterns of a neural network for a single example, but they might offer

less insight about the relationship between different examples,

different states of the network as it's being trained, or how the data

in the example flows through the different layers of a single network.

 Therefore, we instead aim to enable visualizations of the *context around*

 our objects of interest: what is the difference between the present

training epoch and the next one? How does the classification of a

network converge (or diverge) as the image is fed through the network?

 Linear methods are attractive because they are particularly easy to

reason about.

 The Grand Tour works by generating a random, smoothly changing

rotation of the dataset, and then projecting the data to the

two-dimensional screen: both are linear processes.

 Although deep neural networks are clearly not linear processes, they

often confine their nonlinearity to a small set of operations, enabling

us to still reason about their behavior.

 Our proposed method better preserves context by providing more

 consistency: it should be possible to know *how the visualization

 would change, if the data had been different in a particular

 way*.

Working Examples

----------------

 To illustrate the technique we will present, we trained deep neural

 network models (DNNs) with 3 common image classification datasets:

 MNIST

 MNIST contains grayscale images of 10 handwritten digits

 Image credit to <https://en.wikipedia.org/wiki/File:MnistExamples.png>

,

 fashion-MNIST

 Fashion-MNIST contains grayscale images of 10 types of fashion items:

 and CIFAR-10

 CIFAR-10 contains RGB images of 10 classes of objects

 Image credit to <https://www.cs.toronto.edu/~kriz/cifar.html>

.

 While our architecture is simpler and smaller than current DNNs, it's

still indicative of modern networks, and is complex enough to

demonstrate both our proposed techniques and shortcomings of typical

approaches.

 The following figure presents a simple functional diagram of the

neural network we will use throughout the article. The neural network is

 a sequence of linear (both convolutional

 A convolution calculates weighted sums of regions in the input.

 For example

 ![](Visualizing%20Neural%20Networks%20with%20the%20Grand%20Tour_files/conv.gif)

  and fully-connected

 A fully-connected layer computes output neurons as weighted sum of

input neurons. In matrix form, it is a matrix that linearly transforms

the input vector into the output vector.

 ), max-pooling, and ReLU

 Image credit to <https://pytorch.org/docs/stable/nn.html#relu>

 layers, culminating in a softmax

 layer.

Neural network opened. The colored blocks are building-block

 functions (i.e. neural network layers), the gray-scale heatmaps are

either the input image or intermediate activation vectors after some

layers.

 Even though neural networks are capable of incredible feats of

classification, deep down, they really are just pipelines of relatively

simple functions.

 For images, the input is a 2D array of scalar values for gray scale

images or RGB triples for colored images.

 -dimensional vector.

 Similarly, the intermediate values after any one of the functions in

composition, or activations of neurons after a layer, can also be seen

as vectors in Rn\mathbb{R}^nRn, where nnn

 is the number of neurons in the layer.

 The softmax, for example, can be seen as a 10-vector whose values are

positive real numbers that sum up to 1.

 This vector view of data in neural network not only allows us

represent complex data in a mathematically compact form, but also hints

us on how to visualize them in a better way.

 Most of the simple functions fall into two categories: they are either

 linear transformations of their inputs (like fully-connected layers or

convolutional layers), or relatively simple non-linear functions that

work component-wise (like sigmoid activations

 Image credit to <https://en.wikipedia.org/wiki/Sigmoid_function>

 or ReLU activations).

 Some operations, notably max-pooling

 Max-pooling calculates maximum of a region in the input. For example

 The above figure helps us look at a single image at a time; however,

it does not provide much context to understand the relationship between

layers, between different examples, or between different class labels.

For that, researchers often turn to more sophisticated visualizations.

Using Visualization to Understand DNNs

--------------------------------------

 Let's start by considering the problem of visualizing the training

process of a DNN.

 When training neural networks, we optimize parameters in the function

to minimize a scalar-valued loss function, typically through some form

of gradient descent.

 We want the loss to keep decreasing, so we monitor the whole history

of training and testing losses over rounds of training (or "epochs"), to

 make sure that the loss decreases over time.

 The following figure shows a line plot of the training loss for the

MNIST classifier.

<PHONE>.10.512Training EpochLoss

 Although its general trend meets our expectation as the loss steadily

decreases, we see something strange around epochs 14 and 21: the curve

goes almost flat before starting to drop again.

 What happened? What caused that?

<PHONE>.050.10.5123Training EpochLoss

 loss like above, we see that the two drops were caused by the classses 1

 and 7; the model learns different classes at very different times in

the training process.

 Although the network learns to recognize digits 0, 2, 3, 4, 5, 6, 8

and 9 early on, it is not until epoch 14 that it starts successfully

recognizing digit 1, or until epoch 21 that it recognizes digit 7.

 If we knew ahead of time to be looking for class-specific error rates,

 then this chart works well. But what if we didn't really know what to

look for?

 examples at once, looking

 to find patterns like class-specific behavior, and other patterns

besides.

 Should there be only two neurons in that layer, a simple

two-dimensional scatter plot would work.

 However, the points in the softmax layer for our example datasets are

10 dimensional (and in larger-scale classification problems this number

can be much larger).

 We need to either show two dimensions at a time (which does not scale

well as the number of possible charts grows quadratically),

### The State-of-the-art Dimensionality Reduction is Non-linear

 Modern dimensionality reduction techniques such as t-SNE and UMAP are

capable of impressive feats of summarization, providing two-dimensional

images where similar points tend to be clustered together very

effectively.

 However, these methods are not particularly good to understand the

behavior of neuron activations at a fine scale.

 Consider the aforementioned intriguing feature about the different

learning rate that the MNIST classifier has on digit 1 and 7: the

network did not learn to recognize digit 1 until epoch 14, digit 7 until

 epoch 21.

 We compute t-SNE, Dynamic t-SNE,

 and UMAP projections of the epochs where the phenomenon we described

happens.

 Consider now the task of identifying this class-specific behavior

during training. As a reminder, in this case, the strange behavior

happens with digits 1 and 7, around epochs 14 and 21 respectively.

 While the behavior is not particularly subtle&emdash;digit goes

from misclassified to correctly classified&emdash; it is quite hard

to notice it in any of the plots below.

 Only on careful inspection we can notice that (for example) in the

UMAP plot, the digit 1 which clustered in the bottom in epoch 13 becomes

 a new tentacle-like feature in epoch 14.

 // let sm1 = createSmallMultiple('#smallmultiple1',

 // [13,14,15, 20,21,22], ['t-SNE', 'Dynamic t-SNE', 'UMAP'],

 // 'mnist', true, highlight\_digits);

 let sm1 = createSmallMultiple('#smallmultiple1',

 [13,14,15, 20,21,22], ['t-SNE', 'Dynamic t-SNE', 'UMAP'],

 'mnist', true);

Softmax activations of the MNIST classifier with non-linear

dimensionality reduction. Use the buttons on the right to highlight

digits 1 and 7 in the plot, or drag rectangles around the charts to

select particular point subsets to highlight in the other charts.

 One reason that non-linear embeddings fail in elucidating this

phenomenon is that, for the particular change in the data, the fail the

principle of *data-visual correspondence* .

 More concretely, the principle states that specific visualization tasks

 should be modeled as functions that change the data; the visualization

sends this change from data to visuals, and

 we can study the extent to which the visualization changes are easily

perceptible.

 Ideally, we want the changes in data and visualization to *match in magnitude*:

 a barely noticeable change in visualization should be due to the

smallest possible change in data, and a salient change in visualization

should reflect a significant one in data.

 points in the visualization move dramatically.

 For both UMAP and t-SNE, the position of each single point depends

non-trivially on the whole data distribution in such embedding

algorithms.

 This property is not ideal for visualization because it fails the

 Non-linear embeddings that have non-convex objectives also tend to be

sensitive to initial conditions.

 For example, in MNIST, although the neural network starts to stabilize

 on epoch 30, t-SNE and UMAP still generate quite different projections

between epochs 30, 31 and 32 (in fact, all the way to 99).

 Temporal regularization techniques (such as Dynamic t-SNE) mitigate

these consistency issues, but still suffer from other interpretability

issues.

 Now, let's consider another task, that of identifying classes which

the neural network tends to confuse.

 For this example, we will use the Fashion-MNIST dataset and

classifier, and consider the confusion among sandals, sneakers and ankle

 boots.

 If we know ahead of time that these three classes are likely to

confuse the classifier, then we can directly design an appropriate

linear projection, as can be seen in the last row of the following

figure (we found this particular projection using both the Grand Tour

and the direct manipulation technique we later describe). The pattern in

 this case is quite salient, forming a triangle.

 T-SNE, in contrast, incorrectly separates the class clusters (possibly

 because of an inappropriately-chosen hyperparameter).

 UMAP successfully isolates the three classes, but even in this case

it's not possible to distinguish between three-way confusion for the

classifier in epochs 5 and 10 (portrayed in a linear method by the

presence of points near the center of the triangle), and multiple

two-way confusions in later epochs (evidences by an "empty" center).

 sm3 = createSmallMultiple('#smallmultiple3',

 [2,5,10,20,50,99], ['t-SNE', 'UMAP', 'Linear'],

 'fashion-mnist', true,

 highlight\_shoes\_button,

 highlight\_shoes,

 );

Highlight shoes

Three-way confusion in fashion-MNIST. Notice that in

contrast to non-linear methods, a carefully-constructed linear

projection can offer a better visualization of the classifier behavior.

Linear Methods to the Rescue

----------------------------

 When given the chance, then, we should prefer methods for which

changes in the data produce predictable, visually salient changes in the

 result, and linear dimensionality reductions often have this property.

 Here, we revisit the linear projections described above in an

interface where the user can easily navigate between different training

epochs.

 In addition, we introduce another useful capability which is only

available to linear methods, that of direct manipulation.

 space will be projected to.

 In the context of projecting the final classification layer, this is

especially simple to interpret: they are the destinations of an input

that is classified with 100% confidence to any one particular class.

 If we provide the user with the ability to change these vectors by

dragging around user-interface handles, then users can intuitively set

up new linear projections.

 This setup provides additional nice properties that explain the

salient patterns in the previous illustrations.

 For example, because projections are linear and the coefficients of

vectors in the classification layer sum to one, classification outputs

that are halfway confident between two classes are projected to vectors

that are halfway between the class handles.

 From

 this linear projection, we can easily identify the learning of

 digit 1 on

 epoch 14 and

 digit 7 on

 epoch 21.

 This linear projection clearly shows model's confusion among

 sandals,

 sneakers, and

 ankle boots.

 Similarly, this projection shows the true three-way confusion about

 pullovers,

 coats, and

 shirts.

 (The shirts are also get confused with

 t-shirts/tops. )

 Both projections are found by direct manipulations.

 Examples falling between classes indicate that the model has trouble

distinguishing the two, such as sandals vs. sneakers, and sneakers vs.

ankle boot classes.

 Note, however, that this does not happen as much for sandals vs. ankle

 boots: not many examples fall between these two classes.

 Moreover, most data points are projected close to the edge of the

triangle.

 This tells us that most confusions happen between two out of the three

 classes, they are really two-way confusions.

 Within the same dataset, we can also see pullovers, coats and shirts

filling a triangular *plane*.

 This is different from the sandal-sneaker-ankle-boot case, as examples

 not only fall on the boundary of a triangle, but also in its interior: a

 true three-way confusion.

 Similarly, in the CIFAR-10 dataset we can see confusion between dogs

and cats, airplanes and ships.

 The mixing pattern in CIFAR-10 is not as clear as in fashion-MNIST,

because many more examples are misclassified.

 This linear projection clearly shows model's confusion between

 cats and

 dogs.

 Similarly, this projection shows the confusion about

 airplanes and

 ships.

 Both projections are found by direct manipulations.

The Grand Tour

--------------

 In the previous section, we took advantage of the fact that we knew

which classes to visualize.

 That meant it was easy to design linear projections for the particular

 tasks at hand.

 But what if we don't know ahead of time which projection to choose

from, because we don't quite know what to look for?

 Principal Component Analysis (PCA) is the quintessential linear

dimensionality reduction method,

 choosing to project the data so as to preserve the most variance

possible.

 However, the distribution of data in softmax layers often has similar

variance along many axis directions, because each axis concentrates a

similar number of examples around the class vector.We

 are assuming a class-balanced training dataset. Nevertheless, if the

training dataset is not balanced, PCA will prefer dimensions with more

examples, which might not be help much either.

 As a result, even though PCA projections are interpretable and

consistent through training epochs, the first two principal components

of softmax activations are not substantially better than the third.

 So which of them should we choose?

 Instead of PCA, we propose to visualize this data by smoothly

animating random projections, using a technique called the Grand Tour.

Starting with a random velocity, it smoothly rotates data points around

the origin in high dimensional space, and then projects it down to 2D

for display.

Here are some examples of how Grand Tour acts on some (low-dimensional)

objects:

* On a square, the Grand Tour rotates it with a constant angular velocity.

Grand tours of a square, a cube and a tesseract

### The Grand Tour of the Softmax Layer

 We first look at the Grand Tour of the softmax layer.

 The Grand Tour of softmax layer in the last (99th) epoch, with

 MNIST,

 fashion-MNIST or

 CIFAR-10 dataset.

 The Grand Tour of the softmax layer lets us qualitatively assess the

performance of our model.

 In the particular case of this article, since we used comparable

architectures for three datasets, this also allows us to gauge the

relative difficulty of classifying each dataset.

 We can see that data points are most confidently classified for the

MNIST dataset, where the digits are close to one of the ten corners of

the softmax space. For Fashion-MNIST or CIFAR-10, the separation is not

as clean, and more points appear *inside* the volume.

### The Grand Tour of Training Dynamics

 Linear projection methods naturally give a formulation that is

independent of the input points, allowing us to keep the projection

fixed while the

 data changes.

 To recap our working example, we trained each of the neural networks

for 99 epochs and recorded the entire history of neuron activations on a

 subset of training and testing examples. We can use the Grand Tour,

then, to visualize the actual training process of these networks.

 In the beginning when the neural networks are randomly initialized,

all examples are placed around the center of the softmax space, with

equal weights to each class.

 Through training, examples move to class vectors in the softmax space.

 The Grand Tour also lets us

 compare visualizations of the training and testing data, giving us a

qualitative assessment of over-fitting.

 In the MNIST dataset, the trajectory of testing images through

training is consistent with the training set.

 Data points went directly toward the corner of its true class and all

classes are stabilized after about 50 epochs.

 On the other hand, in CIFAR-10 there is an *inconsistency*

between the training and testing sets. Images from the testing set keep

oscillating while most images from training converges to the

corresponding class corner.

 In epoch 99, we can clearly see a difference in distribution between

these two sets.

 This signals that the model overfits the training set and thus does

not generalize well to the testing set.

 With this view of CIFAR-10 ,

 the color of points are more mixed in testing (right) than training

(left) set, showing an over-fitting in the training process.

 Compare

 CIFAR-10

 with

 MNIST

 or fashion-MNIST,

 where there is less difference between training and testing sets.

### The Grand Tour of Layer Dynamics

 Given the presented techniques of the Grand Tour and direct

manipulations on the axes, we can in theory visualize and manipulate any

 intermediate layer of a neural network by itself. Nevertheless, this

is not a very satisfying approach, for two reasons:

* In the same way that we've kept the projection fixed as the

training data changed, we would like to "keep the projection fixed", as

the data moves through the layers in the neural network. However, this

is not straightforward. For example, different layers in a neural

network have different dimensions. How do we connect rotations of one

layer to rotations of the other?

* The class "axis handles" in the softmax layer convenient, but

that's only practical when the dimensionality of the layer is relatively

 small.

 With hundreds of dimensions, for example, there would be too many

axis handles to naturally interact with.

 In addition, hidden layers do not have as clear semantics as the

softmax layer, so manipulating them would not be as intuitive.

 To address the first problem, we will need to pay closer attention to

the way in which layers transform the data that they are given.

 To see how a linear transformation can be visualized in a particularly

 ineffective way, consider the following (very simple) weights

 A=[−1,00,−1]

 A = egin{bmatrix}

 -1, 0 \

 0, -1

 \end{bmatrix}

 A=[−1,00,−1]

 Imagine that we wish to visualize the behavior of network as the data

 xt=(1−t)⋅x0+t⋅x1=(1−2t)⋅x0

 x\_t = (1-t) \cdot x\_0 + t \cdot x\_1 = (1-2t) \cdot x\_0

 xt=(1−t)⋅x0+t⋅x1=(1−2t)⋅x0

 for t∈[0,1].t \in [0,1].t∈[0,1].

 Effectively, this strategy reuses the linear projection coefficients

from one layer to the next. This is a natural thought, since they have

the same dimension.

 However, notice the following: the transformation given by A is a

 if only that transformation operated on the negative values of the

entries.

 In addition, since the Grand Tour has a rotation itself built-in, for

 into account.

 In effect, the naive interpolation fails the principle of data-visual

correspondence: a simple change in data (negation in 2D/180 degree

rotation) results in a drastic change in visualization (all points cross

 the origin).

 This observation points to a more general strategy: when designing a

visualization, we should be as explicit as possible about which parts of

 the input (or process) we seek to capture in our visualizations.

 We should seek to explicitly articulate what are purely

representational artifacts that we should discard, and what are the real

 features a visualization we should *distill* from the

representation.

 Here, we claim that rotational factors in linear transformations of

neural networks are significantly less important than other factors such

 as scalings and nonlinearities.

 As we will show, the Grand Tour is particularly attractive in this

case because it is can be made to be invariant to rotations in data.

 As a result, the rotational components in the linear transformations

of a neural network will be explicitly made invisible.

 - a simple (coordinate-wise) scaling.

 This is explicitly saying that any linear operation (whose matrix is

represented in standard bases) is a scaling operation with appropriately

 chosen orthonormal bases on both sides.

 So the Grand Tour provides a natural, elegant and computationally

 layers are also linear. One can instantly see that by forming the

linear transformations between flattened feature maps, or by taking the

circulant structure of convolutional layers directly into account

 (For the following portion, we reduce the number of data points to 500

 and epochs to 50, in order to reduce the amount of data transmitted in a

 web-based demonstration.)

 With the linear algebra structure at hand, now we are able to trace

behaviors and patterns from the softmax back to previous layers.

 In fashion-MNIST, for example, we observe a separation of shoes

(sandals, sneakers and ankle boots as a group) from all other classes in

 the softmax layer.

 Tracing it back to earlier layers, we can see that this separation

happened as early as layer 5:

### The Grand Tour of Adversarial Dynamics

 as they are processed by a neural network.

 For this illustration, we use the MNIST dataset, and we adversarially

add perturbations to 89 digit 8s to fool the network into thinking they

are 0s.

 Previously, we either animated the training dynamics or the layer

dynamics.

 We fix a well-trained neural network, and visualize the training

process of adversarial examples, since they are often themselves

generated by an optimization process. Here, we used the Fast Gradient

Sign method.

 Again, because the Grand Tour is a linear method, the change in the

positions of the adversarial examples over time can be faithfully

attributed to changes in how the neural network perceives the images,

rather than potential artifacts of the visualization.

 Let us examine how adversarial examples evolved to fool the network:

 From this view of softmax, we can see how

 adversarial examples

 evolved from 8s

 into 0s.

 Through this adversarial training, the network eventually claims, with

 high confidence, that the inputs given are all 0s.

 If we stay in the softmax layer and slide though the adversarial

training steps in the plot, we can see adversarial examples move from a

high score for class 8 to a high score for class 0.

 Although all adversarial examples are classified as the target class

(digit 0s) eventually, some of them detoured somewhere close to the

centroid of the space (around the 25th epoch) and then moved towards the

 target.

 Comparing the actual images of the two groups, we see those that those

 "detouring" images tend to be noisier.

 More interesting, however, is what happens in the intermediate layers.

Discussion

----------

### Limitations of the Grand Tour

 Early on, we compared several state-of-the-art dimensionality

reduction techniques with the Grand Tour, showing that non-linear

methods do not have as many desirable properties as the Grand Tour for

understanding the behavior of neural networks.

 However, the state-of-the-art non-linear methods come with their own

strength.

 Whenever geometry is concerned, like the case of understanding

multi-way confusions in the softmax layer, linear methods are more

interpretable because they preserve certain geometrical structures of

data in the projection.

 When topology is the main focus, such as when we want to cluster the

data or we need dimensionality reduction for downstream models that are

less sensitive to geometry, we might choose non-linear methods such as

UMAP or t-SNE for they have more freedom in projecting the data, and

will generally make better use of the fewer dimensions available.

### The Power of Animation and Direct Manipulation

 When comparing linear projections with non-linear dimensionality

reductions, we used small multiples to contrast training epochs and

dimensionality reduction methods.

 The Grand Tour, on the other hand, uses a single animated view.

 When comparing small multiples and animations, there is no general

consensus on which one is better than the other in the literature,

aside.

 between small multiples and animated plots.

 Regardless of these concerns, in our scenarios, the use of animation

comes naturally from the direct manipulation and the existence of a

continuum of rotations for the Grand Tour to operate in.

### Non-sequential Models

 In our work we have used models that are purely "sequential", in the

sense that the layers can be put in numerical ordering, and that the

activations for

 With our technique, one can visualize neuron activations on each such

branch, but additional research is required to incorporate multiple

branches directly.

### Scaling to Larger Models

 Modern architectures are also wide. Especially when convolutional

layers are concerned, one could run into issues with scalability if we

see such layers as a large sparse matrix acting on flattened

multi-channel images.

 For the sake of simplicity, in this article we brute-forced the

computation of the alignment of such convolutional layers by writing out

 their explicit matrix representation.

 However, the singular value decomposition of multi-channel 2D

function toggle(event, id){

 let caller = d3.select(event.target); //DOM that received the event

 let callerIsActive = caller.classed('clickable-active');

 let selection = d3.select(id); //DOM to be toggled

 let isHidden = selection.classed('hidden');

 selection.classed('hidden', !isHidden);

}

Technical Details

------------------

This section presents the technical details necessary to implement the

direct manipulation of axis handles and data points, as well as how to

implement the projection consistency technique for layer transitions.

### Notation

 In this section, our notational convention is that data points are

represented as row vectors.

 An entire dataset is laid out as a matrix, where each row is a data

point, and each column represents a different feature/dimension.

 As a result, when a linear transformation is applied to the data, the

row vectors (and the data matrix overall) are left-multiplied by the

transformation matrix.

 This has a side benefit that when applying matrix multiplications in a

 chain, the formula reads from left to right and aligns with a

commutative diagram.

X↦MY

 X

 \overset{M}{\mapsto}

 Y

X↦MY

X↦U↦Σ↦VTY

 X

 \overset{U}{\mapsto}

 \overset{\Sigma}{\mapsto}

 \overset{V^T}{\mapsto} Y

X↦U↦Σ↦VTY

nicely aligns with the formula.

###

Direct Manipulation

 The direct manipulations we presented earlier provide explicit control

 over the possible projections for the data points.

 We provide two modes: directly manipulating class axes (the "axis

mode"), or directly manipulating a group of data points through their

centroid (the "data point mode").

 we may prefer one mode than the other.

 We will see that the axis mode is a special case of data point mode,

because we can view an axis handle as a particular "fictitious" point in

 the dataset.

 Because of its simplicity, we will first introduce the axis mode.

####

 The Axis Mode

 The implied semantics of direct manipulation is that when a user

drags an UI element (in this case, an axis handle), they are signaling

to the system that they wished that the corresponding

 data point had been projected to the location where the UI element

was dropped, rather than where it was dragged from.

 In our case the overall projection is a rotation (originally

determined by the Grand Tour), and an arbitrary user manipulation might

not necessarily generate a new projection that is also a rotation. Our

goal, then, is to find a new rotation which satisfies the user request

and is close to the previous state of the Grand Tour projection, so that

 the resulting state satisfies the user request.

ei↦GTei~↦π2(xi,yi)

 e\_i \overset{GT}{\mapsto} 	ilde{e\_i} \overset{\pi\_2}{\mapsto} (x\_i, y\_i)

ei↦GTei~↦π2(xi,yi)

 The coordinate of the handle becomes:

 Recall that the convention is that vectors are in row form and

linear transformations are matrices that are multiplied on the right.

.

 GT~←GT\widetilde{GT} \leftarrow GTGT

←GT

GT~i,1←GTi,1+dx\widetilde{GT}\_{i,1} \leftarrow GT\_{i,1} + dxGT

i,1←GTi,1+dx

GT~i,2←GTi,2+dy\widetilde{GT}\_{i,2} \leftarrow GT\_{i,2} + dyGT

i,2←GTi,2+dy

 However, GT~\widetilde{GT}GT

 is not orthogonal for arbitrary (dx,dy)(dx, dy)(dx,dy).

 In order to find an approximation to GT~\widetilde{GT}GT

, with the ithi^{th}ith row considered first in the Gram-Schmidt process:

)

  .

####

 The Data Point Mode

 We now explain how we directly manipulate data points.

 Technically speaking, this method only considers one point at a time.

 Next, as the first step in Gram-Schmidt, we normalized this row:

 GTi(new):=normalize(GT~i)=normalize(ei~+Δ~)

 GTi(new):=normalize(GT

i)=normalize(ei~+Δ~)

 in high dimensional space.

 This intuition is precisely how we implemented our direct manipulation

 on arbitrary data points, which we will specify as below.

 Generalizing this observation from axis handle to arbitrary data

point, we want to find the rotation that moves the centroid of a

selected subset of data points c~	ilde{c}c~ to

 c~(new):=(c~+Δ~)⋅∣∣c~∣∣/∣∣c~+Δ~∣∣

 c~(new):=(c~+Δ~)⋅∣∣c~∣∣/∣∣c~+Δ~∣∣

 First, the angle of rotation can be found by their cosine similarity:

 θ=arccos(⟨c~,c~(new)⟩∣∣c~∣∣⋅∣∣c~(new)∣∣) 	heta = 	extrm{arccos}(

 )θ=arccos(∣∣c~∣∣⋅∣∣c~(new)∣∣⟨c~,c~(new)⟩)

 Next, to find the matrix form of the rotation, we need a convenient basis.

 c~⊥(new):=c~−∣∣c~∣∣⋅cosθc~(new)∣∣c~(new)∣∣

 	ilde{c}^{(new)}\_{\perp}

 c~⊥(new):=c~−∣∣c~∣∣⋅cosθ∣∣c~(new)∣∣c~(new)

Q:=[⋯normalize(c~)⋯⋯normalize(c~⊥(new))⋯P]

 Q :=

 egin{bmatrix}

 \cdots 	extsf{normalize}(	ilde{c}) \cdots \

 \cdots 	extsf{normalize}(	ilde{c}^{(new)}\_{\perp}) \cdots \

 P

 \end{bmatrix}

 Q:=⎣⎡⋯normalize(c~)⋯⋯normalize(c~⊥(new))⋯P⎦⎤

 where PPP completes the remaining space.

 ρ=QT[cosθsinθ00⋯−sinθcosθ00⋯00⋮⋮I]Q=:QTR1,2(θ)Q

ho = Q^T

 egin{bmatrix}

 \cos 	heta& \sin 	heta& 0& 0& \cdots\

 -\sin 	heta& \cos 	heta& 0& 0& \cdots\

 0& 0& \

 dots& dots& & I& \

 \end{bmatrix}

 Q

 =: Q^T R\_{1,2}(	heta) Q

 ρ=QT⎣⎢⎢⎢⎢⎡cosθ−sinθ0⋮sinθcosθ0⋮0000I⋯⋯⎦⎥⎥⎥⎥⎤Q=:QTR1,2(θ)Q

 GT(new):=GT⋅ρ

 GT^{(new)} := GT \cdot
ho

 GT(new):=GT⋅ρ

 repeatedly take a random vector, find its orthogonal component to the

span of the current basis vectors and add it to the basis set.

###

 Layer Transitions

####

 ReLU Layers

 Since ReLU does not change the dimensionality and the function is taken

 coordinate wise, we can animate the transition by a simple linear

interpolation: for a time parameter t∈[0,1]t \in [0,1]t∈[0,1],

 X(l−1)→l(t):=(1−t)Xl−1+tXl

 X^{(l-1) 	o l}(t) := (1-t) X^{l-1} + t X^{l}

 X(l−1)→l(t):=(1−t)Xl−1+tXl

####

 Linear Layers

 Transitions between linear layers can seem complicated, but as we will

 show, this comes from choosing mismatching bases on either side of the

transition.

 M=UΣVTM = U \Sigma V^TM=UΣVT

 The Grand Tour has a single parameter that represents the current

rotation of the dataset. Since our goal is to keep the transition

consistent, we notice that UUU and VTV^TVT

 have essentially no significance - they are just rotations to the view

that can be exactly "canceled" by changing the rotation parameter of the

 Grand Tour in either layer.

 Given Xl=Xl−1UΣVTX^{l} = X^{l-1} U \Sigma V^TXl=Xl−1UΣVT, we have

 (XlV)=(Xl−1U)Σ

 (X^{l}V) = (X^{l-1}U)\Sigma

 (XlV)=(Xl−1U)Σ

 For a time parameter t∈[0,1]t \in [0,1]t∈[0,1],

 X(l−1)→l(t):=(1−t)(Xl−1U)+t(XlV)=(1−t)(Xl−1U)+t(Xl−1UΣ)

 X(l−1)→l(t):=(1−t)(Xl−1U)+t(XlV)=(1−t)(Xl−1U)+t(Xl−1UΣ)

####

 Convolutional Layers

 Convolutional layers can be represented as special linear layers.

 With a change of representation, we can animate a convolutional layer

like the previous section.

 For 2D convolutions this change of representation involves flattening

the input and output, and repeating the kernel pattern in a sparse

matrix M∈Rm×nM \in \mathbb{R}^{m 	imes n}M∈Rm×n, where mmm and nnn

 are the dimensionalities of the input and output respectively.

 This change of representation is only practical for a small

dimensionality (e.g. up to 1000), since we need to solve SVD for linear

layers.

 However, the singular value decomposition of multi-channel 2D

####

 Max-pooling Layers

Conclusion

----------

 As powerful as t-SNE and UMAP are, they often fail to offer the

correspondences we need, and such correspondences can come,

surprisingly, from relatively simple methods like the Grand Tour. The

Grand Tour method we presented is particularly useful when direct

manipulation from the user is available or desirable.

 We believe that it might be possible to design methods that highlight

the best of both worlds, using non-linear dimensionality reduction to

create intermediate, relatively low-dimensional representations of the

activation layers, and using the Grand Tour and direct manipulation to

compute the final projection.

The grand tour: a tool for viewing multidimensional data

Visualizing data using t-SNE

Umap: Uniform manifold approximation and projection for dimension reduction

The mythos of model interpretability

Visualizing dataflow graphs of deep learning models in tensorflow

An algebraic process for visualization design

Feature visualization

MNIST handwritten digit database

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Learning multiple layers of features from tiny images

Rectified linear units improve restricted boltzmann machines

Visualizing time-dependent data using dynamic t-SNE

How to use t-sne effectively

We Recommend a Singular Value Decomposition

The singular values of convolutional layers

Explaining and harnessing adversarial examples

Animation, small multiples, and the effect of mental map preservation in dynamic graphs

Animation: can it facilitate?

Highway networks

Going deeper with convolutions


Augustus Odena
Vincent Dumoulin
Chris Olah

Abstract: When we look very closely at images generated by neural networks,  we often see a strange checkerboard pattern of artifacts. It’s more  obvious in some cases than others, but a large fraction of recent models  exhibit this behavior.

Deconvolution and Checkerboard Artifacts

========================================

 dt-byline {

 font-size: 12px;

 line-height: 18px;

 display: block;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 color: rgba(0, 0, 0, 0.5);

 padding-top: 12px;

 padding-bottom: 12px;

 }

 dt-article.centered dt-byline {

 text-align: center;

 }

 dt-byline a,

 dt-article dt-byline a {

 text-decoration: none;

 border-bottom: none;

 }

 dt-article dt-byline a:hover {

 text-decoration: underline;

 border-bottom: none;

 }

 dt-byline .authors {

 text-align: left;

 }

 dt-byline .name {

 display: inline;

 text-transform: uppercase;

 }

 dt-byline .affiliation {

 display: inline;

 }

 dt-byline .date {

 display: block;

 text-align: left;

 }

 dt-byline .year, dt-byline .month {

 display: inline;

 }

 dt-byline .citation {

 display: block;

 text-align: left;

 }

 dt-byline .citation div {

 display: inline;

 }

 @media(min-width: 768px) {

 dt-byline {

 }

 }

 @media(min-width: 1080px) {

 dt-byline {

 border-bottom: none;

 margin-bottom: 70px;

 }

 dt-byline a:hover {

 color: rgba(0, 0, 0, 0.9);

 }

 dt-byline .authors {

 display: inline-block;

 }

 dt-byline .author {

 display: inline-block;

 margin-right: 12px;

 /\*padding-left: 20px;\*/

 /\*border-left: 1px solid #ddd;\*/

 }

 dt-byline .affiliation {

 display: block;

 }

 dt-byline .author:last-child {

 margin-right: 0;

 }

 dt-byline .name {

 display: block;

 }

 dt-byline .date {

 border-left: 1px solid rgba(0, 0, 0, 0.1);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .year, dt-byline .month {

 display: block;

 }

 dt-byline .citation {

 border-left: 1px solid rgba(0, 0, 0, 0.15);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .citation div {

 display: block;

 }

 }

Augustus Odena

[Google Brain](http://g.co/brain)

[Vincent Dumoulin](http://vdumoulin.github.io/)

[Université de Montréal](https://mila.umontreal.ca/en/)

[Chris Olah](http://colah.github.io/)

[Google Brain](http://g.co/brain)

Oct. 17

2016

[Citation:

Odena, et al., 2016](#citation)

When we look very closely at images generated by neural networks,

we often see a strange checkerboard pattern of artifacts. It's more

obvious in some cases than others, but a large fraction of recent models

 exhibit this behavior.

.pattern-examples {

 width: 100%;

}

.pattern-examples > figure {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 position: relative;

}

.pattern-examples > figure:after {

 overflow: visible;

 content: "";

 background-image: url("assets/pointer.svg");

 width: 27px;

 height: 27px;

 position: absolute;

 right: -51px;

}

.pattern-examples .example {

 width: 100%;

 margin-right: 2.5%;

}

.pattern-examples .example:last-of-type {

 margin-right: 0;

}

.pattern-examples .original {

 position: relative;

}

.pattern-examples .original:after {

 content: "";

 display: block;

 width: 100%;

 height: 150%;

 position: absolute;

 top: -20%;

 z-index: 1;

}

.pattern-examples .example .reticle {

 position: absolute;

 pointer-events: none;

}

.pattern-examples .example .reticle:after {

 box-sizing: border-box;

 position: relative;

 width: 100%;

 height: 100%;

 left: -50%;

 top: -50%;

 border-radius: 50%;

 border: 1px solid white;

 display: block;

 content: '';

 background-color: rgba(255, 255, 255, 0.2);

 box-shadow: 0 0 4px rgba(0, 0, 0, 0.5);

}

.pattern-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

.pattern-examples .closeup {

 display: block;

 width: 90%;

 border-radius: 50%;

 margin-bottom: 12px;

 position: relative;

 left: 5%;

 top: -5px;

 border: 2px solid white;

 overflow: hidden;

 box-shadow: 0 2px 8px rgba(0, 0, 0, 0.3);

}

.pattern-examples figcaption {

 width: 90%;

 margin-left: 5%;

 text-align: center;

}

.pattern-examples .closeup img {

 position: absolute;

 width: 800%;

}

.pattern-examples .closeup:after {

 padding-top: 100%;

 display: block;

 content: '';

}

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_dcgan.png)

Radford, et al., 2015 [1]

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_salimans.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_salimans.png)

Salimans et al., 2016 [2]

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_donahue.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_donahue.png)

Donahue, et al., 2016 [3]

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_dumoulin.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_sample_dumoulin.png)

Dumoulin, et al., 2016 [4]

(function() {

var html = d3.select(".pattern-examples");

var original = html.selectAll(".example .original");

html

 .on("mouseleave", () => {

 console.log("out")

 original.each(resetReticle);

 });

original

 .datum(function() {

 var focus = this.getAttribute("data-focus").split(",")

 return {

 zoom: focus[0],

 x: focus[1],

 y: focus[2]

 };

 })

 .on("mouseleave", resetReticle)

 .on("mousemove", updateReticle)

 .each(resetReticle);

function updateReticle(d) {

 original.each(resetReticle);

 var x = d3.event.offsetX / this.getBoundingClientRect().width;

 var y = d3.event.offsetY / this.getBoundingClientRect().height;

 setPosition(this, x, y, d.zoom, 100);

}

function resetReticle(d) {

 setPosition(this, d.x, d.y, d.zoom, 300);

}

function setPosition(element, x, y, zoom, duration) {

 var margin = 1 / zoom / 2;

 x = Math.min(Math.max(margin, x), 1 - margin)

 y = Math.min(Math.max(margin, y), 1 - margin)

 var parent = d3.select(element.parentElement);

 var reticle = parent.select(".reticle");

 var closeup = parent.select(".closeup img");

 console.log(x, y)

 reticle

 .style("width", 100 / zoom + "%")

 .style("height", 100 / zoom + "%")

 .transition()

 .ease(d3.ease("cubic-out"))

 .duration(duration)

 .styleTween("left", function (d, i, a) {

 var from = this.style.left,

 to = x \* 100 + "%";

 return d3.interpolateString(from, to);

 })

 .styleTween("top", function (d, i, a) {

 var from = this.style.top,

 to = y \* 100 + "%";

 return d3.interpolateString(from, to);

 })

 closeup

 .style("width", zoom \* 100 + "%")

 .style("height", zoom \* 100 + "%")

 .transition()

 .ease(d3.ease("cubic-out"))

 .duration(duration)

 .styleTween("left", function (d, i, a) {

 var from = this.style.left,

 to = -x \* 100 \* zoom + 50 + "%";

 return d3.interpolateString(from, to);

 })

 .styleTween("top", function (d, i, a) {

 var from = this.style.top,

 to = -y \* 100 \* zoom + 50 + "%";

 return d3.interpolateString(from, to);

 })

}

})();

Mysteriously, the checkerboard pattern tends to be most prominent

in images with strong colors.

 What's going on? Do neural networks hate bright colors? The actual

cause of these artifacts is actually remarkably simple, as is a method

for avoiding them.

---

Deconvolution & Overlap

-----------------------

When we have neural networks generate images, we often have them build them up

 from low resolution, high-level descriptions.

 We generally do this with the *deconvolution* operation.

 Roughly, deconvolution layers allow the model to use every point

 in the small image to "paint" a square in the larger one.

 We use the name "deconvolution" in this article for brevity.

 For excellent discussion of deconvolution, see [5, 6].)

Unfortunately, deconvolution can easily have "uneven overlap,"

 putting more of the metaphorical paint in some places than others [7].

 In particular, deconvolution has uneven overlap when the kernel size

(the output window size) is not divisible by the stride (the spacing

between points on the top).

 While the network could, in principle, carefully learn weights to

avoid this

  — as we'll discuss in more detail later —

 in practice neural networks struggle to avoid it completely.

stride = 1size = 8stride = 2size = 3

deconv1d();

The overlap pattern also forms in two dimensions.

 The uneven overlaps on the two axes multiply together,

 creating a characteristic checkerboard-like pattern of varying magnitudes.

deconv2d();

In fact, the uneven overlap tends to be more extreme in two dimensions!

 Because the two patterns are multiplied together, the unevenness gets squared.

 but in two dimensions this becomes a factor of four.

 While it's possible for these stacked deconvolutions to cancel out artifacts,

 they often compound, creating artifacts on a variety of scales.

stride = 1size = 5stride = 2size = 3stride = 2size = 3

deconv1d\_multi();

Stride 1 deconvolutions —

 which we often see as the last layer in successful models (eg. [2])

  — are quite effective at dampening artifacts.

 They can remove artifacts of frequencies

 size. However, artifacts can still leak through, as seen in many recent models.

In addition to the high frequency checkerboard-like artifacts we observed above,

 early deconvolutions can create lower-frequency artifacts,

 which we'll explore in more detail later.

These artifacts tend to be most prominent when outputting unusual colors.

 Since neural network layers typically have a bias

 (a learned value added to the output) it's easy to output the average color.

 The further a color — like bright red — is away from the average color,

 the more deconvolution needs to contribute.

---

Overlap & Learning

------------------

Thinking about things in terms of uneven overlap is — while a useful framing —

 is evenly balanced.

In fact, not only do models with uneven overlap not learn to avoid this,

 but models with even overlap often learn kernels that cause similar artifacts!

 While it isn't their default behavior the way it is for uneven overlap,

 it's still very easy for even overlap deconvolution to cause artifacts.

Completely avoiding artifacts is still a significant restriction on filters,

 (See [4],

 which uses stride 2 size 4 deconvolutions, as an example.)

There are probably a lot of factors at play here.

 For example, in the case of Generative Adversarial Networks (GANs),

one issue may be the discriminator and its gradients

 (we'll discuss this more later).

 But a big part of the problem seems to be deconvolution.

 At best, deconvolution is fragile because it very easily represents

artifact creating functions, even when the size is carefully chosen.

 At worst, creating artifacts is the default behavior of deconvolution.

Is there a different way to upsample that is more resistant to artifacts?

---

Better Upsampling

-----------------

 Ideally, it would go further, and be biased against such artifacts.

 avoiding the overlap issue.

 This is equivalent to "sub-pixel convolution," a technique which has recently

 had success in image super-resolution [8].

Both deconvolution and the different resize-convolution approaches

are linear operations, and can be interpreted as matrices.

 This a helpful way to see the differences between them.

 Where deconvolution has a unique entry for each output window,

resize-convolution is implicitly weight-tying in a way that discourages

high frequency artifacts.

We've had our best results with nearest-neighbor interpolation, and

 had difficulty making bilinear resize work.

 This may simply mean that, for our models, the nearest-neighbor

happened to work well with hyper-parameters optimized for deconvolution.

 It might also point at trickier issues with naively using bilinear

interpolation, where it resists high-frequency image features too

strongly.

 We don't necessarily think that either approach is the final solution

to upsampling, but they do fix the checkerboard artifacts.

### Code

---

Image Generation Results

------------------------

One case where we've found this approach to help is Generative

Adversarial Networks. Simply switching out the standard deconvolutional

layers for nearest-neighbor resize followed by convolution causes

artifacts of different frequencies to disappear.

.fix-examples > .row {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 /\*align-items: center;\*/

 margin-bottom: 10px;

}

.fix-examples > .row > .example {

 flex-grow: 1;

 width: 100%;

 margin-right: 10px;

}

.fix-examples > .row > .explanation {

 flex-grow: 2;

 width: 200%;

 margin-left: 10px;

}

.fix-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

 Deconv in last two layers.

 Other layers use resize-convolution.

*Artifacts of frequency 2 and 4.*

 Deconv only in last layer.

 Other layers use resize-convolution.

*Artifacts of frequency 2.*

 All layers use resize-convolution.

*No artifacts.*

In fact, the difference in artifacts can be seen before any training occurs.

 we can already see the artifacts:

.fix-step0-examples > .row {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 /\*align-items: center;\*/

 margin-bottom: 10px;

}

.fix-step0-examples > .row > .exampleA {

 flex-grow: 1;

 width: 100%;

 margin-right: 10px;

}

.fix-step0-examples > .row > .exampleB {

 flex-grow: 1;

 width: 100%;

 margin-right: 20px;

}

.fix-step0-examples > .row > .explanation {

 flex-grow: 2;

 width: 200%;

}

.fix-step0-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_two_0.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_two_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_one_0.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_one_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_zero_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/im_rand_zero_2.png)

 Deconvolution in last two layers.

 *Artifacts prior to any training.*

 Deconvolution only in last layer.

*Artifacts prior to any training.*

 All layers use resize-convolution.

*No artifacts before or after training.*

This suggests that the artifacts are due to this method of

generating images, rather than adversarial training.

 (It also suggests that we might be able to learn a lot about good

generator design without the slow feedback cycle of training models.)

Another reason to believe these artifacts aren't GAN specific is

that we see them in other kinds of models, and have found that they also

 go away when we switch to resize-convolution upsampling.

.style-examples > .row {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 /\*align-items: center;\*/

 margin-bottom: 10px;

}

.style-examples > .row > .example {

 flex-grow: 4;

 width: 400%;

 margin-right: 10px;

}

.style-examples > .row > .explanation {

 flex-grow: 2;

 width: 200%;

 margin-left: 10px;

}

.style-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/style_artifacts.png)

 Using deconvolution.

*Heavy checkerboard artifacts.*

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/style_clean.png)

 Using resize-convolution.

*No checkerboard artifacts.*

Forthcoming papers from the Google Brain team will demonstrate the

benefits of this technique

 in more thorough experiments and state-of-the-art results.

 (We've chosen to present this technique separately because we felt it

merited more detailed discussion, and because it cut across multiple

papers.)

---

Artifacts in Gradients

----------------------

Whenever we compute the gradients of a convolutional layer,

 we do deconvolution (transposed convolution) on the backward pass.

 This can cause checkerboard patterns in the gradient,

 just like when we use deconvolution to generate images.

The presence of high-frequency "noise" in image model gradients is

 Somehow, feature visualization methods must compensate for this noise.

For example, DeepDream [11]

 seems to cause destructive interference between artifacts in a number of ways,

.deepdream-examples {

 position: relative;

}

.deepdream-examples::after {

 overflow: visible;

 content: "";

 background-image: url("assets/pointer.svg");

 width: 27px;

 height: 27px;

 position: absolute;

 right: -51px;

 top: 0;

}

.deepdream-examples .example {

 position: relative;

}

.deepdream-examples .original {

 position: relative;

 width: 45%;

}

.deepdream-examples .original:after {

 content: "";

 display: block;

 width: 100%;

 height: 70%;

 position: absolute;

 top: -20%;

 z-index: 1;

}

.deepdream-examples .example .reticle {

 position: absolute;

 pointer-events: none;

}

.deepdream-examples .example .reticle:after {

 box-sizing: border-box;

 position: relative;

 width: 100%;

 height: 100%;

 left: -50%;

 top: -50%;

 border-radius: 50%;

 border: 1px solid white;

 display: block;

 content: '';

 background-color: rgba(255, 255, 255, 0.2);

 box-shadow: 0 0 4px rgba(0, 0, 0, 0.5);

}

.deepdream-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

.deepdream-examples .closeup img {

 -webkit-filter: contrast(150%);

 filter: contrast(150%);

}

.deepdream-examples .closeup {

 position: absolute;

 right: 34%;

 top: 0;

 width: 22.5%;

 border-radius: 50%;

 margin-bottom: 12px;

 border: 2px solid white;

 overflow: hidden;

 box-shadow: 0 2px 8px rgba(0, 0, 0, 0.3);

}

.deepdream-examples figcaption {

 position: absolute;

 right: 0%;

 top: 0;

 width: 30%;

}

.deepdream-examples .closeup img {

 position: absolute;

 width: 800%;

}

.deepdream-examples .closeup:after {

 padding-top: 100%;

 display: block;

 content: '';

}

 DeepDream only applying the neural network to a fixed position.

*Severe artifacts.*

 DeepDream applying the network to a different position each step.

*Reduced artifacts.*

(function() {

var html = d3.select(".deepdream-examples");

var original = html.selectAll(".example .original");

html

 .on("mouseleave", () => {

 console.log("out")

 original.each(resetReticle);

 });

original

 .datum(function() {

 var focus = this.getAttribute("data-focus").split(",")

 return {

 zoom: focus[0],

 x: focus[1],

 y: focus[2]

 };

 })

 .on("mouseleave", resetReticle)

 .on("mousemove", updateReticle)

 .each(resetReticle);

function updateReticle(d) {

 original.each(resetReticle);

 var x = d3.event.offsetX / this.getBoundingClientRect().width;

 var y = d3.event.offsetY / this.getBoundingClientRect().height;

 setPosition(this, x, y, d.zoom, 100);

}

function resetReticle(d) {

 setPosition(this, d.x, d.y, d.zoom, 300);

}

function setPosition(element, x, y, zoom, duration) {

 var marginX = 1 / zoom / 4;

 var marginY = 1 / zoom / 2;

 x = Math.min(Math.max(marginX, x), 1 - marginX)

 y = Math.min(Math.max(marginY, y), 1 - marginY)

 var parent = d3.select(element.parentElement);

 var reticle = parent.select(".reticle");

 var closeup = parent.select(".closeup img");

 reticle

 .style("width", 50 / zoom + "%")

 .style("height", 100 / zoom + "%")

 .transition()

 .ease(d3.ease("cubic-out"))

 .duration(duration)

 .styleTween("left", function (d, i, a) {

 var from = this.style.left,

 to = x \* 100 + "%";

 return d3.interpolateString(from, to);

 })

 .styleTween("top", function (d, i, a) {

 var from = this.style.top,

 to = y \* 100 + "%";

 return d3.interpolateString(from, to);

 })

 closeup

 .style("width", zoom \* 200 + "%")

 .style("height", zoom \* 100 + "%")

 .transition()

 .ease(d3.ease("cubic-out"))

 .duration(duration)

 .styleTween("left", function (d, i, a) {

 var from = this.style.left,

 to = -x \* 200 \* zoom + 50 + "%";

 return d3.interpolateString(from, to);

 })

 .styleTween("top", function (d, i, a) {

 var from = this.style.top,

 to = -y \* 100 \* zoom + 50 + "%";

 return d3.interpolateString(from, to);

 })

}

})();

(While some of the artifacts are our standard checkerboard pattern,

 others are a less organized high-frequency pattern.

 We believe these to be caused by max pooling.

 Max pooling was previously linked to high-frequency artifacts in [12].)

More recent work in feature visualization (eg. [13]),

Do these gradient artifacts affect GANs?

 If gradient artifacts can affect an image being optimized based on a

neural networks gradients in feature visualization,

 we might also expect it to affect the family of images parameterized

by the generator as they're optimized by the discriminator in GANs.

We've found that this does happen in some cases.

 When the generator is neither biased for or against checkerboard patterns,

 strided convolutions in the discriminator can cause them.

.discrim-examples > .row {

 display: -ms-flexbox;

 display: -webkit-flex;

 display: flex;

 /\*align-items: center;\*/

 margin-bottom: 10px;

}

.discrim-examples > .row > .exampleA {

 flex-grow: 1;

 width: 100%;

 margin-right: 10px;

}

.discrim-examples > .row > .exampleB {

 flex-grow: 1;

 width: 100%;

 margin-right: 20px;

}

.discrim-examples > .row > .explanationA {

 flex-grow: 3;

 width: 300%;

 padding-right: 20px;

}

.discrim-examples > .row > .explanationB {

 flex-grow: 3;

 width: 300%;

 padding-right: 10px;

}

.discrim-examples img {

 display: block;

 width: 100%;

 -ms-interpolation-mode: nearest-neighbor;

 image-rendering: optimizeSpeed;

 image-rendering: pixelated;

}

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride2_0.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride2_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride2_2.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride1_1.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride1_2.png)

![](Deconvolution%20and%20Checkerboard%20Artifacts_files/discrim_stride1_0.png)

 Discriminator has stride 2 convolution in first layer.

 *Strong frequency 2 artifacts.*

 Discriminator has regular convolution in first layer.

 *Very mild artifacts.*

It's unclear what the broader implications of these gradient artifacts are.

 Neither of those sounds ideal.

It seems possible that having some pixels affect the network output

 much more than others may exaggerate adversarial counter-examples.

 Because the derivative is concentrated on small number of pixels,

 small perturbations of those pixels may have outsized effects.

 We have not investigated this.

---

Conclusion

----------

The standard approach of producing images with

deconvolution — despite its successes! — has some conceptually simple

issues that lead to artifacts in produced images.

 Using a natural alternative without these issues causes the artifacts

to go away

 (Analogous arguments suggest that standard strided convolutional

layers may also have issues).

This seems like an exciting opportunity to us!

 It suggests that there is low-hanging fruit to be found in carefully

thinking through neural network architectures, even ones where we seem

to have clean working solutions.

In the meantime, we've provided an easy to use solution that

improves the quality of many approaches to generating images with neural

 networks. We look forward to seeing what people do with it, and whether

 it helps in domains like audio, where high frequency artifacts would be

 particularly problematic.