A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarially Robust Neural Style Transfer

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

 A figure in Ilyas, et. al. that struck me as particularly

 interesting

 their

 tendency to learn similar non-robust features.

 non-robust features.

 non-robust features in an image.

 Notice how far back VGG is compared to the other models.

 known to not work very well This phenomenon is discussed at length in [this

 transfer actually look more correct to humans!**

 make sense either.

 simple technique previously established in feature visualization.

 through the neural network.

 non-robust features.

A quick experiment

------------------

 Testing our hypothesis is fairly straightforward:

 Use an adversarially robust classifier for neural style transfer and see

 what happens.

 al. on their performance on neural style transfer.

 For comparison, I performed the same algorithm with a regular VGG-19

  .

 Further details can be read in a footnote

 L-BFGS was used for optimization as it showed faster convergence

 over Adam.

 relu4\_2relu4\\_2relu4\_2.

 or observed in the accompanying Colaboratory notebook.

 The results of this experiment can be explored in the diagram below.

 #style-transfer-slider.juxtapose {

 max-height: 512px;

 max-width: 512px;

 }

**Content image**

**Style image**

  Compare VGG or Robust

 ResNet

'use strict';

(function () {

 // Initialize slider

 var currentContent = 'ben';

 var currentStyle = 'scream';

 var currentLeft = 'nonrobust';

 var compareVGGCheck = document.getElementById("check-compare-vgg");

 var styleTransferSliderDiv = document.getElementById("style-transfer-slider");

 function refreshSlider() {

 while (styleTransferSliderDiv.firstChild) {

 styleTransferSliderDiv.removeChild(styleTransferSliderDiv.firstChild);

 }

 new juxtapose.JXSlider('#style-transfer-slider', [{

 src: imgPath1, // TODO: Might need to use absolute\_url?

 label: currentLeft === 'nonrobust' ? 'Non-robust ResNet50' : 'VGG'

 }, {

 src: imgPath2,

 label: 'Robust ResNet50'

 }], {

 animate: true,

 showLabels: true,

 showCredits: false,

 startingPosition: "50%",

 makeResponsive: true

 });

 }

 refreshSlider();

 compareVGGCheck.onclick = function (evt) {

 currentLeft = evt.target.checked ? 'vgg' : 'nonrobust';

 refreshSlider();

 };

 // Initialize selector

 $("#content-select").imagepicker({

 changed: function changed(oldVal, newVal, event) {

 currentContent = newVal;

 refreshSlider();

 }

 });

 $("#style-select").imagepicker({

 changed: function changed(oldVal, newVal, event) {

 currentStyle = newVal;

 refreshSlider();

 }

 });

})();

 Success!

 The robust ResNet shows drastic improvement over the regular ResNet.

 exactly the same!

 A more interesting comparison can be done between VGG-19 and the robust ResNet.

 At first glance, the robust ResNet’s outputs seem on par with VGG-19.

 Gaussian noise..

 Texture synthesized with VGG.

*Mild artifacts.*

 Texture synthesized with robust ResNet.

*Severe artifacts.*

 A comparison of artifacts between textures synthesized by VGG and ResNet.

 Interact by hovering around the images.

 This diagram was repurposed from

 by Odena, et. al.

 It is currently unclear exactly what causes these artifacts.

 One theory is that they are checkerboard artifacts

 caused by

 non-divisible kernel size and stride in the convolution layers.

 They could also be artifacts caused by the presence of max pooling layers

 in ResNet.

 problem that

 adversarial robustness solves in neural style transfer.

VGG remains a mystery

---------------------

 nets, it

 did not provide an explanation for this phenomenon.

 the box

 naturally

 more robust than other architectures.

 A few papers

 indeed show

 that VGG architectures are slightly more robust than ResNet.

 However, they also show that AlexNet, not known to work well

 for

 neural style transferAs shown by Dávid Komorowicz

 in

 this [blog post](https://dawars.me/neural-style-transfer-deep-learning/).

 , is

 *above* VGG in terms of this “natural robustness”.

 architectures fail at style transfer (or other similar algorithms

 optimization

 maximization works on robust classifiers *without*

 enforcing

 by

 previous work. In a recent chat with Chris

 Olah, he

 *without* these priors, just like style transfer!

 future

 work.

 To cite Ilyas et al.’s response, please cite their

**Response Summary**: Very interesting

 results, highlighting the effect of non-robust features and the utility of

 robust models for downstream tasks. We’re excited to see what kind of impact

 robustly trained models will have in neural network art! We were also really

 intrigued by the mysteriousness of VGG in the context of style transfer

. As such, we took a

 deeper dive which found some interesting links between robustness and style

 transfer that suggest that perhaps robustness does indeed play a role here.

**Response**: These experiments are really cool! It is interesting that

 preventing the reliance of a model on non-robust features improves performance

 on style transfer, even without an explicit task-related objective (i.e. we

 didn’t train the networks to be better for style transfer).

 We also found the discussion of VGG as a “mysterious network” really

 performance more generally. Though not a complete answer, we made a couple of

 observations while investigating further:

*Style transfer does work with AlexNet:* One wrinkle in the idea that

 the most naturally robust network — AlexNet is. However, based on our own

 testing, style transfer does seem to work with AlexNet out-of-the-box, as

 long as we use a few early layers in the network (in a similar manner to

 VGG):

 Style transfer using AlexNet, using conv\_1 through conv\_4.

 Observe that even though style transfer still works, there are checkerboard

 patterns emerging — this seems to be a similar phenomenon to the one noticed

 in the comment in the context of robust models.

 This might be another indication that these two phenomena (checkerboard

 patterns and style transfer working) are not as intertwined as previously

 thought.

*From prediction robustness to layer robustness:*  Another

 potential wrinkle here is that both AlexNet and VGG are not that

 much more robust than ResNets (for which style transfer completely fails),

 and yet seem to have dramatically better performance. To try to

 explain this, recall that style transfer is implemented as a minimization of a

 combined objective consisting of a style loss and a content loss. We found,

 however, that the network we use to compute the

 style loss is far more important

 than the one for the content loss. The following demo illustrates this — we can

 actually use a non-robust ResNet for the content loss and everything works just

 fine:

Style transfer seems to be rather

 invariant to the choice of content network used, and very sensitive

 to the style network used.

Therefore, from now on, we use a fixed ResNet-50 for the content loss as a

 control, and only worry about the style loss.

Now, note that the way that style loss works is by using the first few

 layers of the relevant network. Thus, perhaps it is not about the robustness of

 for style transfer?

 To test this hypothesis, we measure the robustness of a layer fff as:

R(f)=Ex1∼D[maxx′∥f(x′)−f(x1)∥2]Ex1,x2∼D[∥f(x1)−f(x2)∥2]

 {\mathbb{E}\_{x\_1, x\_2 \sim D}\left[\|f(x\_1) - f(x\_2)\|\_2\right]}

 R(f)=Ex1,x2∼D[∥f(x1)−f(x2)∥2]Ex1∼D[maxx′∥f(x′)−f(x1)∥2]

 Essentially, this quantity tells us how much we can change the

 representations are between images in general. We’ve plotted this value for

 the first few layers in a couple of different networks below:

The robustness R(f)R(f)R(f) of the first

 four layers of VGG16, AlexNet, and robust/standard ResNet-50

 trained on ImageNet.

 Here, it becomes clear that, the first few layers of VGG and AlexNet are

 actually almost as robust as the first few layers of the robust ResNet!

 This is perhaps a more convincing indication that robustness might have

 something to with VGG’s success in style transfer after all.

 Finally, suppose we restrict style transfer to only use a single layer of

 the network when computing the style lossUsually style transfer uses

 actually confers some style onto the image).. Again, the more

 robust layers seem to indeed work better for style transfer! Since all of the

 layers in the robust ResNet are robust, style transfer yields non-trivial

 results even using the last layer alone. Conversely, VGG and AlexNet seem to

 excel in the earlier layers (where they are non-trivially robust) but fail when

 using exclusively later (non-robust) layers:

Style transfer using a single layer. The

 names of the layers and their robustness R(f)R(f)R(f) are printed below

 each style transfer result. We find that for both networks, the robust

 layers seem to work (for the robust ResNet, every layer is robust).

 Of course, there is much more work to be done here, but we are excited

 to see further work into understanding the role of both robustness and the VGG

 in network-based image manipulation.

A Neural Algorithm of Artistic Style

Differentiable Image Parameterizations

Feature Visualization

Learning Perceptually-Aligned Representations via Adversarial Robustness

On the limited memory BFGS method for large scale optimization

Geodesics of learned representations

Batch Normalization is a Cause of Adversarial Vulnerability

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Is Robustness the Cost of Accuracy? - A Comprehensive Study on the Robustness of 18 Deep Image Classification Models

ImageNet Classification with Deep Convolutional Neural Networks

Neural Style transfer with Deep Learning

The Building Blocks of Interpretability

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'_ Adversarially Robust Neural Style Transfer.html


Abstract: By some metrics, research on Generative Adversarial Networks (GANs) has progressed substantially in the past 2 years.     Practical improvements to image synthesis models are being made  almost too quickly to keep up with:

Odena et al., 2016

Miyato et al., 2017

Zhang et al., 2018

Brock et al., 2018

 However, by other metrics, less has happened. For instance, there is

 still widespread disagreement about how GANs should be evaluated.

 Given that current image synthesis benchmarks seem somewhat

saturated, we think now is a good time to reflect on research goals for

this sub-field.

 Lists of open problems have helped other fields with this.

 We also believe that writing this article has clarified our thinking about

 We assume a fair amount of background (or willingness to look things up)

 because we reference too many results to explain all those results in detail.

What are the Trade-Offs Between GANs and other Generative Models?

-----------------------------------------------------------------

 This statement shouldn’t be taken too literally.

 that aren’t easy to describe as belonging to just one of those clusters.

 I’ve also left out VAEs entirely;

 they’re arguably no longer considered state-of-the-art at any tasks of record.

 .

 Roughly speaking, Flow Models apply a

 stack of invertible transformations to a sample from a prior

 so that exact log-likelihoods of observations can be computed.

 On the other hand, Autoregressive Models factorize the

 distribution over observations into conditional distributions

 at a time.)

 Recent research suggests that these models have different

 performance characteristics and trade-offs.

 to the model families is an interesting open question.

 At first glance, Flow Models seem like they might make GANs unnecessary.

 Flow Models allow for exact log-likelihood computation and exact inference,

 A lot of effort is spent on training GANs,

 so it seems like we should care about whether Flow Models make GANs obsolete

 image-to-image translation.

 .

 The GLOW model is trained to generate 256x256 celebrity faces using

 40 GPUs for 2 weeks and about 200 million parameters.

 In contrast, progressive GANs are trained on a similar face dataset

 to generate images with 16 times fewer pixels.

 This comparison isn’t perfect,

 For instance, it’s possible that the progressive growing

 technique could be applied to Flow Models as well.

 but it gives you a sense of things.

 Why are the Flow Models less efficient?

 We see two possible reasons:

 you will be penalized infinitely harshly!

 and this penalty is less harsh.

 Section 6.1 of  does some small experiments on expressivity, but at

 present we’re not aware of any in-depth analysis of this question.

 It turns out that Autoregressive Models can be expressed as Flow Models

 (because they are both reversible) that are not parallelizable.

 Parallelizable is somewhat imprecise in this context.

 There may be ways around this limitation though.

 Thus, GANs are parallel and efficient but not reversible,

 Flow Models are reversible and parallel but not efficient, and

 Autoregressive models are reversible and efficient, but not parallel.

|  | Parallel | Efficient | Reversible |

| --- | --- | --- | --- |

| GANs | Yes | Yes | No |

| Flow Models | Yes | No | Yes |

| Autoregressive Models | No | Yes | Yes |

 This brings us to our first open problem:

Problem 1

What are the fundamental trade-offs between GANs and other generative models?

 This has been considered for hybrid GAN/Flow Models, but we think that

 this approach is still underexplored.

 to do better than chance if the generator does this.

What Sorts of Distributions Can GANs Model?

-------------------------------------------

 Most GAN research focuses on image synthesis.

 MNIST,

 CIFAR-10,

 STL-10,

 CelebA,

 and Imagenet.

 There is some folklore about which of these datasets is ‘easiest’ to model.

 ‘extremely regular’.

 Others have noted that ‘a high number of classes is what makes

 ImageNet synthesis difficult for GANs’.

 synthesis model on CelebA generates images that seem

 Imagenet.

 trying to train GANs on ever larger and more complicated datasets.

 In particular, we’ve mostly studied how GANs perform on the datasets that

 happened to be laying around for object recognition.

 training a generative model, and then say something like ‘this dataset will be

 easy for a GAN to model, but not a VAE’.

 There has been some progress on this topic,

 but we feel that more can be done.

 We can now state the problem:

Problem 2

 We might ask the following related questions as well:

 What do we mean by ‘model the distribution’? Are we satisfied with a

 low-support representation, or do we want a true density model?

 Are there distributions that a GAN can never learn to model?

 Are there distributions that are learnable for a GAN in principle,

but are not

 efficiently learnable, for some reasonable model of

resource-consumption?

 Are the answers to these questions actually any different for GANs

than they are for other

 generative models?

 We propose two strategies for answering these questions:

 For example, in  the authors create a dataset of synthetic triangles.

 We feel that this

 angle is under-explored.

 allowing for systematic study.

 modify the assumptions to account

 for different properties of the dataset.

 what happens to them when the data distribution becomes multi-modal.

How Can we Scale GANs Beyond Image Synthesis?

---------------------------------------------

 Aside from applications like image-to-image

 translation

 and domain-adaptation

 most GAN successes have been in image synthesis.

 Attempts to use GANs beyond images have focused on three domains:

* **Text** -

 The discrete nature of text makes it difficult to apply GANs.

 This is because GANs rely on backpropagating a signal from the

discriminator through the generated content into the generator.

 There are two approaches to addressing this difficulty.

 The first is to have the GAN act only on continuous

 representations of the discrete data, as in.

 The second is use an actual discrete model and attempt to train the GAN using

 gradient estimation as in.

 Other, more sophisticated treatments exist,

 with likelihood-based language models.

* **Structured Data** -

 What about other non-euclidean structured data, like graphs?

 The study of this type of data is called geometric deep

 learning.

 GANs have had limited success here, but so have other deep learning techniques,

 so it’s hard to tell how much the GAN aspect matters.

 We’re aware of one attempt to use GANs in this space,

 which has the generator produce (and the discriminator ‘critique’) random walks

 that are meant to resemble those sampled from a source graph.

* **Audio** -

 Audio is the domain in which GANs are closest to achieving the success

 they’ve enjoyed with images.

 The first serious attempt at applying GANs to unsupervised audio synthesis

 is, in which the authors

 More recent work suggests GANs can even

 outperform autoregressive models on some perceptual metrics.

 Despite these attempts, images are clearly the easiest domain for GANs.

 This leads us to the statement of the problem:

Problem 3

How can GANs be made to perform well on non-image data?

Does scaling GANs to other domains require new training techniques,

 or does it simply require better implicit priors for each domain?

 but that it will require better implicit priors.

 in a given domain.

 For structured data or data that is not continuous, we’re less sure.

 One approach might be to make both the generator and discriminator

 large-scale computational resources.

 Finally, this problem may just require fundamental research progress.

What can we Say About the Global Convergence of GAN Training?

-------------------------------------------------------------

 the generator and discriminator for opposing objectives.

 Under certain assumptions

 These assumptions are very strict.

 The referenced paper assumes (roughly speaking) that

 the equilibrium we are looking for exists and that

 we are already very close to it.

 ,

 this simultaneous optimization

 is locally asymptotically stable.

 But all neural networks have this problem!

 This brings us to our question:

Problem 4

When can we prove that GANs are globally convergent?

Which neural network convergence results can be applied to GANs?

 There has been nontrivial progress on this question.

 Broadly speaking, there are 3 existing techniques, all of which have generated

 promising results but none of which have been studied to completion:

* **Simplifying Assumptions** -

 For example, the simplified LGQ GAN —

 linear generator, Gaussian data, and quadratic discriminator — can be

shown to be globally convergent, if optimized with a special technique

 and some additional assumptions.

 As another example,  show under different simplifying assumptions that

 It seems promising to gradually relax those assumptions to see what happens.

 For example, we could move away from unimodal distributions.

* **Use Techniques from Normal Neural Networks** -

 to answer questions about convergence of GANs.

 For instance, it’s argued in that the non-convexity

 of deep neural networks isn’t a problem,

 A fact that practitioners already kind of suspected.

 because low-quality local minima of the loss function

 become exponentially rare as the network gets larger.

 Can this analysis be ‘lifted into GAN space’?

 In fact, it seems like a generally useful heuristic to take analyses

 of deep neural networks used as classifiers and see if they apply to

GANs.

* **Game Theory** -

 The final strategy is to model GAN training using notions from game theory.

 but do so using unreasonably large resource constraints.

How Should we Evaluate GANs and When Should we Use Them?

--------------------------------------------------------

 Suggestions include:

* **Inception Score and FID** -

 Both these scores

 use a pre-trained image classifier and both have

 known issues .

 A common criticism is that these scores measure

 ‘sample quality’ and don’t really capture ‘sample diversity’.

* **MS-SSIM** -

  propose using MS-SSIM to

* **AIS** -

  propose putting a Gaussian observation model on the outputs

 of a GAN and using annealed importance sampling to estimate

 the log likelihood under this model, but  show that

 .

* **Geometry Score** -

  suggest computing geometric properties of the generated data manifold

 and comparing those properties to the real data.

* **Precision and Recall** -

  attempt to measure both the ‘precision’ and ‘recall’ of GANs.

* **Skill Rating** -

  have shown that trained GAN discriminators can contain useful information

 with which evaluation can be performed.

 Those are just a small fraction of the proposed GAN evaluation schemes.

 *when to use GANs*.

 Thus, we have bundled those two questions into one:

Problem 5

When should we use GANs instead of other generative models?

How should we evaluate performance in those contexts?

 What should we use GANs for?

 If you want an actual density model, GANs probably aren’t the best choice.

 , which means there may be substantial parts of the test

 set to which a GAN (implicitly) assigns zero likelihood.

 Rather than worrying too much about this,

 Though trying to fix this issue is a valid research agenda as well.

 GANs are likely to be well-suited to tasks with a perceptual flavor.

 all fall under this umbrella.

 How should we evaluate GANs on these perceptual tasks?

 Ideally, we would just use a human judge, but this is expensive.

 This is called a classifier two-sample test (C2STs)

 .

 (e.g., ) this will dominate the evaluation.

 One approach might be to make a critic that is blind to the dominant defect.

 creating an ordered list of the most important defects and

 critics that ignore them.

 more and more of the higher variance components.

 Finally, we could evaluate on humans despite the expense.

 This would allow us to measure the thing that we actually care about.

 the prediction is uncertain.

How does GAN Training Scale with Batch Size?

--------------------------------------------

 At first glance, it seems like the answer should be yes — after all,

 the discriminator in most GANs is just an image classifier.

 Larger batches can accelerate training if it is bottlenecked on

gradient noise.

 However, GANs have a separate bottleneck that classifiers don’t: the

 training procedure can diverge.

 Thus, we can state our problem:

Problem 6

How does GAN training scale with batch size?

How big a role does gradient noise play in GAN training?

Can GAN training be modified so that it scales better with batch size?

 Can alternate training procedures make better use of large batches?

 asynchronous SGD interacts in a special way with GAN training.

What is the Relationship Between GANs and Adversarial Examples?

---------------------------------------------------------------

 It’s well known that image classifiers suffer from adversarial examples:

 but are exponentially hard to learn robustly.

 Despite the large bodies of literature on GANs and adversarial examples,

 there doesn’t seem to be much work on how they relate.

 Thus, we can ask the question:

Problem 7

How does the adversarial robustness of the discriminator affect GAN training?

 How can we begin to think about this problem?

 Consider a fixed discriminator **D**.

 An adversarial example for **D** would exist if

 there were a generator sample **G(z)** correctly classified as fake and

 a small perturbation **p** such that **G(z) + p** is classified as real.

 a new generator **G’** where **G’(z) = G(z) + p**.

 Is this concern realistic?

  shows that deliberate attacks on generative models can work,

 but we are more worried about something you might call an ‘accidental attack’.

 There are reasons to believe that these accidental attacks are less likely.

 First, the generator is only allowed to make one gradient update before

 the discriminator is updated again.

 for every gradient step.

 We think this is a fruitful topic for further exploration.

Conditional Image Synthesis With Auxiliary Classifier GANs

Self-Attention Generative Adversarial Networks

Spectral Normalization for Generative Adversarial Networks

Large Scale GAN Training for High Fidelity Natural Image Synthesis

A style-based generator architecture for generative adversarial networks

Open Questions in Physics

Not especially famous, long-open problems which anyone can understand

Hilbert's Problems

Smale's Problems

Auto-Encoding Variational Bayes

NICE: Non-linear Independent Components Estimation

Density estimation using Real NVP

Glow: Generative Flow with Invertible 1x1 Convolutions

Normalizing Flows Tutorial

Pixel Recurrent Neural Networks

Conditional Image Generation with PixelCNN Decoders

PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

WaveNet: A Generative Model for Raw Audio

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Variational Inference with Normalizing Flows

Parallel WaveNet: Fast High-Fidelity Speech Synthesis

Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services

Flow-GAN: Bridging implicit and prescribed learning in generative models

Comparison of maximum likelihood and gan-based training of real nvps

Is Generator Conditioning Causally Related to GAN Performance?

Gradient-Based Learning Applied to Document Recognition

Learning Multiple Layers of Features from Tiny Images

An analysis of single-layer networks in unsupervised feature learning

Deep Learning Face Attributes in the Wild

ImageNet Large Scale Visual Recognition Challenge

PSA

Are GANs Created Equal? A Large-Scale Study

Disconnected Manifold Learning for Generative Adversarial Networks

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Improved training of wasserstein gans

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

MaskGAN: Better Text Generation via Filling in the ______

Long short-term memory

Geometric deep learning: going beyond Euclidean data

NetGAN: Generating Graphs via Random Walks

Synthesizing Audio with Generative Adversarial Networks

GANSynth: Adversarial Neural Audio Synthesis

OpenAI Five

Gradient descent GAN optimization is locally stable

Which Training Methods for GANs do actually Converge?

Understanding GANs: the LQG Setting

Global Convergence to the Equilibrium of GANs using Variational Inequalities

The Mechanics of n-Player Differentiable Games

Approximation and Convergence Properties of Generative Adversarial Learning

The Inductive Bias of Restricted f-GANs

On the Limitations of First-Order Approximation in GAN Dynamics

The Loss Surface of Multilayer Networks

GANGs: Generative Adversarial Network Games

Beyond Local Nash Equilibria for Adversarial Networks

An Online Learning Approach to Generative Adversarial Networks

Improved Techniques for Training GANs

GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium

A Note on the Inception Score

Towards GAN Benchmarks Which Require Generalization

Multiscale structural similarity for image quality assessment

On the Quantitative Analysis of Decoder-Based Generative Models

Annealed importance sampling

Geometry Score: A Method For Comparing Generative Adversarial Networks

Assessing Generative Models via Precision and Recall

Skill Rating for Generative Models

Discriminator Rejection Sampling

Revisiting Classifier Two-Sample Tests

Parametric Adversarial Divergences are Good Task Losses for Generative Modeling

Deconvolution and Checkerboard Artifacts

Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability

Deep reinforcement learning from human preferences

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Scaling sgd batch size to 32k for imagenet training

Train longer, generalize better: closing the generalization gap in large batch training of neural networks

Don't Decay the Learning Rate, Increase the Batch Size

An Empirical Model of Large-Batch Training

Science and research policy at the end of Moore’s law

In-datacenter performance analysis of a tensor processing unit

Improving GANs using optimal transport

Large Scale Distributed Deep Networks

Deep learning with Elastic Averaging SGD

Staleness-aware Async-SGD for Distributed Deep Learning

Faster Asynchronous SGD

The Unusual Effectiveness of Averaging in GAN Training

Intriguing properties of neural networks

Adversarial examples from computational constraints

Efficient noise-tolerant learning from statistical queries

Adversarial examples for generative models

Towards deep learning models resistant to adversarial attacks

Open Questions about Generative Adversarial Networks.html


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarially Robust Neural Style Transfer

Abstract: This article is part of a discussion of the Ilyas et al. paper         “Adversarial examples are not bugs, they are features”.         You can learn more in the                      main discussion article         .

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

 A figure in Ilyas, et. al. that struck me as particularly

 interesting

 their

 tendency to learn similar non-robust features.

![](./Adversarially Robust Neural Style Transfer_files/transferability.png)

 non-robust features.

 non-robust features in an image.

 Notice how far back VGG is compared to the other models.

 known to not work very well This phenomenon is discussed at length in [this

 transfer actually look more correct to humans!**

 make sense either.

 simple technique previously established in feature visualization.

 through the neural network.

 non-robust features.

A quick experiment

------------------

 Testing our hypothesis is fairly straightforward:

 Use an adversarially robust classifier for neural style transfer and see

 what happens.

 al. on their performance on neural style transfer.

 For comparison, I performed the same algorithm with a regular VGG-19

  .

 Further details can be read in a footnote

 L-BFGS was used for optimization as it showed faster convergence

 over Adam.

 relu4\_2relu4\\_2relu4\_2.

 or observed in the accompanying Colaboratory notebook.

 The results of this experiment can be explored in the diagram below.

 #style-transfer-slider.juxtapose {

 max-height: 512px;

 max-width: 512px;

 }

**Content image**

* ![](./Adversarially Robust Neural Style Transfer_files/ben.jpg)

* ![](./Adversarially Robust Neural Style Transfer_files/dancing.jpg)

* ![](./Adversarially Robust Neural Style Transfer_files/tubingen.jpg)

* ![](./Adversarially Robust Neural Style Transfer_files/stata.jpg)

**Style image**

* ![](./Adversarially Robust Neural Style Transfer_files/scream.jpg)

* ![](./Adversarially Robust Neural Style Transfer_files/starrynight.jpg)

* ![](./Adversarially Robust Neural Style Transfer_files/woman.jpg)

* ![](./Adversarially Robust Neural Style Transfer_files/picasso.jpg)

  Compare VGG or Robust

 ResNet

'use strict';

(function () {

 // Initialize slider

 var currentContent = 'ben';

 var currentStyle = 'scream';

 var currentLeft = 'nonrobust';

 var compareVGGCheck = document.getElementById("check-compare-vgg");

 var styleTransferSliderDiv = document.getElementById("style-transfer-slider");

 function refreshSlider() {

 while (styleTransferSliderDiv.firstChild) {

 styleTransferSliderDiv.removeChild(styleTransferSliderDiv.firstChild);

 }

 new juxtapose.JXSlider('#style-transfer-slider', [{

 src: imgPath1, // TODO: Might need to use absolute\_url?

 label: currentLeft === 'nonrobust' ? 'Non-robust ResNet50' : 'VGG'

 }, {

 src: imgPath2,

 label: 'Robust ResNet50'

 }], {

 animate: true,

 showLabels: true,

 showCredits: false,

 startingPosition: "50%",

 makeResponsive: true

 });

 }

 refreshSlider();

 compareVGGCheck.onclick = function (evt) {

 currentLeft = evt.target.checked ? 'vgg' : 'nonrobust';

 refreshSlider();

 };

 // Initialize selector

 $("#content-select").imagepicker({

 changed: function changed(oldVal, newVal, event) {

 currentContent = newVal;

 refreshSlider();

 }

 });

 $("#style-select").imagepicker({

 changed: function changed(oldVal, newVal, event) {

 currentStyle = newVal;

 refreshSlider();

 }

 });

})();

 Success!

 The robust ResNet shows drastic improvement over the regular ResNet.

 exactly the same!

 A more interesting comparison can be done between VGG-19 and the robust ResNet.

 At first glance, the robust ResNet’s outputs seem on par with VGG-19.

 Gaussian noise..

![](./Adversarially Robust Neural Style Transfer_files/vgg_texture.jpg)

![](./Adversarially Robust Neural Style Transfer_files/vgg_texture.jpg)

 Texture synthesized with VGG.

*Mild artifacts.*

![](./Adversarially Robust Neural Style Transfer_files/resnet_texture.jpg)

![](./Adversarially Robust Neural Style Transfer_files/resnet_texture.jpg)

 Texture synthesized with robust ResNet.

*Severe artifacts.*

 A comparison of artifacts between textures synthesized by VGG and ResNet.

 Interact by hovering around the images.

 This diagram was repurposed from

 by Odena, et. al.

 It is currently unclear exactly what causes these artifacts.

 One theory is that they are checkerboard artifacts

 caused by

 non-divisible kernel size and stride in the convolution layers.

 They could also be artifacts caused by the presence of max pooling layers

 in ResNet.

 problem that

 adversarial robustness solves in neural style transfer.

VGG remains a mystery

---------------------

 nets, it

 did not provide an explanation for this phenomenon.

 the box

 naturally

 more robust than other architectures.

 A few papers

 indeed show

 that VGG architectures are slightly more robust than ResNet.

 However, they also show that AlexNet, not known to work well

 for

 neural style transferAs shown by Dávid Komorowicz

 in

 this [blog post](https://dawars.me/neural-style-transfer-deep-learning/).

 , is

 *above* VGG in terms of this “natural robustness”.

 architectures fail at style transfer (or other similar algorithms

 optimization

 maximization works on robust classifiers *without*

 enforcing

 by

 previous work. In a recent chat with Chris

 Olah, he

 *without* these priors, just like style transfer!

 future

 work.

 To cite Ilyas et al.’s response, please cite their

**Response Summary**: Very interesting

 results, highlighting the effect of non-robust features and the utility of

 robust models for downstream tasks. We’re excited to see what kind of impact

 robustly trained models will have in neural network art! We were also really

 intrigued by the mysteriousness of VGG in the context of style transfer

. As such, we took a

 deeper dive which found some interesting links between robustness and style

 transfer that suggest that perhaps robustness does indeed play a role here.

**Response**: These experiments are really cool! It is interesting that

 preventing the reliance of a model on non-robust features improves performance

 on style transfer, even without an explicit task-related objective (i.e. we

 didn’t train the networks to be better for style transfer).

 We also found the discussion of VGG as a “mysterious network” really

 performance more generally. Though not a complete answer, we made a couple of

 observations while investigating further:

*Style transfer does work with AlexNet:* One wrinkle in the idea that

 the most naturally robust network — AlexNet is. However, based on our own

 testing, style transfer does seem to work with AlexNet out-of-the-box, as

 long as we use a few early layers in the network (in a similar manner to

 VGG):

![](./Adversarially Robust Neural Style Transfer_files/alexnetworks.png)

 Style transfer using AlexNet, using conv\_1 through conv\_4.

 Observe that even though style transfer still works, there are checkerboard

 patterns emerging — this seems to be a similar phenomenon to the one noticed

 in the comment in the context of robust models.

 This might be another indication that these two phenomena (checkerboard

 patterns and style transfer working) are not as intertwined as previously

 thought.

*From prediction robustness to layer robustness:*  Another

 potential wrinkle here is that both AlexNet and VGG are not that

 much more robust than ResNets (for which style transfer completely fails),

 and yet seem to have dramatically better performance. To try to

 explain this, recall that style transfer is implemented as a minimization of a

 combined objective consisting of a style loss and a content loss. We found,

 however, that the network we use to compute the

 style loss is far more important

 than the one for the content loss. The following demo illustrates this — we can

 actually use a non-robust ResNet for the content loss and everything works just

 fine:

![](./Adversarially Robust Neural Style Transfer_files/stylematters.png)

Style transfer seems to be rather

 invariant to the choice of content network used, and very sensitive

 to the style network used.

Therefore, from now on, we use a fixed ResNet-50 for the content loss as a

 control, and only worry about the style loss.

Now, note that the way that style loss works is by using the first few

 layers of the relevant network. Thus, perhaps it is not about the robustness of

 for style transfer?

 To test this hypothesis, we measure the robustness of a layer fff as:

R(f)=Ex1∼D[maxx′∥f(x′)−f(x1)∥2]Ex1,x2∼D[∥f(x1)−f(x2)∥2]

 {\mathbb{E}\_{x\_1, x\_2 \sim D}\left[\|f(x\_1) - f(x\_2)\|\_2\right]}

 R(f)=Ex1,x2∼D[∥f(x1)−f(x2)∥2]Ex1∼D[maxx′∥f(x′)−f(x1)∥2]

 Essentially, this quantity tells us how much we can change the

 representations are between images in general. We’ve plotted this value for

 the first few layers in a couple of different networks below:

![](./Adversarially Robust Neural Style Transfer_files/robustnesses.png)

The robustness R(f)R(f)R(f) of the first

 four layers of VGG16, AlexNet, and robust/standard ResNet-50

 trained on ImageNet.

 Here, it becomes clear that, the first few layers of VGG and AlexNet are

 actually almost as robust as the first few layers of the robust ResNet!

 This is perhaps a more convincing indication that robustness might have

 something to with VGG’s success in style transfer after all.

 Finally, suppose we restrict style transfer to only use a single layer of

 the network when computing the style lossUsually style transfer uses

 actually confers some style onto the image).. Again, the more

 robust layers seem to indeed work better for style transfer! Since all of the

 layers in the robust ResNet are robust, style transfer yields non-trivial

 results even using the last layer alone. Conversely, VGG and AlexNet seem to

 excel in the earlier layers (where they are non-trivially robust) but fail when

 using exclusively later (non-robust) layers:

![](./Adversarially Robust Neural Style Transfer_files/styletransfer.png)

Style transfer using a single layer. The

 names of the layers and their robustness R(f)R(f)R(f) are printed below

 each style transfer result. We find that for both networks, the robust

 layers seem to work (for the robust ResNet, every layer is robust).

 Of course, there is much more work to be done here, but we are excited

 to see further work into understanding the role of both robustness and the VGG

 in network-based image manipulation.

Very deep convolutional networks for large-scale image recognition

A Neural Algorithm of Artistic Style

Differentiable Image Parameterizations

Feature Visualization

Learning Perceptually-Aligned Representations via Adversarial Robustness

On the limited memory BFGS method for large scale optimization

Deconvolution and checkerboard artifacts

Geodesics of learned representations

Batch Normalization is a Cause of Adversarial Vulnerability

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Is Robustness the Cost of Accuracy? - A Comprehensive Study on the Robustness of 18 Deep Image Classification Models

ImageNet Classification with Deep Convolutional Neural Networks

Neural Style transfer with Deep Learning

The Building Blocks of Interpretability

Adversarially Robust Neural Style Transfer.html


Mingwei Li
Zhenge Zhao
Carlos Scheidegger

Abstract: The Grand Tour is a classic visualization technique for high-dimensional point clouds that projects  a high-dimensional dataset into two dimensions.    Over time, the Grand Tour smoothly animates its projection so that  every possible view of the dataset is (eventually) presented to the  viewer.    Unlike modern nonlinear projection methods such as t-SNE and UMAP, the Grand Tour is fundamentally a linear  method.    In this article, we show how to leverage the linearity of the Grand  Tour to enable a number of capabilities that are uniquely useful to  visualize the behavior of neural networks.      Concretely, we present three use cases of interest: visualizing the  training process as the network weights change, visualizing the  layer-to-layer behavior as the data goes through the network and  visualizing both how adversarial examples are crafted and how they fool a neural network.

 a high-dimensional dataset into two dimensions.

 Over time, the Grand Tour smoothly animates its projection so that

every possible view of the dataset is (eventually) presented to the

viewer.

 method.

 In this article, we show how to leverage the linearity of the Grand

Tour to enable a number of capabilities that are uniquely useful to

visualize the behavior of neural networks.

 Concretely, we present three use cases of interest: visualizing the

training process as the network weights change, visualizing the

layer-to-layer behavior as the data goes through the network and

Introduction

------------

 Deep neural networks often achieve best-in-class performance in

supervised learning contests such as the ImageNet Large Scale Visual

Recognition Challenge (ILSVRC).

 In this article, we present a method to visualize the responses of a

neural network which leverages properties of deep neural networks and

properties of the *Grand Tour*.

 As we will show, this data-visual correspondence is central to the

method we present, especially when compared to other non-linear

projection methods like UMAP and t-SNE.

 These kinds of visualizations are useful to elucidate the activation

patterns of a neural network for a single example, but they might offer

less insight about the relationship between different examples,

different states of the network as it’s being trained, or how the data

in the example flows through the different layers of a single network.

 Therefore, we instead aim to enable visualizations of the *context around*

 our objects of interest: what is the difference between the present

training epoch and the next one? How does the classification of a

network converge (or diverge) as the image is fed through the network?

 Linear methods are attractive because they are particularly easy to

reason about.

 The Grand Tour works by generating a random, smoothly changing

rotation of the dataset, and then projecting the data to the

two-dimensional screen: both are linear processes.

 Although deep neural networks are clearly not linear processes, they

often confine their nonlinearity to a small set of operations, enabling

us to still reason about their behavior.

 Our proposed method better preserves context by providing more

 consistency: it should be possible to know *how the visualization

 would change, if the data had been different in a particular

 way*.

Working Examples

----------------

 To illustrate the technique we will present, we trained deep neural

 network models (DNNs) with 3 common image classification datasets:

 MNIST

 MNIST contains grayscale images of 10 handwritten digits

 Image credit to <https://en.wikipedia.org/wiki/File:MnistExamples.png>

,

 fashion-MNIST

 Fashion-MNIST contains grayscale images of 10 types of fashion items:

 and CIFAR-10

 CIFAR-10 contains RGB images of 10 classes of objects

 Image credit to <https://www.cs.toronto.edu/~kriz/cifar.html>

.

 While our architecture is simpler and smaller than current DNNs, it’s

still indicative of modern networks, and is complex enough to

demonstrate both our proposed techniques and shortcomings of typical

approaches.

 The following figure presents a simple functional diagram of the

neural network we will use throughout the article. The neural network is

 a sequence of linear (both convolutional

 A convolution calculates weighted sums of regions in the input.

 For example

 ![](Visualizing%20Neural%20Networks%20with%20the%20Grand%20Tour_files/conv.gif)

  and fully-connected

 A fully-connected layer computes output neurons as weighted sum of

input neurons. In matrix form, it is a matrix that linearly transforms

the input vector into the output vector.

 ), max-pooling, and ReLU

 Image credit to <https://pytorch.org/docs/stable/nn.html#relu>

 layers, culminating in a softmax

 layer.

Neural network opened. The colored blocks are building-block

 functions (i.e. neural network layers), the gray-scale heatmaps are

either the input image or intermediate activation vectors after some

layers.

 Even though neural networks are capable of incredible feats of

classification, deep down, they really are just pipelines of relatively

simple functions.

 For images, the input is a 2D array of scalar values for gray scale

images or RGB triples for colored images.

 -dimensional vector.

 Similarly, the intermediate values after any one of the functions in

composition, or activations of neurons after a layer, can also be seen

as vectors in Rn\mathbb{R}^nRn, where nnn

 is the number of neurons in the layer.

 The softmax, for example, can be seen as a 10-vector whose values are

positive real numbers that sum up to 1.

 This vector view of data in neural network not only allows us

represent complex data in a mathematically compact form, but also hints

us on how to visualize them in a better way.

 Most of the simple functions fall into two categories: they are either

 linear transformations of their inputs (like fully-connected layers or

convolutional layers), or relatively simple non-linear functions that

work component-wise (like sigmoid activations

 Image credit to <https://en.wikipedia.org/wiki/Sigmoid_function>

 or ReLU activations).

 Some operations, notably max-pooling

 Max-pooling calculates maximum of a region in the input. For example

 The above figure helps us look at a single image at a time; however,

it does not provide much context to understand the relationship between

layers, between different examples, or between different class labels.

For that, researchers often turn to more sophisticated visualizations.

Using Visualization to Understand DNNs

--------------------------------------

 Let’s start by considering the problem of visualizing the training

process of a DNN.

 When training neural networks, we optimize parameters in the function

to minimize a scalar-valued loss function, typically through some form

of gradient descent.

 We want the loss to keep decreasing, so we monitor the whole history

of training and testing losses over rounds of training (or “epochs”), to

 make sure that the loss decreases over time.

 The following figure shows a line plot of the training loss for the

MNIST classifier.

014215075990.10.512Training EpochLoss

 Although its general trend meets our expectation as the loss steadily

decreases, we see something strange around epochs 14 and 21: the curve

goes almost flat before starting to drop again.

 What happened? What caused that?

014215075990.050.10.5123Training EpochLoss

 loss like above, we see that the two drops were caused by the classses 1

 and 7; the model learns different classes at very different times in

the training process.

 Although the network learns to recognize digits 0, 2, 3, 4, 5, 6, 8

and 9 early on, it is not until epoch 14 that it starts successfully

recognizing digit 1, or until epoch 21 that it recognizes digit 7.

 If we knew ahead of time to be looking for class-specific error rates,

 then this chart works well. But what if we didn’t really know what to

look for?

 examples at once, looking

 to find patterns like class-specific behavior, and other patterns

besides.

 Should there be only two neurons in that layer, a simple

two-dimensional scatter plot would work.

 However, the points in the softmax layer for our example datasets are

10 dimensional (and in larger-scale classification problems this number

can be much larger).

 We need to either show two dimensions at a time (which does not scale

well as the number of possible charts grows quadratically),

### The State-of-the-art Dimensionality Reduction is Non-linear

 Modern dimensionality reduction techniques such as t-SNE and UMAP are

capable of impressive feats of summarization, providing two-dimensional

images where similar points tend to be clustered together very

effectively.

 However, these methods are not particularly good to understand the

behavior of neuron activations at a fine scale.

 Consider the aforementioned intriguing feature about the different

learning rate that the MNIST classifier has on digit 1 and 7: the

network did not learn to recognize digit 1 until epoch 14, digit 7 until

 epoch 21.

 We compute t-SNE, Dynamic t-SNE,

 and UMAP projections of the epochs where the phenomenon we described

happens.

 Consider now the task of identifying this class-specific behavior

during training. As a reminder, in this case, the strange behavior

happens with digits 1 and 7, around epochs 14 and 21 respectively.

 While the behavior is not particularly subtle&emdash;digit goes

from misclassified to correctly classified&emdash; it is quite hard

to notice it in any of the plots below.

 Only on careful inspection we can notice that (for example) in the

UMAP plot, the digit 1 which clustered in the bottom in epoch 13 becomes

 a new tentacle-like feature in epoch 14.

 // let sm1 = createSmallMultiple('#smallmultiple1',

 // [13,14,15, 20,21,22], ['t-SNE', 'Dynamic t-SNE', 'UMAP'],

 // 'mnist', true, highlight\_digits);

 let sm1 = createSmallMultiple('#smallmultiple1',

 [13,14,15, 20,21,22], ['t-SNE', 'Dynamic t-SNE', 'UMAP'],

 'mnist', true);

Softmax activations of the MNIST classifier with non-linear

dimensionality reduction. Use the buttons on the right to highlight

digits 1 and 7 in the plot, or drag rectangles around the charts to

select particular point subsets to highlight in the other charts.

 One reason that non-linear embeddings fail in elucidating this

phenomenon is that, for the particular change in the data, the fail the

principle of *data-visual correspondence* .

 More concretely, the principle states that specific visualization tasks

 should be modeled as functions that change the data; the visualization

sends this change from data to visuals, and

 we can study the extent to which the visualization changes are easily

perceptible.

 Ideally, we want the changes in data and visualization to *match in magnitude*:

 a barely noticeable change in visualization should be due to the

smallest possible change in data, and a salient change in visualization

should reflect a significant one in data.

 points in the visualization move dramatically.

 For both UMAP and t-SNE, the position of each single point depends

non-trivially on the whole data distribution in such embedding

algorithms.

 This property is not ideal for visualization because it fails the

 Non-linear embeddings that have non-convex objectives also tend to be

sensitive to initial conditions.

 For example, in MNIST, although the neural network starts to stabilize

 on epoch 30, t-SNE and UMAP still generate quite different projections

between epochs 30, 31 and 32 (in fact, all the way to 99).

 Temporal regularization techniques (such as Dynamic t-SNE) mitigate

these consistency issues, but still suffer from other interpretability

issues.

 Now, let’s consider another task, that of identifying classes which

the neural network tends to confuse.

 For this example, we will use the Fashion-MNIST dataset and

classifier, and consider the confusion among sandals, sneakers and ankle

 boots.

 If we know ahead of time that these three classes are likely to

confuse the classifier, then we can directly design an appropriate

linear projection, as can be seen in the last row of the following

figure (we found this particular projection using both the Grand Tour

and the direct manipulation technique we later describe). The pattern in

 this case is quite salient, forming a triangle.

 T-SNE, in contrast, incorrectly separates the class clusters (possibly

 because of an inappropriately-chosen hyperparameter).

 UMAP successfully isolates the three classes, but even in this case

it’s not possible to distinguish between three-way confusion for the

classifier in epochs 5 and 10 (portrayed in a linear method by the

presence of points near the center of the triangle), and multiple

two-way confusions in later epochs (evidences by an “empty” center).

 sm3 = createSmallMultiple('#smallmultiple3',

 [2,5,10,20,50,99], ['t-SNE', 'UMAP', 'Linear'],

 'fashion-mnist', true,

 highlight\_shoes\_button,

 highlight\_shoes,

 );

Highlight shoes

Three-way confusion in fashion-MNIST. Notice that in

contrast to non-linear methods, a carefully-constructed linear

projection can offer a better visualization of the classifier behavior.

Linear Methods to the Rescue

----------------------------

 When given the chance, then, we should prefer methods for which

changes in the data produce predictable, visually salient changes in the

 result, and linear dimensionality reductions often have this property.

 Here, we revisit the linear projections described above in an

interface where the user can easily navigate between different training

epochs.

 In addition, we introduce another useful capability which is only

available to linear methods, that of direct manipulation.

 space will be projected to.

 In the context of projecting the final classification layer, this is

especially simple to interpret: they are the destinations of an input

that is classified with 100% confidence to any one particular class.

 If we provide the user with the ability to change these vectors by

dragging around user-interface handles, then users can intuitively set

up new linear projections.

 This setup provides additional nice properties that explain the

salient patterns in the previous illustrations.

 For example, because projections are linear and the coefficients of

vectors in the classification layer sum to one, classification outputs

that are halfway confident between two classes are projected to vectors

that are halfway between the class handles.

 From

 this linear projection, we can easily identify the learning of

 digit 1 on

 epoch 14 and

 digit 7 on

 epoch 21.

 This linear projection clearly shows model’s confusion among

 sandals,

 sneakers, and

 ankle boots.

 Similarly, this projection shows the true three-way confusion about

 pullovers,

 coats, and

 shirts.

 (The shirts are also get confused with

 t-shirts/tops. )

 Both projections are found by direct manipulations.

 Examples falling between classes indicate that the model has trouble

distinguishing the two, such as sandals vs. sneakers, and sneakers vs.

ankle boot classes.

 Note, however, that this does not happen as much for sandals vs. ankle

 boots: not many examples fall between these two classes.

 Moreover, most data points are projected close to the edge of the

triangle.

 This tells us that most confusions happen between two out of the three

 classes, they are really two-way confusions.

 Within the same dataset, we can also see pullovers, coats and shirts

filling a triangular *plane*.

 This is different from the sandal-sneaker-ankle-boot case, as examples

 not only fall on the boundary of a triangle, but also in its interior: a

 true three-way confusion.

 Similarly, in the CIFAR-10 dataset we can see confusion between dogs

and cats, airplanes and ships.

 The mixing pattern in CIFAR-10 is not as clear as in fashion-MNIST,

because many more examples are misclassified.

 This linear projection clearly shows model’s confusion between

 cats and

 dogs.

 Similarly, this projection shows the confusion about

 airplanes and

 ships.

 Both projections are found by direct manipulations.

The Grand Tour

--------------

 In the previous section, we took advantage of the fact that we knew

which classes to visualize.

 That meant it was easy to design linear projections for the particular

 tasks at hand.

 But what if we don’t know ahead of time which projection to choose

from, because we don’t quite know what to look for?

 Principal Component Analysis (PCA) is the quintessential linear

dimensionality reduction method,

 choosing to project the data so as to preserve the most variance

possible.

 However, the distribution of data in softmax layers often has similar

variance along many axis directions, because each axis concentrates a

similar number of examples around the class vector.We

 are assuming a class-balanced training dataset. Nevertheless, if the

training dataset is not balanced, PCA will prefer dimensions with more

examples, which might not be help much either.

 As a result, even though PCA projections are interpretable and

consistent through training epochs, the first two principal components

of softmax activations are not substantially better than the third.

 So which of them should we choose?

 Instead of PCA, we propose to visualize this data by smoothly

animating random projections, using a technique called the Grand Tour.

Starting with a random velocity, it smoothly rotates data points around

the origin in high dimensional space, and then projects it down to 2D

for display.

Here are some examples of how Grand Tour acts on some (low-dimensional)

objects:

* On a square, the Grand Tour rotates it with a constant angular velocity.

Grand tours of a square, a cube and a tesseract

### The Grand Tour of the Softmax Layer

 We first look at the Grand Tour of the softmax layer.

 The Grand Tour of softmax layer in the last (99th) epoch, with

 MNIST,

 fashion-MNIST or

 CIFAR-10 dataset.

 The Grand Tour of the softmax layer lets us qualitatively assess the

performance of our model.

 In the particular case of this article, since we used comparable

architectures for three datasets, this also allows us to gauge the

relative difficulty of classifying each dataset.

 We can see that data points are most confidently classified for the

MNIST dataset, where the digits are close to one of the ten corners of

the softmax space. For Fashion-MNIST or CIFAR-10, the separation is not

as clean, and more points appear *inside* the volume.

### The Grand Tour of Training Dynamics

 Linear projection methods naturally give a formulation that is

independent of the input points, allowing us to keep the projection

fixed while the

 data changes.

 To recap our working example, we trained each of the neural networks

for 99 epochs and recorded the entire history of neuron activations on a

 subset of training and testing examples. We can use the Grand Tour,

then, to visualize the actual training process of these networks.

 In the beginning when the neural networks are randomly initialized,

all examples are placed around the center of the softmax space, with

equal weights to each class.

 Through training, examples move to class vectors in the softmax space.

 The Grand Tour also lets us

 compare visualizations of the training and testing data, giving us a

qualitative assessment of over-fitting.

 In the MNIST dataset, the trajectory of testing images through

training is consistent with the training set.

 Data points went directly toward the corner of its true class and all

classes are stabilized after about 50 epochs.

 On the other hand, in CIFAR-10 there is an *inconsistency*

between the training and testing sets. Images from the testing set keep

oscillating while most images from training converges to the

corresponding class corner.

 In epoch 99, we can clearly see a difference in distribution between

these two sets.

 This signals that the model overfits the training set and thus does

not generalize well to the testing set.

 With this view of CIFAR-10 ,

 the color of points are more mixed in testing (right) than training

(left) set, showing an over-fitting in the training process.

 Compare

 CIFAR-10

 with

 MNIST

 or fashion-MNIST,

 where there is less difference between training and testing sets.

### The Grand Tour of Layer Dynamics

 Given the presented techniques of the Grand Tour and direct

manipulations on the axes, we can in theory visualize and manipulate any

 intermediate layer of a neural network by itself. Nevertheless, this

is not a very satisfying approach, for two reasons:

* In the same way that we’ve kept the projection fixed as the

training data changed, we would like to “keep the projection fixed”, as

the data moves through the layers in the neural network. However, this

is not straightforward. For example, different layers in a neural

network have different dimensions. How do we connect rotations of one

layer to rotations of the other?

* The class “axis handles” in the softmax layer convenient, but

that’s only practical when the dimensionality of the layer is relatively

 small.

 With hundreds of dimensions, for example, there would be too many

axis handles to naturally interact with.

 In addition, hidden layers do not have as clear semantics as the

softmax layer, so manipulating them would not be as intuitive.

 To address the first problem, we will need to pay closer attention to

the way in which layers transform the data that they are given.

 To see how a linear transformation can be visualized in a particularly

 ineffective way, consider the following (very simple) weights

 A=[−1,00,−1]

 A = \begin{bmatrix}

 -1, 0 \\

 0, -1

 \end{bmatrix}

 A=[−1,00,−1]

 Imagine that we wish to visualize the behavior of network as the data

 xt=(1−t)⋅x0+t⋅x1=(1−2t)⋅x0

 x\_t = (1-t) \cdot x\_0 + t \cdot x\_1 = (1-2t) \cdot x\_0

 xt=(1−t)⋅x0+t⋅x1=(1−2t)⋅x0

 for t∈[0,1].t \in [0,1].t∈[0,1].

 Effectively, this strategy reuses the linear projection coefficients

from one layer to the next. This is a natural thought, since they have

the same dimension.

 However, notice the following: the transformation given by A is a

 if only that transformation operated on the negative values of the

entries.

 In addition, since the Grand Tour has a rotation itself built-in, for

 into account.

 In effect, the naive interpolation fails the principle of data-visual

correspondence: a simple change in data (negation in 2D/180 degree

rotation) results in a drastic change in visualization (all points cross

 the origin).

 This observation points to a more general strategy: when designing a

visualization, we should be as explicit as possible about which parts of

 the input (or process) we seek to capture in our visualizations.

 We should seek to explicitly articulate what are purely

representational artifacts that we should discard, and what are the real

 features a visualization we should *distill* from the

representation.

 Here, we claim that rotational factors in linear transformations of

neural networks are significantly less important than other factors such

 as scalings and nonlinearities.

 As we will show, the Grand Tour is particularly attractive in this

case because it is can be made to be invariant to rotations in data.

 As a result, the rotational components in the linear transformations

of a neural network will be explicitly made invisible.

 - a simple (coordinate-wise) scaling.

 This is explicitly saying that any linear operation (whose matrix is

represented in standard bases) is a scaling operation with appropriately

 chosen orthonormal bases on both sides.

 So the Grand Tour provides a natural, elegant and computationally

 layers are also linear. One can instantly see that by forming the

linear transformations between flattened feature maps, or by taking the

circulant structure of convolutional layers directly into account

 (For the following portion, we reduce the number of data points to 500

 and epochs to 50, in order to reduce the amount of data transmitted in a

 web-based demonstration.)

 With the linear algebra structure at hand, now we are able to trace

behaviors and patterns from the softmax back to previous layers.

 In fashion-MNIST, for example, we observe a separation of shoes

(sandals, sneakers and ankle boots as a group) from all other classes in

 the softmax layer.

 Tracing it back to earlier layers, we can see that this separation

happened as early as layer 5:

### The Grand Tour of Adversarial Dynamics

 as they are processed by a neural network.

 For this illustration, we use the MNIST dataset, and we adversarially

add perturbations to 89 digit 8s to fool the network into thinking they

are 0s.

 Previously, we either animated the training dynamics or the layer

dynamics.

 We fix a well-trained neural network, and visualize the training

process of adversarial examples, since they are often themselves

generated by an optimization process. Here, we used the Fast Gradient

Sign method.

 Again, because the Grand Tour is a linear method, the change in the

positions of the adversarial examples over time can be faithfully

attributed to changes in how the neural network perceives the images,

rather than potential artifacts of the visualization.

 Let us examine how adversarial examples evolved to fool the network:

 From this view of softmax, we can see how

 adversarial examples

 evolved from 8s

 into 0s.

 Through this adversarial training, the network eventually claims, with

 high confidence, that the inputs given are all 0s.

 If we stay in the softmax layer and slide though the adversarial

training steps in the plot, we can see adversarial examples move from a

high score for class 8 to a high score for class 0.

 Although all adversarial examples are classified as the target class

(digit 0s) eventually, some of them detoured somewhere close to the

centroid of the space (around the 25th epoch) and then moved towards the

 target.

 Comparing the actual images of the two groups, we see those that those

 “detouring” images tend to be noisier.

 More interesting, however, is what happens in the intermediate layers.

Discussion

----------

### Limitations of the Grand Tour

 Early on, we compared several state-of-the-art dimensionality

reduction techniques with the Grand Tour, showing that non-linear

methods do not have as many desirable properties as the Grand Tour for

understanding the behavior of neural networks.

 However, the state-of-the-art non-linear methods come with their own

strength.

 Whenever geometry is concerned, like the case of understanding

multi-way confusions in the softmax layer, linear methods are more

interpretable because they preserve certain geometrical structures of

data in the projection.

 When topology is the main focus, such as when we want to cluster the

data or we need dimensionality reduction for downstream models that are

less sensitive to geometry, we might choose non-linear methods such as

UMAP or t-SNE for they have more freedom in projecting the data, and

will generally make better use of the fewer dimensions available.

### The Power of Animation and Direct Manipulation

 When comparing linear projections with non-linear dimensionality

reductions, we used small multiples to contrast training epochs and

dimensionality reduction methods.

 The Grand Tour, on the other hand, uses a single animated view.

 When comparing small multiples and animations, there is no general

consensus on which one is better than the other in the literature,

aside.

 between small multiples and animated plots.

 Regardless of these concerns, in our scenarios, the use of animation

comes naturally from the direct manipulation and the existence of a

continuum of rotations for the Grand Tour to operate in.

### Non-sequential Models

 In our work we have used models that are purely “sequential”, in the

sense that the layers can be put in numerical ordering, and that the

activations for

 With our technique, one can visualize neuron activations on each such

branch, but additional research is required to incorporate multiple

branches directly.

### Scaling to Larger Models

 Modern architectures are also wide. Especially when convolutional

layers are concerned, one could run into issues with scalability if we

see such layers as a large sparse matrix acting on flattened

multi-channel images.

 For the sake of simplicity, in this article we brute-forced the

computation of the alignment of such convolutional layers by writing out

 their explicit matrix representation.

 However, the singular value decomposition of multi-channel 2D

function toggle(event, id){

 let caller = d3.select(event.target); //DOM that received the event

 let callerIsActive = caller.classed('clickable-active');

 let selection = d3.select(id); //DOM to be toggled

 let isHidden = selection.classed('hidden');

 selection.classed('hidden', !isHidden);

}

Technical Details

------------------

This section presents the technical details necessary to implement the

direct manipulation of axis handles and data points, as well as how to

implement the projection consistency technique for layer transitions.

### Notation

 In this section, our notational convention is that data points are

represented as row vectors.

 An entire dataset is laid out as a matrix, where each row is a data

point, and each column represents a different feature/dimension.

 As a result, when a linear transformation is applied to the data, the

row vectors (and the data matrix overall) are left-multiplied by the

transformation matrix.

 This has a side benefit that when applying matrix multiplications in a

 chain, the formula reads from left to right and aligns with a

commutative diagram.

X↦MY

 X

 \overset{M}{\mapsto}

 Y

X↦MY

X↦U↦Σ↦VTY

 X

 \overset{U}{\mapsto}

 \overset{\Sigma}{\mapsto}

 \overset{V^T}{\mapsto} Y

X↦U↦Σ↦VTY

nicely aligns with the formula.

###

Direct Manipulation

 The direct manipulations we presented earlier provide explicit control

 over the possible projections for the data points.

 We provide two modes: directly manipulating class axes (the “axis

mode”), or directly manipulating a group of data points through their

centroid (the “data point mode”).

 we may prefer one mode than the other.

 We will see that the axis mode is a special case of data point mode,

because we can view an axis handle as a particular “fictitious” point in

 the dataset.

 Because of its simplicity, we will first introduce the axis mode.

####

 The Axis Mode

 The implied semantics of direct manipulation is that when a user

drags an UI element (in this case, an axis handle), they are signaling

to the system that they wished that the corresponding

 data point had been projected to the location where the UI element

was dropped, rather than where it was dragged from.

 In our case the overall projection is a rotation (originally

determined by the Grand Tour), and an arbitrary user manipulation might

not necessarily generate a new projection that is also a rotation. Our

goal, then, is to find a new rotation which satisfies the user request

and is close to the previous state of the Grand Tour projection, so that

 the resulting state satisfies the user request.

ei↦GTei~↦π2(xi,yi)

 e\_i \overset{GT}{\mapsto} \tilde{e\_i} \overset{\pi\_2}{\mapsto} (x\_i, y\_i)

ei↦GTei~↦π2(xi,yi)

 The coordinate of the handle becomes:

 Recall that the convention is that vectors are in row form and

linear transformations are matrices that are multiplied on the right.

.

 GT~←GT\widetilde{GT} \leftarrow GTGT

←GT

GT~i,1←GTi,1+dx\widetilde{GT}\_{i,1} \leftarrow GT\_{i,1} + dxGT

i,1←GTi,1+dx

GT~i,2←GTi,2+dy\widetilde{GT}\_{i,2} \leftarrow GT\_{i,2} + dyGT

i,2←GTi,2+dy

 However, GT~\widetilde{GT}GT

 is not orthogonal for arbitrary (dx,dy)(dx, dy)(dx,dy).

 In order to find an approximation to GT~\widetilde{GT}GT

, with the ithi^{th}ith row considered first in the Gram-Schmidt process:

)

  .

####

 The Data Point Mode

 We now explain how we directly manipulate data points.

 Technically speaking, this method only considers one point at a time.

 Next, as the first step in Gram-Schmidt, we normalized this row:

 GTi(new):=normalize(GT~i)=normalize(ei~+Δ~)

 GTi(new):=normalize(GT

i)=normalize(ei~+Δ~)

 in high dimensional space.

 This intuition is precisely how we implemented our direct manipulation

 on arbitrary data points, which we will specify as below.

 Generalizing this observation from axis handle to arbitrary data

point, we want to find the rotation that moves the centroid of a

selected subset of data points c~\tilde{c}c~ to

 c~(new):=(c~+Δ~)⋅∣∣c~∣∣/∣∣c~+Δ~∣∣

 c~(new):=(c~+Δ~)⋅∣∣c~∣∣/∣∣c~+Δ~∣∣

 First, the angle of rotation can be found by their cosine similarity:

 θ=arccos(⟨c~,c~(new)⟩∣∣c~∣∣⋅∣∣c~(new)∣∣) \theta = \textrm{arccos}(

 )θ=arccos(∣∣c~∣∣⋅∣∣c~(new)∣∣⟨c~,c~(new)⟩)

 Next, to find the matrix form of the rotation, we need a convenient basis.

 c~⊥(new):=c~−∣∣c~∣∣⋅cosθc~(new)∣∣c~(new)∣∣

 \tilde{c}^{(new)}\_{\perp}

 c~⊥(new):=c~−∣∣c~∣∣⋅cosθ∣∣c~(new)∣∣c~(new)

Q:=[⋯normalize(c~)⋯⋯normalize(c~⊥(new))⋯P]

 Q :=

 \begin{bmatrix}

 \cdots \textsf{normalize}(\tilde{c}) \cdots \\

 \cdots \textsf{normalize}(\tilde{c}^{(new)}\_{\perp}) \cdots \\

 P

 \end{bmatrix}

 Q:=⎣⎡⋯normalize(c~)⋯⋯normalize(c~⊥(new))⋯P⎦⎤

 where PPP completes the remaining space.

 ρ=QT[cosθsinθ00⋯−sinθcosθ00⋯00⋮⋮I]Q=:QTR1,2(θ)Q

 \rho = Q^T

 \begin{bmatrix}

 \cos \theta& \sin \theta& 0& 0& \cdots\\

 -\sin \theta& \cos \theta& 0& 0& \cdots\\

 0& 0& \\

 \vdots& \vdots& & I& \\

 \end{bmatrix}

 Q

 =: Q^T R\_{1,2}(\theta) Q

 ρ=QT⎣⎢⎢⎢⎢⎡cosθ−sinθ0⋮sinθcosθ0⋮0000I⋯⋯⎦⎥⎥⎥⎥⎤Q=:QTR1,2(θ)Q

 GT(new):=GT⋅ρ

 GT^{(new)} := GT \cdot \rho

 GT(new):=GT⋅ρ

 repeatedly take a random vector, find its orthogonal component to the

span of the current basis vectors and add it to the basis set.

###

 Layer Transitions

####

 ReLU Layers

 Since ReLU does not change the dimensionality and the function is taken

 coordinate wise, we can animate the transition by a simple linear

interpolation: for a time parameter t∈[0,1]t \in [0,1]t∈[0,1],

 X(l−1)→l(t):=(1−t)Xl−1+tXl

 X^{(l-1) \to l}(t) := (1-t) X^{l-1} + t X^{l}

 X(l−1)→l(t):=(1−t)Xl−1+tXl

####

 Linear Layers

 Transitions between linear layers can seem complicated, but as we will

 show, this comes from choosing mismatching bases on either side of the

transition.

 M=UΣVTM = U \Sigma V^TM=UΣVT

 The Grand Tour has a single parameter that represents the current

rotation of the dataset. Since our goal is to keep the transition

consistent, we notice that UUU and VTV^TVT

 have essentially no significance - they are just rotations to the view

that can be exactly “canceled” by changing the rotation parameter of the

 Grand Tour in either layer.

 Given Xl=Xl−1UΣVTX^{l} = X^{l-1} U \Sigma V^TXl=Xl−1UΣVT, we have

 (XlV)=(Xl−1U)Σ

 (X^{l}V) = (X^{l-1}U)\Sigma

 (XlV)=(Xl−1U)Σ

 For a time parameter t∈[0,1]t \in [0,1]t∈[0,1],

 X(l−1)→l(t):=(1−t)(Xl−1U)+t(XlV)=(1−t)(Xl−1U)+t(Xl−1UΣ)

 X(l−1)→l(t):=(1−t)(Xl−1U)+t(XlV)=(1−t)(Xl−1U)+t(Xl−1UΣ)

####

 Convolutional Layers

 Convolutional layers can be represented as special linear layers.

 With a change of representation, we can animate a convolutional layer

like the previous section.

 For 2D convolutions this change of representation involves flattening

the input and output, and repeating the kernel pattern in a sparse

matrix M∈Rm×nM \in \mathbb{R}^{m \times n}M∈Rm×n, where mmm and nnn

 are the dimensionalities of the input and output respectively.

 This change of representation is only practical for a small

dimensionality (e.g. up to 1000), since we need to solve SVD for linear

layers.

 However, the singular value decomposition of multi-channel 2D

####

 Max-pooling Layers

Conclusion

----------

 As powerful as t-SNE and UMAP are, they often fail to offer the

correspondences we need, and such correspondences can come,

surprisingly, from relatively simple methods like the Grand Tour. The

Grand Tour method we presented is particularly useful when direct

manipulation from the user is available or desirable.

 We believe that it might be possible to design methods that highlight

the best of both worlds, using non-linear dimensionality reduction to

create intermediate, relatively low-dimensional representations of the

activation layers, and using the Grand Tour and direct manipulation to

compute the final projection.

The grand tour: a tool for viewing multidimensional data

Visualizing data using t-SNE

Umap: Uniform manifold approximation and projection for dimension reduction

Intriguing properties of neural networks

The mythos of model interpretability

Visualizing dataflow graphs of deep learning models in tensorflow

An algebraic process for visualization design

Feature visualization

MNIST handwritten digit database

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Learning multiple layers of features from tiny images

Rectified linear units improve restricted boltzmann machines

Visualizing time-dependent data using dynamic t-SNE

How to use t-sne effectively

We Recommend a Singular Value Decomposition

The singular values of convolutional layers

Explaining and harnessing adversarial examples

Animation, small multiples, and the effect of mental map preservation in dynamic graphs

Animation: can it facilitate?

Highway networks

Going deeper with convolutions

Visualizing Neural Networks with the Grand Tour.html


Jochen Görtler
Rebecca Kehlbeck
Oliver Deussen

 Even if you have spent some time reading about machine learning,

chances are that you have never heard of Gaussian processes.

 And if you have, rehearsing the basics is always a good way to

refresh your memory.

 With this blog post we want to give an introduction to Gaussian

processes and make the mathematical intuition behind them more

approachable.

 Gaussian processes are a powerful tool in the machine learning toolbox.

 Their most obvious area of application is *fitting* a function to the data.

 The mean of this probability distribution then represents the most

probable characterization of the data.

 Furthermore, using a probabilistic approach allows us to incorporate

 the confidence of the prediction into the regression result.

 We will first explore the mathematical foundation that Gaussian

processes are built on — we invite you to follow along using the

interactive figures and hands-on examples.

 They help to explain the impact of individual components, and show

the flexibility of Gaussian processes.

 After following this article we hope that you will have a visual

intuition on how Gaussian processes work and how you can configure them

for different types of data.

Multivariate Gaussian distributions

-----------------------------------

 distribution) is the basic building block of Gaussian processes.

 In particular, we are interested in the multivariate case of this

distribution, where each random variable is distributed normally and

their joint distribution is also Gaussian.

 The mean vector μ\muμ describes the expected value of the distribution.

 Each of its components describes the mean of the corresponding dimension.

 The covariance matrix is always symmetric and positive semi-definite.

X=[X1X2⋮Xn]∼N(μ,Σ)

 X=⎣⎢⎢⎡X1X2⋮Xn⎦⎥⎥⎤∼N(μ,Σ)

Σ=Cov(Xi,Xj)=E[(Xi−μi)(Xj−μj)T]

 Σ=Cov(Xi,Xj)=E[(Xi−μi)(Xj−μj)T]

 shows the influence of these parameters on a two-dimensional Gaussian

distribution. The variances for each random variable are on the diagonal

 of the covariance matrix, while the other values show the covariance

between them.

Covariance matrix (Σ)10.70.72By dragging the handles you

 Gaussian distributions are widely used to model the real world.

 One of the implications of this theorem is that a collection of

independent, identically distributed random variables with finite

variance have a mean that is distributed normally.

 .

 In the next section we will take a closer look at how to

manipulate Gaussian distributions and extract useful information from

them.

### Marginalization and Conditioning

 Gaussian distributions have the nice algebraic property of being

closed under conditioning and marginalization.

 Being closed under conditioning and marginalization means that the

resulting distributions from these operations are also Gaussian, which

makes many problems in statistics and machine learning tractable.

 In the following we will take a closer look at both of these

operations, as they are the foundation for Gaussian processes.

PX,Y=[XY]∼N(μ,Σ)=N([μXμY],[ΣXXΣXYΣYXΣYY])P\_{X,Y}

 = \begin{bmatrix} X \\ Y \end{bmatrix} \sim \mathcal{N}(\mu, \Sigma) =

\mathcal{N} \left( \begin{bmatrix} \mu\_X \\ \mu\_Y \end{bmatrix},

\begin{bmatrix} \Sigma\_{XX} \, \Sigma\_{XY} \\ \Sigma\_{YX} \, \Sigma\_{YY}

 \end{bmatrix} \right)PX,Y=[XY]∼N(μ,Σ)=N([μXμY],[ΣXXΣXYΣYXΣYY])

With XXX and YYY representing subsets of original random variables.

Through *marginalization* we can extract partial information

 from multivariate probability distributions. In particular, given a

X∼N(μX,ΣXX)Y∼N(μY,ΣYY)

 \begin{aligned}

 X &\sim \mathcal{N}(\mu\_X, \Sigma\_{XX}) \\

 Y &\sim \mathcal{N}(\mu\_Y, \Sigma\_{YY})

 \end{aligned}

 XY∼N(μX,ΣXX)∼N(μY,ΣYY)

pX(x)=∫ypX,Y(x,y)dy=∫ypX∣Y(x∣y)pY(y)dy

 p\_X(x) = \int\_y p\_{X,Y}(x,y)dy = \int\_y p\_{X|Y}(x|y) p\_Y(y) dy

 pX(x)=∫ypX,Y(x,y)dy=∫ypX∣Y(x∣y)pY(y)dy

 X=xX = xX=x, we need to consider all possible outcomes of

 YYY that can jointly lead to the result

 The corresponding [Wikipedia

 .

 Another important operation for Gaussian processes is *conditioning*.

 Conditioning is defined by:

X∣Y∼N(μX+ΣXYΣYY−1(Y−μY),ΣXX−ΣXYΣYY−1ΣYX)Y∣X∼N(μY+ΣYXΣXX−1(X−μX),ΣYY−ΣYXΣXX−1ΣXY)

 \begin{aligned}

 X|Y &\sim \mathcal{N}(\:\mu\_X + \Sigma\_{XY}\Sigma\_{YY}^{-1}(Y -

 \mu\_Y),\: \Sigma\_{XX}-\Sigma\_{XY}\Sigma\_{YY}^{-1}\Sigma\_{YX}\:) \\

 Y|X &\sim \mathcal{N}(\:\mu\_Y + \Sigma\_{YX}\Sigma\_{XX}^{-1}(X -

 \mu\_X),\: \Sigma\_{YY}-\Sigma\_{YX}\Sigma\_{XX}^{-1}\Sigma\_{XY}\:) \\

 \end{aligned}

 Note that the new mean only depends on the conditioned variable,

while the covariance matrix is independent from this variable.

 Now that we have worked through the necessary equations, we will

think about how we can understand the two operations visually.

 While marginalization and conditioning can be applied to

multivariate distributions of many dimensions, it makes sense to

 Marginalization can be seen as integrating along one of the

dimensions of the Gaussian distribution, which is in line with the

general definition of the marginal distribution.

 Conditioning also has a nice geometric interpretation — we can

imagine it as making a cut through the multivariate distribution,

yielding a new Gaussian distribution with fewer dimensions.

#### Marginalization (Y)

#### Conditioning (X = 1.2)

μY = 0σY = 1.4

X = 1.2

μY|X = 0.96σY|X = 1.4

 A bivariate normal distribution in the center.

 On the left you can see the result of marginalizing this

distribution for Y, akin to integrating along the X axis. On the right

you can see the distribution conditioned on a given X, which is similar

to a cut through the original distribution. The Gaussian distribution

and the conditioned variable can be changed by dragging the handles.

Gaussian Processes

------------------

 Now that we have recalled some of the basic properties of

multivariate Gaussian distributions, we will combine them together to

define Gaussian processes, and show how they can be used to tackle

regression problems.

 First, we will move from the continuous view to the discrete

representation of a function:

 rather than finding an implicit function, we are interested in

 In addition, each of these random variables has a corresponding index iii.

x1x2

μx1x2

 Here, we have a two-dimensional normal distribution.

 Each dimension xix\_ixi is assigned an index i∈{1,2}i \in \{1,2\}i∈{1,2}.

 gets larger and vice versa.

 You can also drag the handles in the figure to the right and

observe the probability of such a configuration in the figure to the

left.

 Respective to the test data XXX, we will denote the training data as YYY.

 In the case of Gaussian processes, this information is the training data.

 Thus, we are interested in the conditional probability PX∣YP\_{X|Y}PX∣Y.

 of the Gaussian process.

 We will talk about this in detail in the next section.

 But before we come to this, let us reflect on how we can use

multivariate Gaussian distributions to estimate function values.

f(x) = ?We are interested in predicting the

 In Gaussian processes we treat each test point as a random variable.

 Since we want to predict the function values at

 ∣X∣=N|X|=N∣X∣=N

 test points, the corresponding multivariate Gaussian distribution is also

 NNN

 -dimensional.

### Kernels

 This process is also called *centering* of the data.

 The covariance matrix will not only describe the shape of our

distribution, but ultimately determines the characteristics of the

function that we want to predict.

k:Rn×Rn→R,Σ=Cov(X,X′)=k(t,t′)

 k: \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R},\quad

 \Sigma = \text{Cov}(X,X’) = k(t,t’)

 k:Rn×Rn→R,Σ=Cov(X,X′)=k(t,t′)

 This step is also depicted in the [figure above](#PriorFigure).

 In order to get a better intuition for the role of the kernel,

let’s think about what the entries in the covariance matrix describe.

 random variable.

 Since the kernel describes the similarity between the values of

our function, it controls the possible shape that a fitted function can

adopt.

 Note that when we choose a kernel, we need to make sure that the

resulting matrix adheres to the properties of a covariance matrix.

 Many of these kernels conceptually embed the input points into a

#### RBF Kernel

\sigma^2 \exp \left( - \frac{||t-t'||^2}{2 l^2} \right)

 =

 =

#### Periodic

\sigma^2 \exp \left( - \frac{2 \sin^2(\pi |t-t'| / p)}{l^2} \right)

 =

 =

 =

#### Linear

\sigma\_b^2 + \sigma^2 (t - c)(t' - c)

 =

 =

 =

For the

 kernel

 the parameter

 ...

 determines

 ...

 There are many more kernels that can describe different classes of

functions, which can be used to model the desired shape of the function.

 A good overview of different kernels is given by Duvenaud.

 It is also possible to combine several kernels — but we will get to this later.

### Prior Distribution

 We will now shift our focus back to the original task of regression.

 In [this figure above](#DimensionSwap), we show this connection:

 Recall that we usually assume μ=0\mu=0μ=0.

 In the previous section we have looked at examples of different

kernels.

 The kernel is used to define the entries of the covariance matrix.

 Consequently, the covariance matrix determines which type of

functions from the space of all possible functions are more probable.

 As the prior distribution does not yet contain any additional

information, it is perfect to visualize the influence of the kernel on

the distribution of functions.

RBF

Periodic

Linear

y = 0μ + 2σμ - 2σ(click to start)

 =

 =

 Clicking on the graph results in continuous samples drawn from a

 Gaussian process using the selected

 functions are distributed normally around the mean µ .

 This also varies the confidence of the prediction.

### Posterior Distribution

 So what happens if we observe training data?

 Let’s get back to the model of Bayesian inference, which states

that we can incorporate this additional information into our model,

yielding the *posterior* distribution PX∣YP\_{X|Y}PX∣Y.

 We will now take a closer look at how to do this for Gaussian processes.

 Using *conditioning* we can find PX∣YP\_{X|Y}PX∣Y from PX,YP\_{X,Y}PX,Y.

 More details can be found in the [related section](#MargCond)

 on conditioning multivariate Gaussian distributions.

 The intuition behind this step is that the training points

constrain the set of functions to those that pass through the training

points.

f(x) = ?Adding training points (■) changes

 In many cases this can lead to fitted functions that are unnecessarily complex.

 Also, up until now, we have considered the training points YYY

 to be perfect measurements.

 But in real-world scenarios this is an unrealistic assumption,

since most of our data is afflicted with measurement errors or

uncertainty.

 Gaussian processes offer a simple solution to this problem by

modeling the error of the measurements.

Y=f(X)+ϵ

 Y = f(X) + \epsilon

 Y=f(X)+ϵ

PX,Y=[XY]∼N(0,Σ)=N([00],[ΣXXΣXYΣYXΣYY+ψ2I])P\_{X,Y}

 = \begin{bmatrix} X \\ Y \end{bmatrix} \sim \mathcal{N}(0, \Sigma) =

\mathcal{N} \left( \begin{bmatrix} 0 \\ 0 \end{bmatrix}, \begin{bmatrix}

 \Sigma\_{XX} & \Sigma\_{XY} \\ \Sigma\_{YX} & \Sigma\_{YY}+\psi^2I

\end{bmatrix} \right)PX,Y=[XY]∼N(0,Σ)=N([00],[ΣXXΣYXΣXYΣYY+ψ2I])

 In this formulation, ψ\psiψ is an additional parameter of our model.

 Analogous to the prior distribution, we could obtain a prediction

for our function values by sampling from this distribution.

 But, since sampling involves randomness, the resulting fit to the

data would not be deterministic and our prediction could end up being an

 outlier.

 In order to make a more meaningful prediction we can use the other

 basic operation of Gaussian distributions.

 Extracting μ′\mu’μ′ and σ′\sigma’σ′

 does not only lead to a more meaningful prediction, it also allows us

to make a statement about the confidence of the prediction.

 At first, no training points have been observed.

 The training points can be activated by clicking on them, which

leads to a constrained distribution.

 This change is reflected in the entries of the covariance matrix,

and leads to an adjustment of the mean and the standard deviation of the

 predicted function.

 As we would expect, the uncertainty of the prediction is small in

regions close to the training data and grows as we move further away

from those points.

y = 0μ + 2σμ - 2σ(click to resample)

Without any activated training data,

 value on its neighbours.

 The distribution changes when we observe training data.

 Individual points can be activated by clicking on them.

 training data.

 Therefore, the function must pass directly through it.

### Combining different kernels

 As described earlier, the power of Gaussian processes lies in the

choice of the kernel function.

 This property allows experts to introduce domain knowledge into

the process and lends Gaussian processes their flexibility to capture

trends in the training data.

 For example, by choosing a suitable bandwidth for the RBF kernel,

we can control how smooth the resulting function will be.

 A big benefit that kernels provide is that they can be combined

together, resulting in a more specialized kernel.

 The decision which kernel to use is highly dependent on prior

knowledge about the data, e.g. if certain characteristics are expected.

 Examples for this would be stationary nature, or global trends and

 patterns.

 The most common kernel combinations would be addition and multiplication.

 This is how we would multiply the two:

k∗(t,t′)=klin(t,t′)⋅kper(t,t′)

 k^{\ast}(t,t’) = k\_{\text{lin}}(t,t’) \cdot k\_{\text{per}}(t,t’)

 k∗(t,t′)=klin(t,t′)⋅kper(t,t′)

 However, combinations are not limited to the above example, and

there are more possibilities such as concatenation or composition with a

 function.

 To show the impact of a kernel combination and how it might retain

 If we add a periodic and a linear kernel, the global trend of the

linear kernel is incorporated into the combined kernel.

 The result is a periodic function that follows a linear trend.

 When combining the same kernels through multiplication instead, the

result is a periodic function with a linearly growing amplitude away

from linear kernel parameter ccc.

y = 0

Linear

y = 0

Periodic

y = 0

Linear + Periodic

y = 0

Linear ⋅ Periodic

 If we draw samples from a combined linear and periodic kernel, we

can observe the different retained characteristics in the new sample.

 Addition results in a periodic function with a global trend, while

 the multiplication increases the periodic amplitude outwards.

 Knowing more about how kernel combinations influence the shape of

the resulting distribution, we can move on to a more complex example.

 At first glance, the RBF kernel accurately approximates the points.

 But since the RBF kernel is stationary it will always return to μ=0\mu=0μ=0

 in regions further away from observed training data.

 This decreases the accuracy for predictions that reach further

into the past or the future.

 An improved model can be created by combining the individual

kernels through addition, which maintains both the periodic nature and

the global ascending trend of the data.

 This procedure can be used, for example, in the analysis of

weather data.

 RBF Periodic Linear

y = 0(hover for information)RBF:

 Variance σ = 1, Length l = 1,

 As discussed in the [section about GPs](#GaussianProcesses),

 a Gaussian process can model uncertain observations.

 This can be seen when only selecting the linear kernel, as it

allows us to perform linear regression even if more than two points have

 been observed, and not all functions have to pass directly through the

observed training data.

Conclusion

----------

 understanding on how they work.

 make them even more versatile.

 For instance, sometimes it might not be possible to describe the

kernel in simple terms.

 To overcome this challenge, learning specialized kernel functions

 different purposes, e.g. *model-peeling* and hypothesis testing.

 appropriate combination and parameterization of the kernel.

Gaussian Processes in Machine Learning

Gaussian Processes for Object Categorization

Clustering Based on Gaussian Processes

Covariance matrix (Encyclopedia of Mathematics)

The Nature of Statistical Learning Theory

Automatic model construction with Gaussian processes

Information theory, inference and learning algorithms

Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

Deep Kernel Learning

Deep Neural Networks as Gaussian Processes

Deep Gaussian Processes

A Visual Exploration of Gaussian Processes.html