 #rebuttal,
 .comment-info {
 background-color: hsl(54, 78%, 96%);
 border-left: solid hsl(54, 33%, 67%) 1px;
 padding: 1em;
 color: hsla(0, 0%, 0%, 0.67);
 }

 #header-info {
 margin-top: 0;
 margin-bottom: 1.5rem;
 display: grid;
 grid-template-columns: 65px max-content 1fr;
 grid-template-areas:
 "icon explanation explanation"
 "icon back comment";
 grid-column-gap: 1.5em;
 }

 #header-info .icon-multiple-pages {
 grid-area: icon;
 padding: 0.5em;
 content: url(images/multiple-pages.svg);
 }

 #header-info .explanation {
 grid-area: explanation;
 font-size: 85%;
 }

 #header-info .back {
 grid-area: back;
 }

 #header-info .back::before {

 content: "←";
 margin-right: 0.5em;
 }

 #header-info .comment {
 grid-area: comment;
 scroll-behavior: smooth;
 }

 #header-info .comment::before {
 content: "↓";
 margin-right: 0.5em;
 }

 #header-info a.back,
 #header-info a.comment {
 font-size: 80%;
 font-weight: 600;
 border-bottom: none;
 text-transform: uppercase;
 color: #2e6db7;
 display: block;
 margin-top: 0.25em;
 letter-spacing: 0.25px;
 }

 This article is part of a discussion of the Ilyas et al. paper
 *“Adversarial examples are not bugs, they are features”.*
 You can learn more in the
 [main discussion article](/2019/advex-bugs-discussion/) .

[Other Comments](/2019/advex-bugs-discussion/#commentaries)
[Comment by Ilyas et al.](#rebuttal)

 Section 3.2 of Ilyas et al. (2019) shows that training a model on only adversarial errors leads to
 non-trivial generalization on the original test set. We show that these experiments are a specific case of
 learning from errors. We start with a counterintuitive result — we take a completely mislabeled training set
 (without modifying the inputs) and use it to train a model that generalizes to the original test set. We
 then show that this result, and the results of Ilyas et al. (2019), are a special case of model
 distillation. In particular, since the incorrect labels are generated using a trained model, information
 about the trained model is being “leaked” into the dataset.
 We begin with the following question: what if we took the images in the training set (without any
 adversarial perturbations) and mislabeled them? Since the inputs are unmodified and mislabeled, intuition
 says that a model trained on this dataset should not generalize to the correctly-labeled test set.
 Nevertheless, we show that this intuition fails — a model *can* generalize.
 We first train a ResNet-18 on the CIFAR-10 training set for two epochs. The model reaches a training
 accuracy of 62.5% and a test accuracy of 63.1%. Next, we run the model on all of the 50,000 training data
 points and relabel them according to the model’s predictions. Then, we filter out *all the correct
 predictions*. We are now left with an incorrectly labeled training set of size 18,768. We show four
 examples on the left of the Figure below:

![](images/image1.png)

[1](#figure-1)

 We then randomly initialize a new ResNet-18 and train it only on this mislabeled dataset. We train for 50
 epochs and reach an accuracy of 49.7% on the *original* test set. The new model has only ever seen
 incorrectly labeled, unperturbed images but can still non-trivially generalize.

This is Model Distillation Using Incorrect Predictions
------------------------------------------------------

 How can this model and the models in Ilyas et al. (2019) generalize without seeing any correctly labeled
 data? Here, we show that since the incorrect labels are generated using a trained model, information is
 being “leaked” about that trained model into the mislabeled examples. In particular, this an indirect form
 of model distillation — training on this dataset allows a new
 model to somewhat recover the features of the original model.

 We first illustrate this distillation phenomenon using a two-dimensional problem. Then, we explore other
 peculiar forms of distillation for neural networks — -we transfer knowledge despite the inputs being from
 another task.

### Two-dimensional Illustration of Model Distillation

 We construct a dataset of adversarial examples using a two-dimensional binary classification problem. We
 generate 32 random two-dimensional data points in [0,1]2[0,1]^2[0,1]2 and assign each point a random binary label. We
 then train a small feed-forward neural network on these examples, predicting 32/32 of the examples correctly
 (panel (a) in the Figure below).

![](images/image2.png)

[2](#figure-2)

 Next, we create adversarial examples for the original model using an l∞l\_{\infty}l∞ ball of radius
 ϵ=0.12\epsilon=0.12ϵ=0.12. In panel (a) of the Figure above, we display the ϵ\epsilonϵ-ball around each training
 point. In panel (b), we show the adversarial examples which cause the model to change its prediction (from
 correct to incorrect). We train a new feed-forward neural network on this dataset, resulting in the model in
 panel (c).

 Although this new model has never seen a correctly labeled example, it is able to perform non-trivially on
 the original dataset, predicting 23/3223/3223/32 of the inputs correctly (panel (d) in the Figure). The new model’s
 decision boundary loosely matches the original model’s decision boundary, i.e., the original model has been
 somewhat distilled after training on its adversarial examples. This two-dimensional problem presents an
 illustrative version of the intriguing result that distillation can be performed using incorrect
 predictions.

###
 Other Peculiar Forms of Distillation

 Our experiments show that we can distill models using mislabeled examples. In what other peculiar ways can
 we learn about the original model? Can we use only *out-of-domain* data?

 We train a simple CNN model on MNIST, reaching 99.1% accuracy. We then run this model on the FashionMNIST
 training set and save its argmax predictions. The resulting dataset is nonsensical to humans — a “dress” is
 labeled as an “8″.

![](images/image3.png)

[3](#figure-3)

 We then initialize a new CNN model and train it on this mislabeled FashionMNIST data. The resulting model
 reaches 91.04% accuracy on the MNIST test set. Furthermore, if we normalize the FashionMNIST images using
 the mean and variance statistics for MNIST, the model reaches 94.5% accuracy on the MNIST test set. This is
 another instance of recovering a functionally similar model to the original despite the new model only
 training on erroneous predictions.

###
 Summary

 These results show that training a model using mislabeled adversarial examples is a special case of learning
 from prediction errors. In other words, the perturbations added to adversarial examples in Section 3.2 of
 Ilyas et al. (2019) are not necessary to enable learning.

 To cite Ilyas et al.’s response, please cite their
 [collection of responses](/2019/advex-bugs-discussion/original-authors/#citation).

**Response
 Summary**: Note that since our experiments work across different architectures,
 “distillation” in weight space does not occur. The only distillation that can
 arise is “feature space” distillation, which is actually exactly our hypothesis.
 In particular, feature-space distillation would not work in [World 1](/2019/advex-bugs-discussion/original-authors/#world1) — if the
 adversarial examples we generated did not exploit useful features, we should not
 have been able to “distill” a useful model from them. (In fact, one might think
 of normal model training as just “feature distillation” of the humans that
 labeled the dataset.) Furthermore, the hypothesis that all we need is enough
 model-consistent points in order to recover a model, seems to be disproven by
 Preetum’s [“bugs-only dataset”](/2019/advex-bugs-discussion/response-5)
 and other (e.g. ) settings.
**Response**: Since our experiments work across different architectures,
 “distillation” in weight space cannot arise. Thus, from what we understand, the
 “distillation” hypothesis suggested here is referring to “feature distillation”
 (i.e. getting models which use the same features as the original), which is
 actually precisely our hypothesis too. Notably, this feature distillation would
 not be possible if adversarial examples did not rely on “flipping” features that
 are good for classification (see [World
 1](/2019/advex-bugs-discussion/original-authors/#world1) and
 [World 2](/2019/advex-bugs-discussion/original-authors/#world2)) — in that case, the distilled
 model would only use features that generalize poorly, and would thus generalize
 poorly itself.

 Moreover, we would argue that in the experiments presented (learning from
 mislabeled data), the same kind of distillation is happening. For instance, a
 moderately accurate model might associate “green background” with “frog” thus
 labeling “green” images as “frogs” (e.g., the horse in the comment’s figure).
 Training a new model on this dataset will thus associate “green” with “frog”
 achieving non-trivial accuracy on the test set (similarly for the “learning MNIST
 from Fashion-MNIST” experiment in the comment). This corresponds exactly to
 learning features from labels, akin to how deep networks “distill” a good
 decision boundary from human annotators. In fact, we find these experiments
 a very interesting illustration of feature distillation that complements
 our findings.

 We also note that an analogy to logistic regression here is only possible
 due to the low VC-dimension of linear classifiers (namely, these classifiers
 have dimension ddd). In particular, given any classifier with VC-dimension
 kkk, we need at least kkk points to fully specify the classifier. Conversely, neural
 networks have been shown to have extremely large VC-dimension (in particular,
 bigger than the size of the training set ). So even though
 labelling d+1d+1d+1 random
 points model-consistently is sufficient to recover a linear model, it is not
 necessarily sufficient to recover a deep neural network. For instance, Milli et
 al.  are not able to reconstruct a ResNet-18
 using only its predictions on random Gaussian inputs. (Note that we are using a
 ResNet-50 in our experiments.)

 Finally, it seems that the only potentially problematic explanation for
 our experiments (namely, that enough model-consistent points can recover a
 classifier) is [disproved by Preetum’s experiment](/2019/advex-bugs-discussion/response-5).
 In particular, Preetum is able to design a
 dataset where training on mislabeled inputs *that are model-consistent*
 does not at all recover the decision boundary of the original model. More
 generally, the “model distillation” perspective raised here is unable to
 distinguish between the dataset created by Preetum below, and those created
 with standard PGD (as in our D^det\widehat{\mathcal{D}}\_{det}D
det and
 D^rand\widehat{\mathcal{D}}\_{rand}D
rand datasets).

 You can find more responses in the  [main discussion article](/2019/advex-bugs-discussion/).

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Learning from Incorrectly Labeled Data

Distilling the Knowledge in a Neural Network


 #rebuttal,
 .comment-info {
 background-color: hsl(54, 78%, 96%);
 border-left: solid hsl(54, 33%, 67%) 1px;
 padding: 1em;
 color: hsla(0, 0%, 0%, 0.67);
 }

 #header-info {
 margin-top: 0;
 margin-bottom: 1.5rem;
 display: grid;
 grid-template-columns: 65px max-content 1fr;
 grid-template-areas:
 "icon explanation explanation"
 "icon back comment";
 grid-column-gap: 1.5em;
 }

 #header-info .icon-multiple-pages {
 grid-area: icon;
 padding: 0.5em;
 content: url(images/multiple-pages.svg);
 }

 #header-info .explanation {
 grid-area: explanation;
 font-size: 85%;
 }

 #header-info .back {
 grid-area: back;
 }

 #header-info .back::before {

 content: "←";
 margin-right: 0.5em;
 }

 #header-info .comment {
 grid-area: comment;
 scroll-behavior: smooth;
 }

 #header-info .comment::before {
 content: "↓";
 margin-right: 0.5em;
 }

 #header-info a.back,
 #header-info a.comment {
 font-size: 80%;
 font-weight: 600;
 border-bottom: none;
 text-transform: uppercase;
 color: #2e6db7;
 display: block;
 margin-top: 0.25em;
 letter-spacing: 0.25px;
 }

 This article is part of a discussion of the Ilyas et al. paper
 *“Adversarial examples are not bugs, they are features”.*
 You can learn more in the
 [main discussion article](/2019/advex-bugs-discussion/) .

[Other Comments](/2019/advex-bugs-discussion/#commentaries)
[Comment by Ilyas et al.](#rebuttal)

 We want to thank all the commenters for the discussion and for spending time
 designing experiments analyzing, replicating, and expanding upon our results.
 These comments helped us further refine our understanding of adversarial
 examples (e.g., by visualizing useful non-robust features or illustrating how
 robust models are successful at downstream tasks), but also highlighted aspects
 of our exposition that could be made more clear and explicit.

 Our response is organized as follows: we first recap the key takeaways from
 our paper, followed by some clarifications that this discussion brought to
 light. We then address each comment individually, prefacing each longer response
 with a quick summary.

 We also recall some terminology from
 [our paper](https://arxiv.org/abs/1905.02175) that features in our responses:

 *Datasets*: Our experiments involve the following variants of the given
 dataset DDD (consists of sample-label pairs (xxx, yyy))  The
 exact details for construction of the datasets can be found in our
 [paper](https://arxiv.org/abs/1905.02175), and
 the datasets themselves can be downloaded at  [http://git.io/adv-datasets](http://git.io/adv-
datasets) :

* D^R\widehat{\mathcal{D}}\_{R}D
R: Restrict each sample xxx to features that are used by a *robust*
 model.
* D^NR\widehat{\mathcal{D}}\_{NR}D
NR: Restrict each sample xxx to features that are used by a *standard*
 model.
* D^det\widehat{\mathcal{D}}\_{det}D
det: Adversarially perturb each sample xxx using a standard model in a
 *consistent manner* towards class y+1modCy + 1\mod Cy+1modC.
* D^rand\widehat{\mathcal{D}}\_{rand}D
rand: Adversarially perturb each sample xxx using a standard model
 towards a
 *uniformly random* class.

Main points
-----------

### *Takeaway #1:* Adversarial examples as innate
 brittleness vs. useful features (sensitivity vs reliance)

The goal of our experiments with non-robust features is to understand
 how adversarial examples fit into the following two worlds:

* **World 1: Adversarial examples exploit directions irrelevant for
 classification.** In this world, adversarial examples arise from
 sensitivity to a signal that is unimportant for classification. For
 instance, suppose there is a feature f(x)f(x)f(x) that is not generalizing
 on the dataNote that f(x)f(x)f(x) could be correlated with the label
 in the training set but not in expectation/on the test
 set., but the model for some reason puts a lot of weight on
 it,  *i.e., this sensitivity is an aberration “hallucinated” by the
 model*. Adversarial examples correspond to perturbing the input
 to change this feature by a small amount. This perturbation, however,
 would be orthogonal to how the model actually typically makes
 predictions (on natural data). (Note that this is just a single
 illustrative example — the key characteristic of this world is that
 features “flipped” when making adversarial examples are separate from the
 ones actually used to classify inputs.)
* **World 2: Adversarial examples exploit features that are useful for
 classification.** In this world, adversarial perturbations
 can correspond to changes in the input that manipulate features relevant to
 classification. Thus, models base their (mostly correct) predictions on
 features that can be altered via small perturbations.

 Recent works provide some theoretical evidence that adversarial examples
 can arise from finite-sample overfitting
 or
 other concentration of
 measure-based phenomena, thus
 supporting
 the “World 1” viewpoint on
 adversarial examples. The question is: is “World 1” the right way to
 think about adversarial examples? If so, this would be good news — under
 this mindset, adversarial robustness might just be a matter of getting
 better, “bug-free” models (for example, by reducing overfitting).

 Our findings show, however, that the “World 1” mindset alone does not
 fully capture adversarial vulnerability; “World 2“ must be taken into
 account. Adversarial examples can — and do, if generated via standard
 methods — rely on “flipping” features that are actually useful for
 classification. Specifically, we show that by relying *only* on
 perturbations corresponding to standard first-order adversarial attacks
 one can learn models that generalize to the test set. This means that
 these perturbations truly correspond to directions that are relevant for
 classifying new, unmodified inputs from the dataset. In summary, our
 message is:

**Adversarial vulnerability can arise from
 flipping features in the data that are useful for
 classification of *correct* inputs.**

In particular, note that our experiments (training on the
 D^rand\widehat{\mathcal{D}}\_{rand}D
rand and D^det\widehat{\mathcal{D}}\_{det}D
det
 datasets) would not have the same result in World 1. Concretely, in
 the “cartoon example” of World 1 presented above, the classifier puts large weight www on a feature
 coordinate f(x)f(x)f(x) that is not generalizing for “natural images.” Then,
 adversarial examples towards either class can be made by simply making
 f(x)f(x)f(x) slightly positive or slightly negative. However, a classifier
 learned from these adversarial examples would  *not*  generalize to
 the true dataset (since it would learn to depend on a feature that is not
 useful on natural images).

### *Takeaway #2*: Learning from “meaningless” data

Another implication of our experiments is that models may not even
 *need* any information which we as humans view as “meaningful” in order
 to do well (in the generalization sense) on standard image datasets. (Our
 D^NR\widehat{\mathcal{D}}\_{NR}D
NR dataset is a perfect example of this.)

### *Takeaway #3*: Cannot fully attribute adversarial
 examples to X

We also show that we cannot
 conclusively fully attribute adversarial examples to any specific aspect of the
 standard training framework (BatchNorm, ResNets, SGD, etc.). In particular, our
 “robust dataset” D^R\widehat{\mathcal{D}}\_{R}D
R is a counterexample to any claim of the form “given any
 dataset, training with BatchNorm/SGD/ResNets/overparameterization/etc. leads to
 adversarial vulnerability” (as classifiers with all of these components,
 when trained on D^R\widehat{\mathcal{D}}\_{R}D
R, generalize robustly to
 CIFAR-10). In that sense, the dataset clearly plays a role in
 the emergence of adversarial examples. (Also, further corroborating this is
 Preetum’s “adversarial squares” dataset [here](#PreetumResponse),
 where standard networks do not become adversarially vulnerable as long as there is no
 label noise or overfitting.)

A Few Clarifications
--------------------

In addition to further refining our understanding of adversarial examples,
 the comments were also very useful in pointing out which aspects of our
 claims could benefit from further clarification. To this end, we make these
 clarifications below in the form of a couple “non-claims” — claims that we did
 *not* intend to make. We’ll also update our paper in order to make
 these clarifications explicit.

### Non-Claim #1: “Adversarial examples *cannot* be bugs”

 Our goal is to say that since adversarial examples can arise from
 well-generalizing features, simply patching up the “bugs” in ML models will
 not get rid of adversarial vulnerability — we also need to make sure our
 models learn the right features. This, however, does not mean that
 adversarial vulnerability *cannot* arise from “bugs”. In fact, note
 that several papers

 have proven that adversarial vulnerability can
 arise from what we refer to as “bugs,” e.g. finite-sample overfitting,
 concentration of measure, high dimensionality, etc. Furthermore,
 We would like to thank Preetum for pointing out that this issue may be a
 natural misunderstanding, and for exploring this point in even more depth
 in his response below.

### Non-Claim #2: “Adversarial examples are purely a result of the dataset”

 Even though we [demonstrated](#cannotpin) that datasets do
 play a role in the emergence of adversarial examples, we do not intend to
 claim that this role is exclusive. In particular, just because the data
  *admits*  non-robust functions that are well-generalizing (useful
 non-robust features), doesn’t mean that  *any*  model will learn to
 pick up these features. For example, it could be that the well-generalizing
 features that cause adversarial examples are only learnable by certain
 architectures. However, we do show that there is a way, via only
 altering the dataset, to induce robust models — thus, our results indicate
 that adversarial vulnerability indeed cannot be completely disentangled
 from the dataset (more on this in [Takeaway #3](#cannotpin)).

 The following responses may also be viewed in context with the comment they’re addressing. [Return to the discussion article](/2019/advex-bugs-discussion/#commentaries) for a list of
 summaries.

Responses to comments
---------------------

### Adversarial Example Researchers Need to Expand What is Meant by
 “Robustness” (Dan Hendrycks, Justin Gilmer)

**Response Summary**:
 The demonstration of models that learn from only high-frequency components of the data is
 an interesting finding that provides us with another way our models can learn from data that
 appears “meaningless” to humans.
 The authors fully agree that studying a wider notion of robustness will become increasingly
 important in ML, and will help us get a better grasp of features we actually want our models
 to rely on.

**Response**: The fact that models can learn to classify correctly based
 purely on the high-frequency component of the training set is neat! This nicely
 complements one of our [takeaways](#takeaway1): models will rely on
 useful features even if these features appear incomprehensible to humans.

 Also, while non-robustness to noise can be an indicator of models using
 non-robust useful features, this is not how the phenomenon was predominantly viewed.
 More often than not, the brittleness of ML models to noise was instead regarded
 as an innate shortcoming of the models, e.g., due to poor margins. (This view is
 even more prevalent in the adversarial robustness community.) Thus, it was often
 expected that progress towards “better”/”bug-free” models will lead to them
 being more robust to noise and adversarial examples.

 Finally, we fully agree that the set of LpL\_pLp-bounded perturbations is a very
 small subset of the perturbations we want our models to be robust to. Note,
 however, that the focus of our work is human-alignment — to that end, we
 demonstrate that models rely on features sensitive to patterns that are
 imperceptible to humans. Thus, the existence of other families of
 incomprehensible but useful features would provide even more support for our
 thesis — identifying and characterizing such features is an interesting area for
 future research.

### Robust Feature Leakage (Gabriel Goh)

**Response Summary**:
 This is a nice in-depth investigation that highlights (and neatly visualizes) one of
 the motivations for designing the D^det\widehat{\mathcal{D}}\_{det}D
det dataset.

**Response**: This comment raises a valid concern which was in fact one of
 the primary reasons for designing the D^det\widehat{\mathcal{D}}\_{det}D
det dataset.
 In particular, recall the construction of the D^rand\widehat{\mathcal{D}}\_{rand}D
rand
 dataset: assign each input a random target label and do PGD towards that label.
 Note that unlike the D^det\widehat{\mathcal{D}}\_{det}D
det dataset (in which the
 target class is deterministically chosen), the D^rand\widehat{\mathcal{D}}\_{rand}D
rand
 dataset allows for robust features to actually have a (small) positive
 correlation with the label.

To see how this can happen, consider the following simple setting: we have a
 single feature f(x)f(x)f(x) that is 111 for cats and −1-1−1 for dogs. If ϵ=0.1\epsilon = 0.1ϵ=0.1
 then f(x)f(x)f(x) is certainly a robust feature. However, randomly assigning labels
 (as in the dataset D^rand\widehat{\mathcal{D}}\_{rand}D
rand) would make this feature
 uncorrelated with the assigned label, i.e., we would have that E[f(x)⋅y]=0E[f(x)\cdot y] = 0E[f(x)⋅y]=0. Performing a
 targeted attack might in this case induce some correlation with the
 assigned label, as we could have E[f(x+η⋅∇f(x))⋅y]>E[f(x)⋅y]=0\mathbb{E}[f(x+\eta\cdot\nabla
 f(x))\cdot y] > \mathbb{E}[f(x)\cdot y] = 0E[f(x+η⋅∇f(x))⋅y]>E[f(x)⋅y]=0, allowing a model to learn
 to correctly classify new inputs.

In other words, starting from a dataset with no features, one can encode
 robust features within small perturbations. In contrast, in the
 D^det\widehat{\mathcal{D}}\_{det}D
det dataset, the robust features are *correlated
 with the original label* (since the labels are permuted) and since they are
 robust, they cannot be flipped to correlate with the newly assigned (wrong)
 label. Still, the D^rand\widehat{\mathcal{D}}\_{rand}D
rand dataset enables us to show
 that (a) PGD-based adversarial examples actually alter features in the data and
 (b) models can learn from human-meaningless/mislabeled training data. The
 D^det\widehat{\mathcal{D}}\_{det}D
det dataset, on the other hand, illustrates that the
 non-robust features are actually sufficient for generalization and can be
 preferred over robust ones in natural settings.

The experiment put forth in the comment is a clever way of showing that such
 leakage is indeed possible. However, we want to stress (as the comment itself
 does) that robust feature leakage does *not* have an impact on our main
 thesis — the D^det\widehat{\mathcal{D}}\_{det}D
det dataset explicitly controls
 for robust
 feature leakage (and in fact, allows us to quantify the models’ preference for
 robust features vs non-robust features — see Appendix D.6 in the
 [paper](https://arxiv.org/abs/1905.02175)).

### Two Examples of Useful, Non-Robust Features (Gabriel Goh)

**Response Summary**: These experiments with linear models are a great first step towards visualizing
 non-robust features for real datasets (and thus a neat corroboration of their existence).
 Furthermore, the theoretical construction of “contaminated” non-robust features opens an
 interesting direction of developing a more fine-grained definition of features.

**Response**: These experiments (visualizing the robustness and
 usefulness of different linear features) are very interesting! They both further
 corroborate the existence of useful, non-robust features and make progress
 towards visualizing what these non-robust features actually look like.

We also appreciate the point made by the provided construction of non-robust
 features (as defined in our theoretical framework) that are combinations of
 useful+robust and useless+non-robust features. Our theoretical framework indeed
 enables such a scenario, even if — as the commenter already notes — our
 experimental results do not. (In this sense, the experimental results and our  [main
 takeaway](#takeaway1)  are actually stronger than our theoretical
 framework technically captures.) Specifically, in such a scenario, during the
 construction of the D^det\widehat{\mathcal{D}}\_{det}D
det dataset, only the non-robust
 and useless term of the feature would be flipped. Thus, a classifier trained on
 such a dataset would associate the predictive robust feature with the
 *wrong* label and would thus not generalize on the test set. In contrast,
 our experiments show that classifiers trained on D^det\widehat{\mathcal{D}}\_{det}D
det
 do generalize.

Overall, our focus while developing our theoretical framework was on
 enabling us to formally describe and predict the outcomes of our experiments. As
 the comment points out, putting forth a theoretical framework that captures
 non-robust features in a very precise way is an important future research
 direction in itself.

### Adversarially Robust Neural Style Transfer
 (Reiichiro Nakano)

**Response Summary**:
 Very interesting results that highlight the potential role of non-robust features and the
 utility of robust models for downstream tasks. We’re excited to see what kind of impact robustly
 trained models will have in neural network art!
 Inspired by these findings, we also take a deeper dive into (non-robust) VGG, and find some
 interesting links between robustness and style transfer.

**Response**: These experiments are really cool! It is interesting that
 preventing the reliance of a model on non-robust features improves performance
 on style transfer, even without an explicit task-related objective (i.e. we
 didn’t train the networks to be better for style transfer).

 We also found the discussion of VGG as a “mysterious network” really
 interesting — it would be valuable to understand what factors drive style transfer
 performance more generally. Though not a complete answer, we made a couple of
 observations while investigating further:

*Style transfer does work with AlexNet:* One wrinkle in the idea that
 robustness is the “secret ingredient” to style transfer could be that VGG is not
 the most naturally robust network — AlexNet is. However, based on our own
 testing, style transfer does seem to work with AlexNet out-of-the-box, as
 long as we use a few early layers in the network (in a similar manner to
 VGG):

Style transfer using AlexNet, using
 conv\_1 through conv\_4.

![](images/alexnetworks.png)

 Observe that even though style transfer still works, there are checkerboard
 patterns emerging — this seems to be a similar phenomenon to the one noticed
 in the comment in the context of robust models.
 This might be another indication that these two phenomena (checkerboard
 patterns and style transfer working) are not as intertwined as previously
 thought.

*From prediction robustness to layer robustness:*  Another
 potential wrinkle here is that both AlexNet and VGG are not that
 much more robust than ResNets (for which style transfer completely fails),
 and yet seem to have dramatically better performance. To try to
 explain this, recall that style transfer is implemented as a minimization of a
 combined objective consisting of a style loss and a content loss. We found,
 however, that the network we use to compute the
 style loss is far more important
 than the one for the content loss. The following demo illustrates this — we can
 actually use a non-robust ResNet for the content loss and everything works just
 fine:

Style transfer seems to be rather
 invariant to the choice of content network used, and very sensitive
 to the style network used.

![](images/stylematters.png)

Therefore, from now on, we use a fixed ResNet-50 for the content loss as a
 control, and only worry about the style loss.

Now, note that the way that style loss works is by using the first few
 layers of the relevant network. Thus, perhaps it is not about the robustness of
 VGG’s predictions, but instead about the robustness of the layers that we actually use
 for style transfer?

 To test this hypothesis, we measure the robustness of a layer fff as:

R(f)=Ex1∼D[maxx′∥f(x′)−f(x1)∥2]Ex1,x2∼D[∥f(x1)−f(x2)∥2]
 R(f) = \frac{\mathbb{E}\_{x\_1\sim D}\left[\max\_{x’} \|f(x’) - f(x\_1)\|\_2 \right]}
 {\mathbb{E}\_{x\_1, x\_2 \sim D}\left[\|f(x\_1) - f(x\_2)\|\_2\right]}
 R(f)=Ex1,x2∼D[∥f(x1)−f(x2)∥2]Ex1∼D[maxx′∥f(x′)−f(x1)∥2]
 Essentially, this quantity tells us how much we can change the
 output of that layer f(x)f(x)f(x) within a small ball, normalized by how far apart
 representations are between images in general. We’ve plotted this value for
 the first few layers in a couple of different networks below:

The robustness R(f)R(f)R(f) of the first
 four layers of VGG16, AlexNet, and robust/standard ResNet-50
 trained on ImageNet.

![](images/robustnesses.png)

 Here, it becomes clear that, the first few layers of VGG and AlexNet are
 actually almost as robust as the first few layers of the robust ResNet!
 This is perhaps a more convincing indication that robustness might have
 something to with VGG’s success in style transfer after all.

 Finally, suppose we restrict style transfer to only use a single layer of
 the network when computing the style lossUsually style transfer uses
 several layers in the loss function to get the most visually appealing results — here we’re only interested in whether or not style transfer works (i.e.
 actually confers some style onto the image).. Again, the more
 robust layers seem to indeed work better for style transfer! Since all of the
 layers in the robust ResNet are robust, style transfer yields non-trivial
 results even using the last layer alone. Conversely, VGG and AlexNet seem to
 excel in the earlier layers (where they are non-trivially robust) but fail when
 using exclusively later (non-robust) layers:

Style transfer using a single layer. The
 names of the layers and their robustness R(f)R(f)R(f) are printed below
 each style transfer result. We find that for both networks, the robust
 layers seem to work (for the robust ResNet, every layer is robust).

![](images/styletransfer.png)

 Of course, there is much more work to be done here, but we are excited
 to see further work into understanding the role of both robustness and the VGG
 in network-based image manipulation.

###  Adversarial Examples are Just Bugs, Too (Preetum
 Nakkiran)

**Response Summary**:
 A fine-grained look at adversarial examples that neatly complements our thesis (i.e. that non-robust
 features exist and adversarial examples arise from them, see [Takeaway #1](#takeaway1)) while
 providing an
 example of adversarial examples that arise from “bugs”.
 The fact that the constructed “bugs”-based adversarial examples don’t transfer constitutes
 another evidence for the link between transferability and (non-robust) features.

**Response**: As mentioned [above](#nonclaim1),
 we did not intend to claim
 that adversarial examples arise *exclusively* from (useful) features but rather
 that useful non-robust features exist and are thus (at least
 partially) responsible for adversarial vulnerability. In fact,
 prior work already shows how in theory adversarial examples can arise from
 insufficient samples  or finite-sample overfitting , and the experiments
 presented here (particularly, the adversarial squares) constitute a neat
 real-world demonstration of these facts.

 Our main thesis that “adversarial examples will not just go away as we fix
 bugs in our models” is not contradicted by the existence of adversarial examples
 stemming from “bugs.” As long as adversarial examples can stem from non-robust
 features (which the commenter seems to agree with), fixing these bugs will not
 solve the problem of adversarial examples.

Moreover, with regards to feature “leakage” from PGD, recall that in
 or D\_det dataset, the non-robust features are associated with the
 correct label whereas the robust features are associated with the wrong
 one. We wanted to emphasize that, as shown in [Appendix D.6](LINK),
 models trained on our D\_det dataset actually generalize *better* to
 the non-robust feature-label association that to the robust
 feature-label association. In contrast, if PGD introduced a small
 “leakage” of non-robust features, then we would expect the trained model
 would still predominantly use the robust feature-label association.

 That said, the experiments cleverly zoom in on some more fine-grained
 nuances in our understanding of adversarial examples. One particular thing that
 stood out to us is that by creating a set of adversarial examples that are
 *explicitly* non-transferable, one also prevents new classifiers from learning
 features from that dataset. This finding thus makes the connection between
 transferability of adversarial examples and their containing generalizing
 features even stronger! Indeed, we can add the constructed dataset into our
 “D^det\widehat{\mathcal{D}}\_{det}D
det learnability vs transferability” plot
 (Figure 3 in the paper) — the point
 corresponding to this dataset fits neatly onto the trendline!

Relationship between models reliance on non-robust features and their susceptibility to transfer
 attacks

![](diagrams/transfer.png)

### Learning from Incorrectly Labeled Data (Eric Wallace)

**Response Summary**:
 These experiments are a creative demonstration of the fact that the underlying phenomenon of
 learning features from “human-meaningless” data can actually arise in a broad range of
 settings.

**Response**: Since our experiments work across different architectures,
 “distillation” in weight space cannot arise. Thus, from what we understand, the
 “distillation” hypothesis suggested here is referring to “feature distillation”
 (i.e. getting models which use the same features as the original), which is
 actually precisely our hypothesis too. Notably, this feature distillation would
 not be possible if adversarial examples did not rely on “flipping” features that
 are good for classification (see [World 1](#world1) and
 [World 2](#world2)) — in that case, the distilled
 model would only use features that generalize poorly, and would thus generalize
 poorly itself.

 Moreover, we would argue that in the experiments presented (learning from
 mislabeled data), the same kind of distillation is happening. For instance, a
 moderately accurate model might associate “green background” with “frog” thus
 labeling “green” images as “frogs” (e.g., the horse in the comment’s figure).
 Training a new model on this dataset will thus associate “green” with “frog”
 achieving non-trivial accuracy on the test set (similarly for the “learning MNIST
 from Fashion-MNIST” experiment in the comment). This corresponds exactly to
 learning features from labels, akin to how deep networks “distill” a good
 decision boundary from human annotators. In fact, we find these experiments
 a very interesting illustration of feature distillation that complements
 our findings.

 We also note that an analogy to logistic regression here is only possible
 due to the low VC-dimension of linear classifiers (namely, these classifiers
 have dimension ddd). In particular, given any classifier with VC-dimension
 kkk, we need at least kkk points to fully specify the classifier. Conversely, neural
 networks have been shown to have extremely large VC-dimension (in particular,
 bigger than the size of the training set ). So even though
 labelling
 d+1d+1d+1 random
 points model-consistently is sufficient to recover a linear model, it is not
 necessarily sufficient to recover a deep neural network. For instance, Milli et
 al.  are not able to reconstruct a ResNet-18
 using only its predictions on random Gaussian inputs. (Note that we are using a
 ResNet-50 in our experiments.)

 Finally, it seems that the only potentially problematic explanation for
 our experiments (namely, that enough model-consistent points can recover a
 classifier) is disproved by the experiment done by Preetum (see  [LINK](#PreetumResponse)). In
 particular, Preetum is able to design a
 dataset where training on mislabeled inputs *that are model-consistent*
 does not at all recover the decision boundary of the original model. More
 generally, the “model distillation” perspective raised here is unable to
 distinguish between the dataset created by Preetum below, and those created
 with standard PGD (as in our D^det\widehat{\mathcal{D}}\_{det}D
det and
 D^rand\widehat{\mathcal{D}}\_{rand}D
rand datasets).

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Discussion and Author Responses

Logan Engstrom
Andrew Ilyas
Aleksander Madry
Shibani Santurkar
Dimitris Tsipras


 By some metrics, research on Generative Adversarial Networks (GANs) has progressed substantially in the past 2 years.
 Practical improvements to image synthesis models are being made  almost too quickly to keep up with:

![](images/gan-progress.png)
Odena et al., 2016
Miyato et al., 2017
Zhang et al., 2018
Brock et al., 2018

 However, by other metrics, less has happened. For instance, there is still widespread disagreement about how GANs should be evaluated.
 Given that current image synthesis benchmarks seem somewhat saturated, we think now is a good time to reflect on research goals for this sub-field.

 Lists of open problems have helped other fields with this.
 This article suggests open research problems that we’d be excited for other researchers to work on.
 We also believe that writing this article has clarified our thinking about
 GANs, and we would encourage other researchers to write similar articles about their own sub-fields.
 We assume a fair amount of background (or willingness to look things up)
 because we reference too many results to explain all those results in detail.

What are the Trade-Offs Between GANs and other Generative Models?
-----------------------------------------------------------------

 In addition to GANs, two other types of generative model are currently popular: Flow Models and Autoregressive Models

 This statement shouldn’t be taken too literally.
 Those are useful terms for describing fuzzy clusters in ‘model-space’, but there are models
 that aren’t easy to describe as belonging to just one of those clusters.
 I’ve also left out VAEs entirely;
 they’re arguably no longer considered state-of-the-art at any tasks of record.
 .
 Roughly speaking, Flow Models apply a
 stack of invertible transformations to a sample from a prior
 so that exact log-likelihoods of observations can be computed.
 On the other hand, Autoregressive Models factorize the
 distribution over observations into conditional distributions
 and process one component of the observation at a time (for images, they may process one pixel
 at a time.)
 Recent research suggests that these models have different
 performance characteristics and trade-offs.
 We think that accurately characterizing these trade-offs and deciding whether they are intrinsic
 to the model families is an interesting open question.

 For concreteness, let’s temporarily focus on the difference in computational cost between GANs and Flow Models.
 At first glance, Flow Models seem like they might make GANs unnecessary.
 Flow Models allow for exact log-likelihood computation and exact inference,
 so if training Flow Models and GANs had the same computational cost, GANs might not be useful.
 A lot of effort is spent on training GANs,
 so it seems like we should care about whether Flow Models make GANs obsolete

 Even in this case, there might still be other reasons to use adversarial training in contexts like
 image-to-image translation.
 It also might still make sense to combine adversarial training with maximum-likelihood training.
 .

 However, there seems to be a substantial gap between the computational cost of training GANs and Flow Models.
 To estimate the magnitude of this gap, we can consider two models trained on datasets of human faces.
 The GLOW model is trained to generate 256x256 celebrity faces using
 40 GPUs for 2 weeks and about 200 million parameters.
 In contrast, progressive GANs are trained on a similar face dataset
 with 8 GPUs for 4 days, using about 46 million parameters, to generate 1024x1024 images.
 Roughly speaking, the Flow Model took 17 times more GPU days and 4 times more parameters
 to generate images with 16 times fewer pixels.
 This comparison isn’t perfect,

 For instance, it’s possible that the progressive growing
 technique could be applied to Flow Models as well.

 but it gives you a sense of things.

 Why are the Flow Models less efficient?
 We see two possible reasons:
 First, maximum likelihood training might be computationally harder to do than adversarial training.
 In particular, if any element of your training set is assigned zero probability by your generative model,
 you will be penalized infinitely harshly!
 A GAN generator, on the other hand, is only penalized indirectly for assigning zero probability to training set elements,
 and this penalty is less harsh.

 Second, normalizing flows might be an inefficient way to represent certain functions.
 Section 6.1 of  does some small experiments on expressivity, but at
 present we’re not aware of any in-depth analysis of this question.

 We’ve discussed the trade-off between GANs and Flow Models, but what about Autoregressive Models?
 It turns out that Autoregressive Models can be expressed as Flow Models
 (because they are both reversible) that are not parallelizable.

 Parallelizable is somewhat imprecise in this context.
 We mean that sampling from Flow Models must in general be done sequentially, one observation at a time.
 There may be ways around this limitation though.

 It also turns out that Autoregressive Models are more time and parameter efficient than Flow Models.
 Thus, GANs are parallel and efficient but not reversible,
 Flow Models are reversible and parallel but not efficient, and
 Autoregressive models are reversible and efficient, but not parallel.

|  | Parallel | Efficient | Reversible |
| --- | --- | --- | --- |
| GANs | Yes | Yes | No |
| Flow Models | Yes | No | Yes |
| Autoregressive Models | No | Yes | Yes |

 This brings us to our first open problem:

Problem 1

What are the fundamental trade-offs between GANs and other generative models?

In particular, can we make some sort of CAP Theorem type statement about reversibility, parallelism, and parameter/time efficiency?

 One way to approach this problem could be to study more models that are a hybrid of multiple model families.
 This has been considered for hybrid GAN/Flow Models, but we think that
 this approach is still underexplored.

 We’re also not sure about whether maximum likelihood training is necessarily harder than GAN training.
 It’s true that placing zero mass on a training data point is not explicitly prohibited under the
 GAN training loss, but it’s also true that a sufficiently powerful discriminator will be able
 to do better than chance if the generator does this.
 It does seem like GANs are learning distributions of low support in practice though.

 Ultimately, we suspect that Flow Models are fundamentally less expressive per-parameter than
 arbitrary decoder functions, and we suspect that this is provable under certain assumptions.

What Sorts of Distributions Can GANs Model?
-------------------------------------------

 Most GAN research focuses on image synthesis.
 In particular, people train GANs on a handful of standard (in the Deep Learning community) image datasets:
 MNIST,
 CIFAR-10,
 STL-10,
 CelebA,
 and Imagenet.

 There is some folklore about which of these datasets is ‘easiest’ to model.
 In particular, MNIST and CelebA are considered easier than Imagenet, CIFAR-10, or STL-10 due to being
 ‘extremely regular’.
 Others have noted that ‘a high number of classes is what makes
 ImageNet synthesis difficult for GANs’.
 These observations are supported by the empirical fact that the state-of-the-art image
 synthesis model on CelebA generates images that seem
 substantially more convincing than the state-of-the-art image synthesis model on
 Imagenet.

 However, we’ve had to come to these conclusions through the laborious and noisy process of
 trying to train GANs on ever larger and more complicated datasets.
 In particular, we’ve mostly studied how GANs perform on the datasets that
 happened to be laying around for object recognition.

 As with any science, we would like to have a simple theory that explains our experimental observations.
 Ideally, we could look at a dataset, perform some computations without ever actually
 training a generative model, and then say something like ‘this dataset will be
 easy for a GAN to model, but not a VAE’.
 There has been some progress on this topic,
 but we feel that more can be done.
 We can now state the problem:

Problem 2

Given a distribution, what can we say about how hard it will be for a GAN to model that distribution?

 We might ask the following related questions as well:
 What do we mean by ‘model the distribution’? Are we satisfied with a low-support representation, or do we want a true density model?
 Are there distributions that a GAN can never learn to model?
 Are there distributions that are learnable for a GAN in principle, but are not
 efficiently learnable, for some reasonable model of resource-consumption?
 Are the answers to these questions actually any different for GANs than they are for other
 generative models?

 We propose two strategies for answering these questions:

* **Synthetic Datasets** - We can study synthetic datasets to probe what traits affect learnability.
 For example, in  the authors create a dataset of synthetic triangles.
 We feel that this
 angle is under-explored.
 Synthetic datasets can even be parameterized by quantities of interest, such as connectedness or smoothness,
 allowing for systematic study.
 Such a dataset could also be useful for studying other types of generative models.
* **Modify Existing Theoretical Results** - We can take existing theoretical results and try to
 modify the assumptions to account
 for different properties of the dataset.
 For instance, we could take results about GANs that apply given unimodal data distributions and see
 what happens to them when the data distribution becomes multi-modal.

How Can we Scale GANs Beyond Image Synthesis?
---------------------------------------------

 Aside from applications like image-to-image
 translation
 and domain-adaptation
 most GAN successes have been in image synthesis.
 Attempts to use GANs beyond images have focused on three domains:

* **Text** -
 The discrete nature of text makes it difficult to apply GANs.
 This is because GANs rely on backpropagating a signal from the discriminator through the generated content into the generator.
 There are two approaches to addressing this difficulty.
 The first is to have the GAN act only on continuous
 representations of the discrete data, as in.
 The second is use an actual discrete model and attempt to train the GAN using
 gradient estimation as in.
 Other, more sophisticated treatments exist,
 but as far as we can tell, none of them produce results that are competitive (in terms of perplexity)
 with likelihood-based language models.
* **Structured Data** -
 What about other non-euclidean structured data, like graphs?
 The study of this type of data is called geometric deep
 learning.
 GANs have had limited success here, but so have other deep learning techniques,
 so it’s hard to tell how much the GAN aspect matters.

 We’re aware of one attempt to use GANs in this space,
 which has the generator produce (and the discriminator ‘critique’) random walks
 that are meant to resemble those sampled from a source graph.
* **Audio** -
 Audio is the domain in which GANs are closest to achieving the success
 they’ve enjoyed with images.
 The first serious attempt at applying GANs to unsupervised audio synthesis
 is, in which the authors
 make a variety of special allowances for the fact that they are operating on audio.
 More recent work suggests GANs can even
 outperform autoregressive models on some perceptual metrics.

 Despite these attempts, images are clearly the easiest domain for GANs.
 This leads us to the statement of the problem:

Problem 3

How can GANs be made to perform well on non-image data?

Does scaling GANs to other domains require new training techniques,
 or does it simply require better implicit priors for each domain?

 We expect GANs to eventually achieve image-synthesis-level success on other continuous data,
 but that it will require better implicit priors.
 Finding these priors will require thinking hard about what makes sense and is computationally feasible
 in a given domain.

 For structured data or data that is not continuous, we’re less sure.
 One approach might be to make both the generator and discriminator
 be agents trained with reinforcement learning. Making this approach work could require
 large-scale computational resources.
 Finally, this problem may just require fundamental research progress.

What can we Say About the Global Convergence of GAN Training?
-------------------------------------------------------------

 Training GANs is different from training other neural networks because we simultaneously optimize
 the generator and discriminator for opposing objectives.
 Under certain assumptions

 These assumptions are very strict.
 The referenced paper assumes (roughly speaking) that
 the equilibrium we are looking for exists and that
 we are already very close to it.
 ,
 this simultaneous optimization
 is locally asymptotically stable.

 Unfortunately, it’s hard to prove interesting things about the fully general case.
 This is because the discriminator/generator’s loss is a non-convex function of its parameters.
 But all neural networks have this problem!
 We’d like some way to focus on just the problems created by simultaneous optimization.
 This brings us to our question:

Problem 4

When can we prove that GANs are globally convergent?

Which neural network convergence results can be applied to GANs?

 There has been nontrivial progress on this question.
 Broadly speaking, there are 3 existing techniques, all of which have generated
 promising results but none of which have been studied to completion:

* **Simplifying Assumptions** -
 The first strategy is to make simplifying assumptions about the generator and discriminator.
 For example, the simplified LGQ GAN — linear generator, Gaussian data, and quadratic discriminator — can be shown to be globally convergent, if optimized with a special technique
 and some additional assumptions.
 Among other things, it’s assumed that we can first learn the means of the Gaussian and then learn the variances.

 As another example,  show under different simplifying assumptions that
 certain types of GANs perform a mixture of moment matching and maximum likelihood estimation.

 It seems promising to gradually relax those assumptions to see what happens.
 For example, we could move away from unimodal distributions.
 This is a natural relaxation to study because ‘mode collapse’ is a standard GAN pathology.
* **Use Techniques from Normal Neural Networks** -
 The second strategy is to apply techniques for analyzing normal neural networks (which are also non-convex)
 to answer questions about convergence of GANs.
 For instance, it’s argued in that the non-convexity
 of deep neural networks isn’t a problem,

 A fact that practitioners already kind of suspected.

 because low-quality local minima of the loss function
 become exponentially rare as the network gets larger.
 Can this analysis be ‘lifted into GAN space’?
 In fact, it seems like a generally useful heuristic to take analyses of deep neural networks used as classifiers and see if they apply to GANs.
* **Game Theory** -
 The final strategy is to model GAN training using notions from game theory.
 These techniques yield training procedures that provably converge to some kind of approximate Nash equilibrium,
 but do so using unreasonably large resource constraints.
 The ‘obvious’ next step in this case is to try and reduce those resource constraints.

How Should we Evaluate GANs and When Should we Use Them?
--------------------------------------------------------

 When it comes to evaluating GANs, there are many proposals but little consensus.
 Suggestions include:

* **Inception Score and FID** -
 Both these scores
 use a pre-trained image classifier and both have
 known issues .
 A common criticism is that these scores measure
 ‘sample quality’ and don’t really capture ‘sample diversity’.
* **MS-SSIM** -
  propose using MS-SSIM to
 separately evaluate diversity, but this technique has some issues and hasn’t really caught on.
* **AIS** -
  propose putting a Gaussian observation model on the outputs
 of a GAN and using annealed importance sampling to estimate
 the log likelihood under this model, but  show that
 estimates computed this way are inaccurate in the case where the GAN generator is also a flow model

 The generator being a flow model allows for computation of exact log-likelihoods in this case.
 .
* **Geometry Score** -
  suggest computing geometric properties of the generated data manifold
 and comparing those properties to the real data.
* **Precision and Recall** -
  attempt to measure both the ‘precision’ and ‘recall’ of GANs.
* **Skill Rating** -
  have shown that trained GAN discriminators can contain useful information
 with which evaluation can be performed.

 Those are just a small fraction of the proposed GAN evaluation schemes.
 Although the Inception Score and FID are relatively popular, GAN evaluation is clearly not a settled issue.
 Ultimately, we think that confusion about *how to evaluate* GANs stems from confusion about
 *when to use GANs*.
 Thus, we have bundled those two questions into one:

Problem 5

When should we use GANs instead of other generative models?

How should we evaluate performance in those contexts?

 What should we use GANs for?
 If you want an actual density model, GANs probably aren’t the best choice.
 There is now good experimental evidence that GANs learn a ‘low support’ representation of the target dataset
 , which means there may be substantial parts of the test
 set to which a GAN (implicitly) assigns zero likelihood.

 Rather than worrying too much about this,
 Though trying to fix this issue is a valid research agenda as well.

 we think it makes sense to focus GAN research on tasks where this is fine or even helpful.
 GANs are likely to be well-suited to tasks with a perceptual flavor.
 Graphics applications like image synthesis, image translation, image infilling, and attribute manipulation
 all fall under this umbrella.

 How should we evaluate GANs on these perceptual tasks?
 Ideally, we would just use a human judge, but this is expensive.
 A cheap proxy is to see if a classifier can distinguish between real and fake examples.
 This is called a classifier two-sample test (C2STs)
 .
 The main issue with C2STs is that if the Generator has even a minor defect that’s systematic across samples
 (e.g., ) this will dominate the evaluation.

 Ideally, we’d have a holistic evaluation that isn’t dominated by a single factor.
 One approach might be to make a critic that is blind to the dominant defect.
 But once we do this, some other defect may dominate, requiring a new critic, and so on.
 If we do this iteratively, we could get a kind of ‘Gram-Schmidt procedure for critics’,
 creating an ordered list of the most important defects and
 critics that ignore them.
 Perhaps this can be done by performing PCA on the critic activations and progressively throwing out
 more and more of the higher variance components.

 Finally, we could evaluate on humans despite the expense.
 This would allow us to measure the thing that we actually care about.
 This kind of approach can be made less expensive by predicting human answers and only interacting with a real human when
 the prediction is uncertain.

How does GAN Training Scale with Batch Size?
--------------------------------------------

 Large minibatches have helped to scale up image classification — can they also help us scale up GANs?
 Large minibatches may be especially important for effectively using highly parallel hardware accelerators.

 At first glance, it seems like the answer should be yes — after all, the discriminator in most GANs is just an image classifier.
 Larger batches can accelerate training if it is bottlenecked on gradient noise.
 However, GANs have a separate bottleneck that classifiers don’t: the training procedure can diverge.
 Thus, we can state our problem:

Problem 6

How does GAN training scale with batch size?

How big a role does gradient noise play in GAN training?

Can GAN training be modified so that it scales better with batch size?

 There’s some evidence that increasing minibatch size improves quantitative results and reduces training time.
 If this phenomenon is robust, it would suggest that gradient noise is a dominating factor.
 However, this hasn’t been systematically studied, so we believe this question remains open.

 Can alternate training procedures make better use of large batches?
 Optimal Transport GANs theoretically have better convergence properties than normal GANs,
 but need a large batch size because they try to align batches of samples and training data.
 As a result, they seem like a promising candidate for scaling to very large batch sizes.

 Finally, asynchronous SGD could be a good alternative for making use of new hardware.
 In this setting, the limiting factor tends to be that gradient updates are computed on ‘stale’ copies of the parameters.
 But GANs seem to actually benefit from training on past parameter snapshots, so we might ask if
 asynchronous SGD interacts in a special way with GAN training.

What is the Relationship Between GANs and Adversarial Examples?
---------------------------------------------------------------

 It’s well known that image classifiers suffer from adversarial examples:
 human-imperceptible perturbations that cause classifiers to give the wrong output when added to images.
 It’s also now known that there are classification problems which can normally be efficiently learned,
 but are exponentially hard to learn robustly.

 Since the GAN discriminator is an image classifier, one might worry about it suffering from adversarial examples.
 Despite the large bodies of literature on GANs and adversarial examples,
 there doesn’t seem to be much work on how they relate.

 There is work on using GANs to generate adversarial examples, but this is not quite the same thing.

 Thus, we can ask the question:

Problem 7

How does the adversarial robustness of the discriminator affect GAN training?

 How can we begin to think about this problem?
 Consider a fixed discriminator **D**.
 An adversarial example for **D** would exist if
 there were a generator sample **G(z)** correctly classified as fake and
 a small perturbation **p** such that **G(z) + p** is classified as real.
 With a GAN, the concern would be that the gradient update for the generator would yield
 a new generator **G’** where **G’(z) = G(z) + p**.

 Is this concern realistic?
  shows that deliberate attacks on generative models can work,
 but we are more worried about something you might call an ‘accidental attack’.
 There are reasons to believe that these accidental attacks are less likely.
 First, the generator is only allowed to make one gradient update before
 the discriminator is updated again.
 In contrast, current adversarial attacks are typically run for tens of iterations.
 Second, the generator is optimized given a batch of samples from the prior, and this batch is different
 for every gradient step.
 Finally, the optimization takes place in the space of parameters of the generator rather than in pixel space.
 However, none of these arguments decisively rules out the generator creating adversarial examples.
 We think this is a fruitful topic for further exploration.

Open Questions about Generative Adversarial Networks

Summary: What we'd like to find out about GANs that we don't know yet.

Conditional Image Synthesis With Auxiliary Classifier GANs

Self-Attention Generative Adversarial Networks

Spectral Normalization for Generative Adversarial Networks

Large Scale GAN Training for High Fidelity Natural Image Synthesis

A style-based generator architecture for generative adversarial networks

Open Questions in Physics

Not especially famous, long-open problems which anyone can understand

Hilbert's Problems

Smale's Problems

Auto-Encoding Variational Bayes

NICE: Non-linear Independent Components Estimation

Density estimation using Real NVP

Glow: Generative Flow with Invertible 1x1 Convolutions

Normalizing Flows Tutorial

Pixel Recurrent Neural Networks

Conditional Image Generation with PixelCNN Decoders

PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

WaveNet: A Generative Model for Raw Audio

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Variational Inference with Normalizing Flows

Parallel WaveNet: Fast High-Fidelity Speech Synthesis

Flow-GAN: Bridging implicit and prescribed learning in generative models

Comparison of maximum likelihood and gan-based training of real nvps

Is Generator Conditioning Causally Related to GAN Performance?

Gradient-Based Learning Applied to Document Recognition

Learning Multiple Layers of Features from Tiny Images

An analysis of single-layer networks in unsupervised feature learning

Deep Learning Face Attributes in the Wild

ImageNet Large Scale Visual Recognition Challenge

PSA

Are GANs Created Equal? A Large-Scale Study

Disconnected Manifold Learning for Generative Adversarial Networks

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Improved training of wasserstein gans

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

MaskGAN: Better Text Generation via Filling in the ______

Geometric deep learning: going beyond Euclidean data

NetGAN: Generating Graphs via Random Walks

Synthesizing Audio with Generative Adversarial Networks

GANSynth: Adversarial Neural Audio Synthesis

OpenAI Five

Gradient descent GAN optimization is locally stable

Which Training Methods for GANs do actually Converge?

Understanding GANs: the LQG Setting

Global Convergence to the Equilibrium of GANs using Variational Inequalities

The Mechanics of n-Player Differentiable Games

Approximation and Convergence Properties of Generative Adversarial Learning

The Inductive Bias of Restricted f-GANs

On the Limitations of First-Order Approximation in GAN Dynamics

The Loss Surface of Multilayer Networks

GANGs: Generative Adversarial Network Games

Beyond Local Nash Equilibria for Adversarial Networks

An Online Learning Approach to Generative Adversarial Networks

Improved Techniques for Training GANs

GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium

A Note on the Inception Score

Towards GAN Benchmarks Which Require Generalization

On the Quantitative Analysis of Decoder-Based Generative Models

Geometry Score: A Method For Comparing Generative Adversarial Networks

Assessing Generative Models via Precision and Recall

Skill Rating for Generative Models

Discriminator Rejection Sampling

Revisiting Classifier Two-Sample Tests

Parametric Adversarial Divergences are Good Task Losses for Generative Modeling

Deconvolution and Checkerboard Artifacts

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Scaling sgd batch size to 32k for imagenet training

Train longer, generalize better: closing the generalization gap in large batch training of neural networks

Don't Decay the Learning Rate, Increase the Batch Size

Science and research policy at the end of Moore’s law

In-datacenter performance analysis of a tensor processing unit

Improving GANs using optimal transport

Large Scale Distributed Deep Networks

Deep learning with Elastic Averaging SGD

Staleness-aware Async-SGD for Distributed Deep Learning

Faster Asynchronous SGD

The Unusual Effectiveness of Averaging in GAN Training

Intriguing properties of neural networks

Adversarial examples from computational constraints

Adversarial examples for generative models

Towards deep learning models resistant to adversarial attacks


 Even if you have spent some time reading about machine learning, chances are that you have never heard of Gaussian processes.
 And if you have, rehearsing the basics is always a good way to refresh your memory.
 With this blog post we want to give an introduction to Gaussian processes and make the mathematical intuition behind them more approachable.

 Gaussian processes are a powerful tool in the machine learning toolbox.
 They allow us to make predictions about our data by incorporating prior knowledge.
 Their most obvious area of application is *fitting* a function to the data.
 This is called regression and is used, for example, in robotics or time series forecasting.
 But Gaussian processes are not limited to regression — they can also be extended to classification and clustering tasks.

 For a given set of training points, there are potentially infinitely many functions that fit the data.
 Gaussian processes offer an elegant solution to this problem by assigning a probability to each of these functions.
 The mean of this probability distribution then represents the most probable characterization of the data.
 Furthermore, using a probabilistic approach allows us to incorporate the confidence of the prediction into the regression result.

 We will first explore the mathematical foundation that Gaussian processes are built on — we invite you to follow along using the interactive figures and hands-on examples.
 They help to explain the impact of individual components, and show the flexibility of Gaussian processes.
 After following this article we hope that you will have a visual intuition on how Gaussian processes work and how you can configure them for different types of data.

Multivariate Gaussian distributions
-----------------------------------

 Before we can explore Gaussian processes, we need to understand the mathematical concepts they are based on.
 As the name suggests, the Gaussian distribution (which is often also referred to as *normal* distribution) is the basic building block of Gaussian processes.
 In particular, we are interested in the multivariate case of this distribution, where each random variable is distributed normally and their joint distribution is also Gaussian.
 The multivariate Gaussian distribution is defined by a mean vector μ\muμ and a covariance matrix Σ\SigmaΣ.
 You can see an interactive example of such distributions in [the figure below](#Multivariate).

 The mean vector μ\muμ describes the expected value of the distribution.
 Each of its components describes the mean of the corresponding dimension.
 Σ\SigmaΣ models the variance along each dimension and determines how the different random variables are correlated.
 The covariance matrix is always symmetric and positive semi-definite.
 The diagonal of Σ\SigmaΣ consists of the variance σi2\sigma\_i^2σi2 of the iii-th random variable.
 And the off-diagonal elements σij\sigma\_{ij}σij describe the correlation between the iii-th and jjj-th random variable.

X=[X1X2⋮Xn]∼N(μ,Σ)
 X = \begin{bmatrix} X\_1 \\ X\_2 \\ \vdots \\ X\_n \end{bmatrix} \sim \mathcal{N}(\mu, \Sigma)
 X=⎣⎢⎢⎡X1X2⋮Xn⎦⎥⎥⎤∼N(μ,Σ)
We say XXX follows a normal distribution. The covariance matrix Σ\SigmaΣ describes the shape of the distribution. It is defined in terms of the expected value EEE:

Σ=Cov(Xi,Xj)=E[(Xi−μi)(Xj−μj)T]
 \Sigma = \text{Cov}(X\_i, X\_j) = E \left[ (X\_i - \mu\_i)(X\_j - \mu\_j)^T \right]
 Σ=Cov(Xi,Xj)=E[(Xi−μi)(Xj−μj)T]
Visually, the distribution is centered around the mean and the covariance matrix defines its shape. The [following figure](#Multivariate) shows the influence of these parameters on a two-dimensional Gaussian distribution. The variances for each random variable are on the diagonal of the covariance matrix, while the other values show the covariance between them.

 Gaussian distributions are widely used to model the real world.
 For example, we can employ them to describe errors of measurements or phenomena under the assumptions of the *central limit theorem*

 One of the implications of this theorem is that a collection of independent, identically distributed random variables with finite variance have a mean that is distributed normally.
 A good introduction to the central limit theorem is given by [this video](https://www.khanacademy.org/math/ap-statistics/sampling-distribution-ap/sampling-distribution-mean/v/central-limit-theorem) from [Khan Academy](https://www.khanacademy.org).
 .
 In the next section we will take a closer look at how to manipulate Gaussian distributions and extract useful information from them.

### Marginalization and Conditioning

 Gaussian distributions have the nice algebraic property of being closed under conditioning and marginalization.
 Being closed under conditioning and marginalization means that the resulting distributions from these operations are also Gaussian, which makes many problems in statistics and machine learning tractable.
 In the following we will take a closer look at both of these operations, as they are the foundation for Gaussian processes.

*Marginalization* and *conditioning* both work on subsets of the original distribution and we will use the following notation:

PX,Y=[XY]∼N(μ,Σ)=N([μXμY],[ΣXXΣXYΣYXΣYY])P\_{X,Y} = \begin{bmatrix} X \\ Y \end{bmatrix} \sim \mathcal{N}(\mu, \Sigma) = \mathcal{N} \left( \begin{bmatrix} \mu\_X \\ \mu\_Y \end{bmatrix}, \begin{bmatrix} \Sigma\_{XX} \, \Sigma\_{XY} \\ \Sigma\_{YX} \, \Sigma\_{YY} \end{bmatrix} \right)PX,Y=[XY]∼N(μ,Σ)=N([μXμY],[ΣXXΣXYΣYXΣYY])
With XXX and YYY representing subsets of original random variables.

Through *marginalization* we can extract partial information from multivariate probability distributions. In particular, given a normal probability distribution P(X,Y)P(X,Y)P(X,Y) over vectors of random variables XXX and YYY, we can determine their marginalized probability distributions in the following way:

X∼N(μX,ΣXX)Y∼N(μY,ΣYY)
 \begin{aligned}
 X &\sim \mathcal{N}(\mu\_X, \Sigma\_{XX}) \\
 Y &\sim \mathcal{N}(\mu\_Y, \Sigma\_{YY})
 \end{aligned}
 XY∼N(μX,ΣXX)∼N(μY,ΣYY)

 The interpretation of this equation is that each partition XXX and YYY only depends on its corresponding entries in μ\muμ and Σ\SigmaΣ.
 To marginalize out a random variable from a Gaussian distribution we can simply drop the variables from μ\muμ and Σ\SigmaΣ.

pX(x)=∫ypX,Y(x,y)dy=∫ypX∣Y(x∣y)pY(y)dy
 p\_X(x) = \int\_y p\_{X,Y}(x,y)dy = \int\_y p\_{X|Y}(x|y) p\_Y(y) dy
 pX(x)=∫ypX,Y(x,y)dy=∫ypX∣Y(x∣y)pY(y)dy
The way to interpret this equation is that if we are interested in the probability density of
 X=xX = xX=x, we need to consider all possible outcomes of
 YYY that can jointly lead to the result
 The corresponding [Wikipedia
 article](https://en.wikipedia.org/wiki/Marginal_distribution) has a good description of the marginal distribution, including several examples.
 .

 Another important operation for Gaussian processes is *conditioning*.
 It is used to determine the probability of one variable depending on another variable.
 Similar to marginalization, this operation is also closed and yields a modified Gaussian distribution.
 This operation is the cornerstone of Gaussian processes since it allows Bayesian inference, which we will talk about in the [next section](#GaussianProcesses).
 Conditioning is defined by:

X∣Y∼N(μX+ΣXYΣYY−1(Y−μY),ΣXX−ΣXYΣYY−1ΣYX)Y∣X∼N(μY+ΣYXΣXX−1(X−μX),ΣYY−ΣYXΣXX−1ΣXY)
 \begin{aligned}
 X|Y &\sim \mathcal{N}(\:\mu\_X + \Sigma\_{XY}\Sigma\_{YY}^{-1}(Y - \mu\_Y),\: \Sigma\_{XX}-\Sigma\_{XY}\Sigma\_{YY}^{-1}\Sigma\_{YX}\:) \\
 Y|X &\sim \mathcal{N}(\:\mu\_Y + \Sigma\_{YX}\Sigma\_{XX}^{-1}(X - \mu\_X),\: \Sigma\_{YY}-\Sigma\_{YX}\Sigma\_{XX}^{-1}\Sigma\_{XY}\:) \\
 \end{aligned}
 X∣YY∣X∼N(μX+ΣXYΣYY−1(Y−μY),ΣXX−ΣXYΣYY−1ΣYX)∼N(μY+ΣYXΣXX−1(X−μX),ΣYY−ΣYXΣXX−1ΣXY)

 Note that the new mean only depends on the conditioned variable, while the covariance matrix is independent from this variable.

 Now that we have worked through the necessary equations, we will think about how we can understand the two operations visually.
 While marginalization and conditioning can be applied to multivariate distributions of many dimensions, it makes sense to consider the two-dimensional case as shown in the [following figure](#MarginalizationConditioning).
 Marginalization can be seen as integrating along one of the dimensions of the Gaussian distribution, which is in line with the general definition of the marginal distribution.
 Conditioning also has a nice geometric interpretation — we can imagine it as making a cut through the multivariate distribution, yielding a new Gaussian distribution with fewer dimensions.

 A bivariate normal distribution in the center.
 On the left you can see the result of marginalizing this distribution for Y, akin to integrating along the X axis. On the right you can see the distribution conditioned on a given X, which is similar to a cut through the original distribution. The Gaussian distribution and the conditioned variable can be changed by dragging the handles.

Gaussian Processes
------------------

 Now that we have recalled some of the basic properties of multivariate Gaussian distributions, we will combine them together to define Gaussian processes, and show how they can be used to tackle regression problems.

 First, we will move from the continuous view to the discrete representation of a function:
 rather than finding an implicit function, we are interested in predicting the function values at concrete points, which we call *test points* XXX.
 So how do we derive this functional view from the multivariate normal distributions that we have considered so far?
 Stochastic processes, such as Gaussian processes, are essentially a set of random variables.
 In addition, each of these random variables has a corresponding index iii.
 We will use this index to refer to the iii-th dimension of our nnn-dimensional multivariate distributions.
 The [following figure](#DimensionSwap) shows an example of this for two dimensions:

 Here, we have a two-dimensional normal distribution.
 Each dimension xix\_ixi is assigned an index i∈{1,2}i \in \{1,2\}i∈{1,2}.
 You can drag the handles to see how a particular sample (left) corresponds to functional values (right).
 This representation also allows us to understand the connection between the covariance and the resulting values:
 the underlying Gaussian distribution has a positive covariance between x1x\_1x1 and x2x\_2x2 — this means that x2x\_2x2 will increases as x1x\_1x1 gets larger and vice versa.
 You can also drag the handles in the figure to the right and observe the probability of such a configuration in the figure to the left.

 Now, the goal of Gaussian processes is to learn this underlying distribution from *training data*.
 Respective to the test data XXX, we will denote the training data as YYY.
 As we have mentioned before, the key idea of Gaussian processes is to model the underlying distribution of XXX together with YYY as a multivariate normal distribution.
 That means that the joint probability distribution PX,YP\_{X,Y}PX,Y spans the space of possible function values for the function that we want to predict.
 Please note that this joint distribution of test and training data has ∣X∣+∣Y∣|X| + |Y|∣X∣+∣Y∣ dimensions.

 In order to perform regression on the training data, we will treat this problem as *Bayesian inference*.
 The essential idea of Bayesian inference is to update the current hypothesis as new information becomes available.
 In the case of Gaussian processes, this information is the training data.
 Thus, we are interested in the conditional probability PX∣YP\_{X|Y}PX∣Y.
 Finally, we recall that Gaussian distributions are closed under conditioning — so PX∣YP\_{X|Y}PX∣Y is also distributed normally.

 Now that we have the basic framework of Gaussian processes together, there is only one thing missing:
 how do we set up this distribution and define the mean μ\muμ and the covariance matrix Σ\SigmaΣ?
 The covariance matrix Σ\SigmaΣ is determined by its *covariance function* kkk, which is often also called the *kernel* of the Gaussian process.
 We will talk about this in detail in the next section.
 But before we come to this, let us reflect on how we can use multivariate Gaussian distributions to estimate function values.
 The [following figure](#PriorFigure) shows an example of this using ten test points at which we want to predict our function:

 In Gaussian processes we treat each test point as a random variable.
 A multivariate Gaussian distribution has the same number of dimensions as the number of random variables.
 Since we want to predict the function values at
 ∣X∣=N|X|=N∣X∣=N
 test points, the corresponding multivariate Gaussian distribution is also
 NNN
 -dimensional.
 Making a prediction using a Gaussian process ultimately boils down to drawing samples from this distribution.
 We then interpret the iii-th component of the resulting vector as the function value corresponding to the iii-th test point.

### Kernels

 Recall that in order to set up our distribution, we need to define μ\muμ and Σ\SigmaΣ.
 In Gaussian processes it is often assumed that μ=0\mu = 0μ=0, which simplifies the necessary equations for conditioning.
 We can always assume such a distribution, even if μ≠0\mu \neq 0μ≠0, and add μ\muμ back to the resulting function values after the prediction step.
 This process is also called *centering* of the data.
 So configuring μ\muμ is straight forward — it gets more interesting when we look at the other parameter of the distribution.

 The clever step of Gaussian processes is how we set up the covariance matrix Σ\SigmaΣ.
 The covariance matrix will not only describe the shape of our distribution, but ultimately determines the characteristics of the function that we want to predict.
 We generate the covariance matrix by evaluating the kernel kkk, which is often also called *covariance function*, pairwise on all the points.
 The kernel receives two points t,t′∈Rnt,t’ \in \mathbb{R}^nt,t′∈Rn as an input and returns a similarity measure between those points in the form of a scalar:

k:Rn×Rn→R,Σ=Cov(X,X′)=k(t,t′)
 k: \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R},\quad
 \Sigma = \text{Cov}(X,X’) = k(t,t’)
 k:Rn×Rn→R,Σ=Cov(X,X′)=k(t,t′)

 We evaluate this function for each pairwise combination of the test points to retrieve the covariance matrix.
 This step is also depicted in the [figure above](#PriorFigure).
 In order to get a better intuition for the role of the kernel, let’s think about what the entries in the covariance matrix describe.
 The entry Σij\Sigma\_{ij}Σij describes how much influence the iii-th and jjj-th point have on each other.
 This follows from the definition of the multivariate Gaussian distribution, which states that Σij\Sigma\_{ij}Σij defines the correlation between the iii-th and the jjj-th random variable.
 Since the kernel describes the similarity between the values of our function, it controls the possible shape that a fitted function can adopt.
 Note that when we choose a kernel, we need to make sure that the resulting matrix adheres to the properties of a covariance matrix.

 Kernels are widely used in machine learning, for example in *support vector machines*.
 The reason for this is that they allow similarity measures that go far beyond the standard euclidean distance (L2L2L2-distance).
 Many of these kernels conceptually embed the input points into a higher dimensional space in which they then measure the similarityIf the kernel follows Mercer’s theorem it can be used to define a Hilbert space. More information on this can be found on [Wikipedia](https://en.wikipedia.org/wiki/Kernel_method)..
 The [following figure](#MultipleKernels) shows examples of some common kernels for Gaussian processes.
 For each kernel, the covariance matrix has been created from N=25N=25N=25 linearly-spaced values ranging from [−5,5][-5,5][−5,5]. Each entry in the matrix shows the covariance between points in the range of [0,1][0,1][0,1].

This figure shows various kernels that can be used with Gaussian processes. Each kernel has different
 parameters, which can be changed by adjusting the according sliders. When grabbing a slider,
 information on how the current parameter influences the kernel will be shown on the right.

 Kernels can be separated into *stationary* and *non-stationary* kernels. *Stationary* kernels, such
 as the RBF kernel or the periodic kernel, are functions invariant to translations, and the covariance of two points is only
 dependent on their relative position. *Non-stationary* kernels, such as the linear kernel, do not have this
 constraint and depend on an absolute location. The stationary nature of the RBF kernel can be observed in the
 banding around the diagonal of its covariance matrix (as shown in [this figure](#MultipleKernels)). Increasing the length parameter increases the banding, as
 points further away from each other become more correlated. For the periodic kernel, we have an additional parameter
 PPP that determines the periodicity, which controls the distance between each repetition of the function.
 In contrast, the parameter CCC of the linear kernel allows us to change the point on which all functions hinge.

 There are many more kernels that can describe different classes of functions, which can be used to model the desired shape of the function.
 A good overview of different kernels is given by Duvenaud.
 It is also possible to combine several kernels — but we will get to this later.

### Prior Distribution

 We will now shift our focus back to the original task of regression.
 As we have mentioned earlier, Gaussian processes define a probability distribution over possible functions.
 In [this figure above](#DimensionSwap), we show this connection:
 each sample of our multivariate normal distribution represents one realization of our function values.
 Because this distribution is a multivariate Gaussian distribution, the distribution of functions is normal.
 Recall that we usually assume μ=0\mu=0μ=0.
 For now, let’s consider the case where we have not yet observed any training data.
 In the context of Bayesian inference, this is called the *prior* distribution PXP\_XPX.

 If we have not yet observed any training examples, this distribution revolves around μ=0\mu=0μ=0, according to our original assumption.
 The prior distribution will have the same dimensionality as the number of test points N=∣X∣N = |X|N=∣X∣.
 We will use the kernel to set up the covariance matrix, which has the dimensions N×NN \times NN×N.

 In the previous section we have looked at examples of different kernels.
 The kernel is used to define the entries of the covariance matrix.
 Consequently, the covariance matrix determines which type of functions from the space of all possible functions are more probable.
 As the prior distribution does not yet contain any additional information, it is perfect to visualize the influence of the kernel on the distribution of functions.
 The [following figure](#Prior) shows samples of potential functions from prior distributions that were created using different kernels:

 Clicking on the graph results in continuous samples drawn from a
 Gaussian process using the selected
 kernel. After each draw, the previous sample fades into the background. Over time, it is possible to see that
 functions are distributed normally around the mean µ .

 Adjusting the parameters allows you to control the shape of the resulting functions.
 This also varies the confidence of the prediction.
 When decreasing the variance σ\sigmaσ, a common parameter for all kernels, sampled functions are more concentrated around the mean μ\muμ.
 For the *Linear* kernel, setting the variance σb=0\sigma\_b=0σb=0 results in a set of functions constrained to perfectly intersect the offset point ccc.
 If we set σb=0.2\sigma\_b=0.2σb=0.2 we can model uncertainty, resulting in functions that pass close to ccc.

### Posterior Distribution

 So what happens if we observe training data?
 Let’s get back to the model of Bayesian inference, which states that we can incorporate this additional information into our model, yielding the *posterior* distribution PX∣YP\_{X|Y}PX∣Y.
 We will now take a closer look at how to do this for Gaussian processes.

 First, we form the joint distribution PX,YP\_{X,Y}PX,Y between the test points XXX and the training points YYY.
 The result is a multivariate Gaussian distribution with dimensions ∣Y∣+∣X∣|Y| + |X|∣Y∣+∣X∣.
 As you can see in the [figure below](#PosteriorFigure), we concatenate the training and the test points to compute the corresponding covariance matrix.

 For the next step we need one operation on Gaussian distributions that we have defined earlier.
 Using *conditioning* we can find PX∣YP\_{X|Y}PX∣Y from PX,YP\_{X,Y}PX,Y.
 The dimensions of this new distribution matches the number of test points NNN and the distribution is also normal.
 It is important to note that conditioning leads to derived versions of the mean and the standard deviation: X∣Y∼N(μ′,Σ′)X|Y \sim \mathcal{N}(\mu’, \Sigma’)X∣Y∼N(μ′,Σ′).
 More details can be found in the [related section](#MargCond) on conditioning multivariate Gaussian distributions.
 The intuition behind this step is that the training points constrain the set of functions to those that pass through the training points.

 As mentioned before, the conditional distribution PX∣YP\_{X|Y}PX∣Y forces the set of functions to precisely pass through each training point.
 In many cases this can lead to fitted functions that are unnecessarily complex.
 Also, up until now, we have considered the training points YYY to be perfect measurements.
 But in real-world scenarios this is an unrealistic assumption, since most of our data is afflicted with measurement errors or uncertainty.
 Gaussian processes offer a simple solution to this problem by modeling the error of the measurements.
 For this, we need to add an error term ϵ∼N(0,ψ2)\epsilon \sim \mathcal{N}(0, \psi^2)ϵ∼N(0,ψ2) to each of our training points:

Y=f(X)+ϵ
 Y = f(X) + \epsilon
 Y=f(X)+ϵ

 We do this by slightly modifying the setup of the joint distribution PX,YP\_{X,Y}PX,Y:

PX,Y=[XY]∼N(0,Σ)=N([00],[ΣXXΣXYΣYXΣYY+ψ2I])P\_{X,Y} = \begin{bmatrix} X \\ Y \end{bmatrix} \sim \mathcal{N}(0, \Sigma) = \mathcal{N} \left( \begin{bmatrix} 0 \\ 0 \end{bmatrix}, \begin{bmatrix} \Sigma\_{XX} & \Sigma\_{XY} \\ \Sigma\_{YX} & \Sigma\_{YY}+\psi^2I \end{bmatrix} \right)PX,Y=[XY]∼N(0,Σ)=N([00],[ΣXXΣYXΣXYΣYY+ψ2I])

 Again, we can use conditioning to derive the predictive distribution PX∣YP\_{X|Y}PX∣Y.
 In this formulation, ψ\psiψ is an additional parameter of our model.

 Analogous to the prior distribution, we could obtain a prediction for our function values by sampling from this distribution.
 But, since sampling involves randomness, the resulting fit to the data would not be deterministic and our prediction could end up being an outlier.
 In order to make a more meaningful prediction we can use the other basic operation of Gaussian distributions.

 Through the *marginalization* of each random variable, we can extract the respective mean function value μi′\mu’\_iμi′ and standard deviation σi′=Σii′\sigma’\_i = \Sigma’\_{ii}σi′=Σii′ for the iii-th test point.
 In contrast to the prior distribution, where we set the mean to μ=0\mu=0μ=0, the result of conditioning the joint distribution of test and training data will most likely have a non-zero mean μ′≠0\mu’ \neq 0μ′≠0.
 Extracting μ′\mu’μ′ and σ′\sigma’σ′ does not only lead to a more meaningful prediction, it also allows us to make a statement about the confidence of the prediction.

 The [following figure](#Posterior) shows an example of the conditional distribution.
 At first, no training points have been observed.
 Accordingly, the mean prediction remains at 000 and the standard deviation is the same for each test point.
 By hovering over the covariance matrix you can see the influence of each point on the current test point.
 As long as no training points have been observed, the influence of neighboring points is limited locally.

 The training points can be activated by clicking on them, which leads to a constrained distribution.
 This change is reflected in the entries of the covariance matrix, and leads to an adjustment of the mean and the standard deviation of the predicted function.
 As we would expect, the uncertainty of the prediction is small in regions close to the training data and grows as we move further away from those points.

 In the constrained covariance matrix, we can see that the correlation of neighbouring points is affected by the
 training data.
 If a predicted point lies on the training data, there is no correlation with other points.
 Therefore, the function must pass directly through it.
 Predicted values further away are also affected by the training data — proportional to their distance.

### Combining different kernels

 As described earlier, the power of Gaussian processes lies in the choice of the kernel function.
 This property allows experts to introduce domain knowledge into the process and lends Gaussian processes their flexibility to capture trends in the training data.
 For example, by choosing a suitable bandwidth for the RBF kernel, we can control how smooth the resulting function will be.

 A big benefit that kernels provide is that they can be combined together, resulting in a more specialized kernel.
 The decision which kernel to use is highly dependent on prior knowledge about the data, e.g. if certain characteristics are expected.
 Examples for this would be stationary nature, or global trends and patterns.
 As introduced in the [section on kernels](#Kernels), stationary means that a kernel is translation invariant and therefore not dependent on the index iii.
 This also means that we cannot model global trends using a strictly stationary kernel.
 Remember that the covariance matrix of Gaussian processes has to be positive semi-definite.
 When choosing the optimal kernel combinations, all methods that preserve this property are allowed.
 The most common kernel combinations would be addition and multiplication.

 Let’s consider two kernels, a linear kernel klink\_{\text{lin}}klin and a periodic kernel kperk\_{\text{per}}kper, for example.
 This is how we would multiply the two:

k∗(t,t′)=klin(t,t′)⋅kper(t,t′)
 k^{\ast}(t,t’) = k\_{\text{lin}}(t,t’) \cdot k\_{\text{per}}(t,t’)
 k∗(t,t′)=klin(t,t′)⋅kper(t,t′)

 However, combinations are not limited to the above example, and there are more possibilities such as concatenation or composition with a function.
 To show the impact of a kernel combination and how it might retain qualitative features of the individual kernels, take a look at the [figure below](#KernelCombinationsStatic).
 If we add a periodic and a linear kernel, the global trend of the linear kernel is incorporated into the combined kernel.
 The result is a periodic function that follows a linear trend.
 When combining the same kernels through multiplication instead, the result is a periodic function with a linearly growing amplitude away from linear kernel parameter ccc.

 If we draw samples from a combined linear and periodic kernel, we can observe the different retained characteristics in the new sample.
 Addition results in a periodic function with a global trend, while the multiplication increases the periodic amplitude outwards.

 Knowing more about how kernel combinations influence the shape of the resulting distribution, we can move on to a more complex example.
 In the [figure below](#KernelCombinations), the observed training data has an ascending trend with a periodic deviation.
 Using only a linear kernel, we can mimic a normal linear regression of the points.
 At first glance, the RBF kernel accurately approximates the points.
 But since the RBF kernel is stationary it will always return to μ=0\mu=0μ=0 in regions further away from observed training data.
 This decreases the accuracy for predictions that reach further into the past or the future.
 An improved model can be created by combining the individual kernels through addition, which maintains both the periodic nature and the global ascending trend of the data.
 This procedure can be used, for example, in the analysis of weather data.

 Using the checkboxes, different kernels can be combined to form a new Gaussian process. Only by using a
 combination of kernels, it is possible to capture the characteristics of more complex training data.

 As discussed in the [section about GPs](#GaussianProcesses), a Gaussian process can model uncertain observations.
 This can be seen when only selecting the linear kernel, as it allows us to perform linear regression even if more than two points have been observed, and not all functions have to pass directly through the observed training data.

Conclusion
----------

 With this article, you should have obtained an overview of Gaussian processes, and developed a deeper
 understanding on how they work.
 As we have seen, Gaussian processes offer a flexible framework for regression and several extensions exist that
 make them even more versatile.

 For instance, sometimes it might not be possible to describe the kernel in simple terms.
 To overcome this challenge, learning specialized kernel functions from the underlying data, for example by using deep learning, is an area of ongoing research.
 Furthermore, links between Bayesian inference, Gaussian processes and deep learning have been described in several papers.
 Even though we mostly talk about Gaussian processes in the context of regression, they can be adapted for
 different purposes, e.g. *model-peeling* and hypothesis testing.
 By comparing different kernels on the dataset, domain experts can introduce additional knowledge through
 appropriate combination and parameterization of the kernel.

 If we have sparked your interest, we have compiled a list of further [blog posts](#FurtherReading) on the topic of Gaussian processes.
 In addition, we have linked two [Python notebooks](#FurtherReading) that will give you some hands-on experience and help you to get started right away.

A Visual Exploration of Gaussian Processes

Jochen Görtler
Rebecca Kehlbeck
Oliver Deussen

Summary: How to turn a collection of small building blocks into a versatile tool for solving regression problems.

Gaussian Processes in Machine Learning

Gaussian Processes for Object Categorization

Clustering Based on Gaussian Processes

The Nature of Statistical Learning Theory

Automatic model construction with Gaussian processes

Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

Deep Kernel Learning

Deep Neural Networks as Gaussian Processes

Deep Gaussian Processes