A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Two Examples of Useful, Non-Robust Features

Abstract: This article is part of a discussion of the Ilyas et al. paper         “Adversarial examples are not bugs, they are features”.         You can learn more in the                      main discussion article         .

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

 Ilyas et al.  define a *feature* as a function fff that

 label. But in the presence of an adversary Ilyas et al.  argues

 the metric that truly matters is a feature’s *robust usefulness*,

E[inf∥δ∥≤ϵyf(x+δ)],

 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)\right],

 E[∥δ∥≤ϵinfyf(x+δ)],

 its correlation with the label while under attack. Ilyas et al.

 like?

### Non-Robust Features in Linear Models

 nonlinear models encountered in deep learning. As Ilyas et al

 to linear features of the form:

 \text{and} \quad \mathbf{E}[x] = 0.

 f(x)=∥a∥ΣaTxwhereΣ=E[xxT]andE[x]=0.

 The robust usefulness of a linear feature admits an elegant decomposition

  This

 \begin{aligned}

 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)\right] &

 =\mathbf{E}\left[yf(x)+\inf\_{\|\delta\|\leq\epsilon}yf(\delta)\right]\\

 &

 \end{aligned}

 into two terms:

 .undomargin {

 position: relative;

 left: -1em;

 top: 0.2em;

 }

E[inf∥δ∥≤ϵyf(x+δ)]

 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)\right]

 E[∥δ∥≤ϵinfyf(x+δ)]

===

E[yf(x)]\mathop{\mathbf{E}[yf(x)]}E[yf(x)]

−-−

ϵ∥a∥∗∥a∥Σ\epsilon\frac{\|a\|\_{\*}}{\|a\|\_{\Sigma}}ϵ∥a∥Σ∥a∥∗

 The robust usefulness of a feature

 the correlation of the feature with the label

 the feature’s non-robustness

 dimensional plot.

 Subject to an

 L\_2

 adversery, observe that high frequency features are both less useful and

 less robust.

  Useful Non-Useful A  B  C  D  E  F

 Pareto frontier of points in the non-robustness and usefulness space.

 positive label.

 \log \left( \frac{\|a\_i\|\_\Sigma}{\|a\_i\|} \right) =

 \log(\lambda\_i) Feature’s log robustness. When

 a\_i's

 are the

 i^{th}

 eigenvalues of

 \Sigma

 , the robustness reduces to the

 i^{th}

 singular value of

 \lambda\_i   ABCDEFf-12-11-10-9-8-7-6-5-4-3-4-3-2-10

 We demonstrate two constructions:

 (1-\alpha) \cdot a\_{\text{non-robust}} + \alpha \cdot a\_{\text{robust}},

 It is surprising, thus, that the experiments of Madry et al.

 To cite Ilyas et al.’s response, please cite their

**Response Summary**: The construction of explicit non-robust features is

 the useful non-robust features detected by our experiments. We also agree that

 non-robust features arising as “distractors” is indeed not precluded by our

 theoretical framework, even if it is precluded by our experiments.

 This simple theoretical framework sufficed for reasoning about and

 predicting the outcomes of our experiments

 We also presented a theoretical setting where we can

 analyze things fully rigorously in Section 4 of our paper..

 However, this comment rightly identifies finding a more comprehensive

 definition of feature as an important future research direction.

**Response**: These experiments (visualizing the robustness and

 corroborate the existence of useful, non-robust features and make progress

 towards visualizing what these non-robust features actually look like.

We also appreciate the point made by the provided construction of non-robust

 features (as defined in our theoretical framework) that are combinations of

 useful+robust and useless+non-robust features. Our theoretical framework indeed

 enables such a scenario, even if — as the commenter already notes — our

 framework technically captures.) Specifically, in such a scenario, during the

 construction of the D^det\widehat{\mathcal{D}}\_{det}D

det dataset, only the non-robust

 and useless term of the feature would be flipped. Thus, a classifier trained on

 such a dataset would associate the predictive robust feature with the

 *wrong* label and would thus not generalize on the test set. In contrast,

det

 do generalize.

Overall, our focus while developing our theoretical framework was on

 the comment points out, putting forth a theoretical framework that captures

 non-robust features in a very precise way is an important future research

 direction in itself.

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'_ Two Examples of Useful, Non-Robust Features.html


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Robust Feature Leakage

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

[Comment by Ilyas et al.](#rebuttal)

 Ilyas et al.  report a surprising result: a model trained on

### Lower Bounding Leakage

 Our technique for quantifying leakage consisting of two steps:

 specify.

2. Next, we train a linear classifier as per ,

 Equation 3 on the datasets D^det\hat{\mathcal{D}}\_{\text{det}}D^det and

 D^rand\hat{\mathcal{D}}\_{\text{rand}}D^rand (Defined , Table 1) on

 these robust features *only*.

 Since Ilyas et al.  only specify robustness in the two class

 setting:

 **Specification 1**

For at least one of the

 classes, the feature is γ\gammaγ-robustly useful   with

 **Specification 2**

 that remain static in a neighborhood of radius 0.25 on the L2L\_2L2 norm ball.

 CIFAR test set incurs an accuracy of **23.5%** (out of 88%). Doing the same on

 features, e.g. from a robust deep neural network.

 The results of D^det\hat{\mathcal{D}}\_{\text{det}}D^det in Table 1 of

 however, are on stronger footing. We find no evidence of

feature leakage (in fact, we find negative leakage — an influx!). We

thus conclude that it is plausible the majority of the accuracy is

driven by

 non-robust features, exactly the thesis of .

 To cite Ilyas et al.’s response, please cite their

**Response Summary**: This

 is a valid concern that was actually one of our motivations for creating the

 D^det\widehat{\mathcal{D}}\_{det}D

det dataset (which, as the comment notes, actually

 has *misleading* robust features). The provided experiment further

 improves our understanding of the underlying phenomenon.

**Response**: This comment raises a valid concern which was in fact one of

 the primary reasons for designing the D^det\widehat{\mathcal{D}}\_{det}D

det dataset.

rand

 dataset: assign each input a random target label and do PGD towards that label.

 Note that unlike the D^det\widehat{\mathcal{D}}\_{det}D

det dataset (in which the

rand

 dataset allows for robust features to actually have a (small) positive

 correlation with the label.

To see how this can happen, consider the following simple setting: we have a

 (as in the dataset D^rand\widehat{\mathcal{D}}\_{rand}D

rand) would make this feature

 targeted attack might in this case induce some correlation with the

 to correctly classify new inputs.

In other words, starting from a dataset with no features, one can encode

 robust features within small perturbations. In contrast, in the

 D^det\widehat{\mathcal{D}}\_{det}D

det dataset, the robust features are *correlated

 with the original label* (since the labels are permuted) and since they are

 robust, they cannot be flipped to correlate with the newly assigned (wrong)

 label. Still, the D^rand\widehat{\mathcal{D}}\_{rand}D

rand dataset enables us to show

 that (a) PGD-based adversarial examples actually alter features in the data and

 (b) models can learn from human-meaningless/mislabeled training data. The

 D^det\widehat{\mathcal{D}}\_{det}D

det dataset, on the other hand, illustrates that the

 non-robust features are actually sufficient for generalization and can be

 preferred over robust ones in natural settings.

The experiment put forth in the comment is a clever way of showing that such

 leakage is indeed possible. However, we want to stress (as the comment itself

 does) that robust feature leakage does *not* have an impact on our main

 thesis — the D^det\widehat{\mathcal{D}}\_{det}D

det dataset explicitly controls

 for robust

 feature leakage (and in fact, allows us to quantify the models’ preference for

 robust features vs non-robust features — see Appendix D.6 in the

 [paper](https://arxiv.org/abs/1905.02175)).

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features' Robust Feature Leakage.html


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Discussion and Author Responses

Logan Engstrom
Andrew Ilyas
Aleksander Madry
Shibani Santurkar
Brandon Tran
Dimitris Tsipras

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

[Comment by Ilyas et al.](#rebuttal)

 We want to thank all the commenters for the discussion and for spending time

 designing experiments analyzing, replicating, and expanding upon our results.

 These comments helped us further refine our understanding of adversarial

 examples (e.g., by visualizing useful non-robust features or illustrating how

 robust models are successful at downstream tasks), but also highlighted aspects

 of our exposition that could be made more clear and explicit.

 Our response is organized as follows: we first recap the key takeaways from

 our paper, followed by some clarifications that this discussion brought to

 with a quick summary.

 We also recall some terminology from

 [our paper](https://arxiv.org/abs/1905.02175) that features in our responses:

 *Datasets*: Our experiments involve the following variants of the given

 dataset DDD (consists of sample-label pairs (xxx, yyy))  The

 exact details for construction of the datasets can be found in our

 [paper](https://arxiv.org/abs/1905.02175), and

 the datasets themselves can be downloaded at <http://git.io/adv-datasets> :

* D^R\widehat{\mathcal{D}}\_{R}D

R: Restrict each sample xxx to features that are used by a *robust*

 model.

* D^NR\widehat{\mathcal{D}}\_{NR}D

NR: Restrict each sample xxx to features that are used by a *standard*

 model.

* D^det\widehat{\mathcal{D}}\_{det}D

det: Adversarially perturb each sample xxx using a standard model in a

 *consistent manner* towards class y+1modCy + 1\mod Cy+1modC.

* D^rand\widehat{\mathcal{D}}\_{rand}D

rand: Adversarially perturb each sample xxx using a standard model

 towards a

 *uniformly random* class.

Main points

-----------

### *Takeaway #1:* Adversarial examples as innate

 brittleness vs. useful features (sensitivity vs reliance)

The goal of our experiments with non-robust features is to understand

 how adversarial examples fit into the following two worlds:

* **World 1: Adversarial examples exploit directions irrelevant for

 classification.** In this world, adversarial examples arise from

 sensitivity to a signal that is unimportant for classification. For

 instance, suppose there is a feature f(x)f(x)f(x) that is not generalizing

 on the dataNote that f(x)f(x)f(x) could be correlated with the label

 in the training set but not in expectation/on the test

 set., but the model for some reason puts a lot of weight on

 it,  *i.e., this sensitivity is an aberration “hallucinated” by the

 model*. Adversarial examples correspond to perturbing the input

 to change this feature by a small amount. This perturbation, however,

 would be orthogonal to how the model actually typically makes

 predictions (on natural data). (Note that this is just a single

 illustrative example — the key characteristic of this world is that

 features “flipped” when making adversarial examples are separate from the

 ones actually used to classify inputs.)

* **World 2: Adversarial examples exploit features that are useful for

 classification.** In this world, adversarial perturbations

 can correspond to changes in the input that manipulate features relevant to

 classification. Thus, models base their (mostly correct) predictions on

 features that can be altered via small perturbations.

 Recent works provide some theoretical evidence that adversarial examples

 can arise from finite-sample overfitting

 or

 other concentration of

 measure-based phenomena, thus

 supporting

 the “World 1” viewpoint on

 adversarial examples. The question is: is “World 1” the right way to

 think about adversarial examples? If so, this would be good news — under

 this mindset, adversarial robustness might just be a matter of getting

 better, “bug-free” models (for example, by reducing overfitting).

 Our findings show, however, that the “World 1” mindset alone does not

 fully capture adversarial vulnerability; “World 2“ must be taken into

 account. Adversarial examples can — and do, if generated via standard

 methods — rely on “flipping” features that are actually useful for

 classification. Specifically, we show that by relying *only* on

 perturbations corresponding to standard first-order adversarial attacks

 one can learn models that generalize to the test set. This means that

 these perturbations truly correspond to directions that are relevant for

 classifying new, unmodified inputs from the dataset. In summary, our

 message is:

**Adversarial vulnerability can arise from

 flipping features in the data that are useful for

 classification of *correct* inputs.**

In particular, note that our experiments (training on the

 D^rand\widehat{\mathcal{D}}\_{rand}D

rand and D^det\widehat{\mathcal{D}}\_{det}D

det

 datasets) would not have the same result in World 1. Concretely, in

 coordinate f(x)f(x)f(x) that is not generalizing for “natural images.” Then,

 adversarial examples towards either class can be made by simply making

 f(x)f(x)f(x) slightly positive or slightly negative. However, a classifier

 learned from these adversarial examples would  *not*  generalize to

 the true dataset (since it would learn to depend on a feature that is not

 useful on natural images).

### *Takeaway #2*: Learning from “meaningless” data

Another implication of our experiments is that models may not even

 *need* any information which we as humans view as “meaningful” in order

 to do well (in the generalization sense) on standard image datasets. (Our

 D^NR\widehat{\mathcal{D}}\_{NR}D

NR dataset is a perfect example of this.)

### *Takeaway #3*: Cannot fully attribute adversarial

 examples to X

We also show that we cannot

 conclusively fully attribute adversarial examples to any specific aspect of the

 standard training framework (BatchNorm, ResNets, SGD, etc.). In particular, our

 “robust dataset” D^R\widehat{\mathcal{D}}\_{R}D

R is a counterexample to any claim of the form “given any

 dataset, training with BatchNorm/SGD/ResNets/overparameterization/etc. leads to

 adversarial vulnerability” (as classifiers with all of these components,

 when trained on D^R\widehat{\mathcal{D}}\_{R}D

R, generalize robustly to

 CIFAR-10). In that sense, the dataset clearly plays a role in

 the emergence of adversarial examples. (Also, further corroborating this is

 Preetum’s “adversarial squares” dataset [here](#PreetumResponse),

 label noise or overfitting.)

A Few Clarifications

--------------------

In addition to further refining our understanding of adversarial examples,

 the comments were also very useful in pointing out which aspects of our

 claims could benefit from further clarification. To this end, we make these

 clarifications below in the form of a couple “non-claims” — claims that we did

 *not* intend to make. We’ll also update our paper in order to make

 these clarifications explicit.

### Non-Claim #1: “Adversarial examples *cannot* be bugs”

 Our goal is to say that since adversarial examples can arise from

 well-generalizing features, simply patching up the “bugs” in ML models will

 not get rid of adversarial vulnerability — we also need to make sure our

 models learn the right features. This, however, does not mean that

 adversarial vulnerability *cannot* arise from “bugs”. In fact, note

 that several papers

 have proven that adversarial vulnerability can

 arise from what we refer to as “bugs,” e.g. finite-sample overfitting,

 concentration of measure, high dimensionality, etc. Furthermore,

 We would like to thank Preetum for pointing out that this issue may be a

 natural misunderstanding, and for exploring this point in even more depth

 in his response below.

### Non-Claim #2: “Adversarial examples are purely a result of the dataset”

 Even though we [demonstrated](#cannotpin) that datasets do

 play a role in the emergence of adversarial examples, we do not intend to

 claim that this role is exclusive. In particular, just because the data

  *admits*  non-robust functions that are well-generalizing (useful

 non-robust features), doesn’t mean that  *any*  model will learn to

 pick up these features. For example, it could be that the well-generalizing

 features that cause adversarial examples are only learnable by certain

 architectures. However, we do show that there is a way, via only

 altering the dataset, to induce robust models — thus, our results indicate

 that adversarial vulnerability indeed cannot be completely disentangled

 from the dataset (more on this in [Takeaway #3](#cannotpin)).

 summaries.

Responses to comments

---------------------

### Adversarial Example Researchers Need to Expand What is Meant by

 “Robustness” (Dan Hendrycks, Justin Gilmer)

**Response Summary**:

 appears “meaningless” to humans.

 to rely on.

**Response**: The fact that models can learn to classify correctly based

 purely on the high-frequency component of the training set is neat! This nicely

 complements one of our [takeaways](#takeaway1): models will rely on

 useful features even if these features appear incomprehensible to humans.

 Also, while non-robustness to noise can be an indicator of models using

 More often than not, the brittleness of ML models to noise was instead regarded

 expected that progress towards “better”/”bug-free” models will lead to them

 being more robust to noise and adversarial examples.

 small subset of the perturbations we want our models to be robust to. Note,

 however, that the focus of our work is human-alignment — to that end, we

 demonstrate that models rely on features sensitive to patterns that are

 imperceptible to humans. Thus, the existence of other families of

 incomprehensible but useful features would provide even more support for our

 future research.

### Robust Feature Leakage (Gabriel Goh)

**Response Summary**:

 the motivations for designing the D^det\widehat{\mathcal{D}}\_{det}D

det dataset.

**Response**: This comment raises a valid concern which was in fact one of

 the primary reasons for designing the D^det\widehat{\mathcal{D}}\_{det}D

det dataset.

rand

 dataset: assign each input a random target label and do PGD towards that label.

 Note that unlike the D^det\widehat{\mathcal{D}}\_{det}D

det dataset (in which the

rand

 dataset allows for robust features to actually have a (small) positive

 correlation with the label.

To see how this can happen, consider the following simple setting: we have a

 (as in the dataset D^rand\widehat{\mathcal{D}}\_{rand}D

rand) would make this feature

 targeted attack might in this case induce some correlation with the

 to correctly classify new inputs.

In other words, starting from a dataset with no features, one can encode

 robust features within small perturbations. In contrast, in the

 D^det\widehat{\mathcal{D}}\_{det}D

det dataset, the robust features are *correlated

 with the original label* (since the labels are permuted) and since they are

 robust, they cannot be flipped to correlate with the newly assigned (wrong)

 label. Still, the D^rand\widehat{\mathcal{D}}\_{rand}D

rand dataset enables us to show

 that (a) PGD-based adversarial examples actually alter features in the data and

 (b) models can learn from human-meaningless/mislabeled training data. The

 D^det\widehat{\mathcal{D}}\_{det}D

det dataset, on the other hand, illustrates that the

 non-robust features are actually sufficient for generalization and can be

 preferred over robust ones in natural settings.

The experiment put forth in the comment is a clever way of showing that such

 leakage is indeed possible. However, we want to stress (as the comment itself

 does) that robust feature leakage does *not* have an impact on our main

 thesis — the D^det\widehat{\mathcal{D}}\_{det}D

det dataset explicitly controls

 for robust

 feature leakage (and in fact, allows us to quantify the models’ preference for

 robust features vs non-robust features — see Appendix D.6 in the

 [paper](https://arxiv.org/abs/1905.02175)).

### Two Examples of Useful, Non-Robust Features (Gabriel Goh)

 interesting direction of developing a more fine-grained definition of features.

**Response**: These experiments (visualizing the robustness and

 corroborate the existence of useful, non-robust features and make progress

 towards visualizing what these non-robust features actually look like.

We also appreciate the point made by the provided construction of non-robust

 features (as defined in our theoretical framework) that are combinations of

 useful+robust and useless+non-robust features. Our theoretical framework indeed

 enables such a scenario, even if — as the commenter already notes — our

 takeaway](#takeaway1)  are actually stronger than our theoretical

 framework technically captures.) Specifically, in such a scenario, during the

 construction of the D^det\widehat{\mathcal{D}}\_{det}D

det dataset, only the non-robust

 and useless term of the feature would be flipped. Thus, a classifier trained on

 such a dataset would associate the predictive robust feature with the

 *wrong* label and would thus not generalize on the test set. In contrast,

det

 do generalize.

Overall, our focus while developing our theoretical framework was on

 the comment points out, putting forth a theoretical framework that captures

 non-robust features in a very precise way is an important future research

 direction in itself.

### Adversarially Robust Neural Style Transfer

 (Reiichiro Nakano)

**Response Summary**:

 trained models will have in neural network art!

 interesting links between robustness and style transfer.

**Response**: These experiments are really cool! It is interesting that

 preventing the reliance of a model on non-robust features improves performance

 on style transfer, even without an explicit task-related objective (i.e. we

 didn’t train the networks to be better for style transfer).

 We also found the discussion of VGG as a “mysterious network” really

 performance more generally. Though not a complete answer, we made a couple of

 observations while investigating further:

*Style transfer does work with AlexNet:* One wrinkle in the idea that

 the most naturally robust network — AlexNet is. However, based on our own

 testing, style transfer does seem to work with AlexNet out-of-the-box, as

 long as we use a few early layers in the network (in a similar manner to

 VGG):

Style transfer using AlexNet, using

 conv\_1 through conv\_4.

 Observe that even though style transfer still works, there are checkerboard

 patterns emerging — this seems to be a similar phenomenon to the one noticed

 in the comment in the context of robust models.

 This might be another indication that these two phenomena (checkerboard

 patterns and style transfer working) are not as intertwined as previously

 thought.

*From prediction robustness to layer robustness:*  Another

 potential wrinkle here is that both AlexNet and VGG are not that

 much more robust than ResNets (for which style transfer completely fails),

 and yet seem to have dramatically better performance. To try to

 explain this, recall that style transfer is implemented as a minimization of a

 combined objective consisting of a style loss and a content loss. We found,

 however, that the network we use to compute the

 style loss is far more important

 than the one for the content loss. The following demo illustrates this — we can

 actually use a non-robust ResNet for the content loss and everything works just

 fine:

Style transfer seems to be rather

 invariant to the choice of content network used, and very sensitive

 to the style network used.

Therefore, from now on, we use a fixed ResNet-50 for the content loss as a

 control, and only worry about the style loss.

Now, note that the way that style loss works is by using the first few

 layers of the relevant network. Thus, perhaps it is not about the robustness of

 for style transfer?

 To test this hypothesis, we measure the robustness of a layer fff as:

R(f)=Ex1∼D[maxx′∥f(x′)−f(x1)∥2]Ex1,x2∼D[∥f(x1)−f(x2)∥2]

 {\mathbb{E}\_{x\_1, x\_2 \sim D}\left[\|f(x\_1) - f(x\_2)\|\_2\right]}

 R(f)=Ex1,x2∼D[∥f(x1)−f(x2)∥2]Ex1∼D[maxx′∥f(x′)−f(x1)∥2]

 Essentially, this quantity tells us how much we can change the

 representations are between images in general. We’ve plotted this value for

 the first few layers in a couple of different networks below:

The robustness R(f)R(f)R(f) of the first

 four layers of VGG16, AlexNet, and robust/standard ResNet-50

 trained on ImageNet.

 Here, it becomes clear that, the first few layers of VGG and AlexNet are

 actually almost as robust as the first few layers of the robust ResNet!

 This is perhaps a more convincing indication that robustness might have

 something to with VGG’s success in style transfer after all.

 Finally, suppose we restrict style transfer to only use a single layer of

 the network when computing the style lossUsually

 style transfer uses

 several layers in the loss function to get the most

visually appealing results — here we’re only interested in whether or

not style transfer works (i.e.

 actually confers some style onto the image).. Again, the more

 robust layers seem to indeed work better for style transfer! Since all of the

 layers in the robust ResNet are robust, style transfer yields non-trivial

 results even using the last layer alone. Conversely, VGG and AlexNet seem to

 excel in the earlier layers (where they are non-trivially robust) but fail when

 using exclusively later (non-robust) layers:

Style transfer using a single layer. The

 names of the layers and their robustness R(f)R(f)R(f) are printed below

 each style transfer result. We find that for both networks, the robust

 layers seem to work (for the robust ResNet, every layer is robust).

 Of course, there is much more work to be done here, but we are excited

 to see further work into understanding the role of both robustness and the VGG

 in network-based image manipulation.

###  Adversarial Examples are Just Bugs, Too (Preetum

 Nakkiran)

**Response Summary**:

 providing an

 example of adversarial examples that arise from “bugs”.

**Response**: As mentioned [above](#nonclaim1),

 we did not intend to claim

 that adversarial examples arise *exclusively* from (useful) features but rather

 that useful non-robust features exist and are thus (at least

 partially) responsible for adversarial vulnerability. In fact,

 prior work already shows how in theory adversarial examples can arise from

 insufficient samples  or finite-sample overfitting , and the experiments

 presented here (particularly, the adversarial squares) constitute a neat

 real-world demonstration of these facts.

 Our main thesis that “adversarial examples will not just go away as we fix

 stemming from “bugs.” As long as adversarial examples can stem from non-robust

 features (which the commenter seems to agree with), fixing these bugs will not

 solve the problem of adversarial examples.

Moreover, with regards to feature “leakage” from PGD, recall that in

 or D\_det dataset, the non-robust features are associated with the

 correct label whereas the robust features are associated with the wrong

 one. We wanted to emphasize that, as shown in [Appendix D.6](LINK),

 models trained on our D\_det dataset actually generalize *better* to

 the non-robust feature-label association that to the robust

 feature-label association. In contrast, if PGD introduced a small

 “leakage” of non-robust features, then we would expect the trained model

 would still predominantly use the robust feature-label association.

 That said, the experiments cleverly zoom in on some more fine-grained

 nuances in our understanding of adversarial examples. One particular thing that

 stood out to us is that by creating a set of adversarial examples that are

 *explicitly* non-transferable, one also prevents new classifiers from learning

 features from that dataset. This finding thus makes the connection between

 transferability of adversarial examples and their containing generalizing

 features even stronger! Indeed, we can add the constructed dataset into our

 “D^det\widehat{\mathcal{D}}\_{det}D

det learnability vs transferability” plot

 (Figure 3 in the paper) — the point

 corresponding to this dataset fits neatly onto the trendline!

 attacks

### Learning from Incorrectly Labeled Data (Eric Wallace)

**Response Summary**:

 settings.

**Response**: Since our experiments work across different architectures,

 “distillation” in weight space cannot arise. Thus, from what we understand, the

 “distillation” hypothesis suggested here is referring to “feature distillation”

 (i.e. getting models which use the same features as the original), which is

 actually precisely our hypothesis too. Notably, this feature distillation would

 are good for classification (see [World 1](#world1) and

 [World 2](#world2)) — in that case, the distilled

 model would only use features that generalize poorly, and would thus generalize

 poorly itself.

 Moreover, we would argue that in the experiments presented (learning from

 mislabeled data), the same kind of distillation is happening. For instance, a

 moderately accurate model might associate “green background” with “frog” thus

 labeling “green” images as “frogs” (e.g., the horse in the comment’s figure).

 Training a new model on this dataset will thus associate “green” with “frog”

 from Fashion-MNIST” experiment in the comment). This corresponds exactly to

 learning features from labels, akin to how deep networks “distill” a good

 decision boundary from human annotators. In fact, we find these experiments

 a very interesting illustration of feature distillation that complements

 our findings.

 We also note that an analogy to logistic regression here is only possible

 due to the low VC-dimension of linear classifiers (namely, these classifiers

 have dimension ddd). In particular, given any classifier with VC-dimension

 networks have been shown to have extremely large VC-dimension (in particular,

 bigger than the size of the training set ). So even though

 labelling

 d+1d+1d+1 random

 points model-consistently is sufficient to recover a linear model, it is not

 necessarily sufficient to recover a deep neural network. For instance, Milli et

 al.  are not able to reconstruct a ResNet-18

 using only its predictions on random Gaussian inputs. (Note that we are using a

 ResNet-50 in our experiments.)

 Finally, it seems that the only potentially problematic explanation for

 our experiments (namely, that enough model-consistent points can recover a

 particular, Preetum is able to design a

 dataset where training on mislabeled inputs *that are model-consistent*

 does not at all recover the decision boundary of the original model. More

 generally, the “model distillation” perspective raised here is unable to

 distinguish between the dataset created by Preetum below, and those created

 with standard PGD (as in our D^det\widehat{\mathcal{D}}\_{det}D

det and

 D^rand\widehat{\mathcal{D}}\_{rand}D

rand datasets).

A boundary tilting persepective on the phenomenon of adversarial examples

Adversarially Robust Generalization Requires More Data

Adversarial spheres

Adversarial vulnerability for any classifier

The curse of concentration in robust learning: Evasion and poisoning attacks from concentration of measure

Are adversarial examples inevitable?

Understanding deep learning requires rethinking generalization

Model reconstruction from model explanations

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features' Discussion and Author Responses.html


Why Momentum Really Works

Why Momentum Really Works

=========================

OptimumSolutionStarting Point

Step-size α = 0.02

00.0030.006

Momentum β = 0.99

0.000.5000.990

 We often think of Momentum as a means of dampening oscillations

and speeding up the iterations, leading to faster convergence. But it

has other interesting behavior. It allows a larger range of step-sizes

to be used, and creates its own oscillations. What is going on?

 dt-byline {

 font-size: 12px;

 line-height: 18px;

 display: block;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 color: rgba(0, 0, 0, 0.5);

 padding-top: 12px;

 padding-bottom: 12px;

 }

 dt-article.centered dt-byline {

 text-align: center;

 }

 dt-byline a,

 dt-article dt-byline a {

 text-decoration: none;

 border-bottom: none;

 }

 dt-article dt-byline a:hover {

 text-decoration: underline;

 border-bottom: none;

 }

 dt-byline .authors {

 text-align: left;

 }

 dt-byline .name {

 display: inline;

 text-transform: uppercase;

 }

 dt-byline .affiliation {

 display: inline;

 }

 dt-byline .date {

 display: block;

 text-align: left;

 }

 dt-byline .year, dt-byline .month {

 display: inline;

 }

 dt-byline .citation {

 display: block;

 text-align: left;

 }

 dt-byline .citation div {

 display: inline;

 }

 @media(min-width: 768px) {

 dt-byline {

 }

 }

 @media(min-width: 1080px) {

 dt-byline {

 border-bottom: none;

 margin-bottom: 70px;

 }

 dt-byline a:hover {

 color: rgba(0, 0, 0, 0.9);

 }

 dt-byline .authors {

 display: inline-block;

 }

 dt-byline .author {

 display: inline-block;

 margin-right: 12px;

 /\*padding-left: 20px;\*/

 /\*border-left: 1px solid #ddd;\*/

 }

 dt-byline .affiliation {

 display: block;

 }

 dt-byline .author:last-child {

 margin-right: 0;

 }

 dt-byline .name {

 display: block;

 }

 dt-byline .date {

 border-left: 1px solid rgba(0, 0, 0, 0.1);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .year, dt-byline .month {

 display: block;

 }

 dt-byline .citation {

 border-left: 1px solid rgba(0, 0, 0, 0.15);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .citation div {

 display: block;

 }

 }

[Gabriel Goh](http://gabgoh.github.io/)

[UC Davis](http://math.ucdavis.edu/)

April. 4

2017

[Citation:

Goh, 2017](#citation)

 // Render Foreground

 .alpha(0.003)

 .beta(0)

 (d3.select("#banana").style("position","relative"))

 var iterChange = iterControl.control

 var getw0 = iterControl.w0

 var StepRange = d3.scaleLinear().domain([0,100]).range([0,0.0062])

 var MomentumRange = d3.scaleLinear().domain([0,100]).range([0,0.98])

 var update = function (i,j) { iterChange(i, 0, getw0()) }

 var slidera = sliderGen([230, 40])

 .ticks([0,0.003,0.006])

 .ticktitles( function(d,i) { return ["0", "0.003", "0.006"][i]})

 .change( function (i) {

 iterChange(getalpha(), getbeta(), getw0() )

 } )

 .startxval(0.003)

 .cRadius(7)

 .shifty(-12)

 .margins(20,20)

 var sliderb = sliderGen([230, 40])

 .ticks([0,0.5,0.99])

 .change( function (i) {

 iterChange(getalpha(), getbeta(), getw0() )

 } )

 .cRadius(7)

 .shifty(-12)

 .startxval(0.74)

 .margins(20,20)

 var getalpha = slidera( d3.select("#sliderAlpha")).xval

 var getbeta = sliderb( d3.select("#sliderBeta")).xval

 iterChange(getalpha(), getbeta(), getw0() )

 Here’s a popular story about momentum [1, 2, 3]:

 gradient descent is a man walking down a hill. He follows the steepest

path downwards; his progress is slow, but steady. Momentum is a heavy

ball rolling down the same hill. The added inertia acts both as a

smoother and an accelerator, dampening oscillations and causing us to

barrel through narrow valleys, small humps and local minima.

 This standard story isn’t wrong, but it fails to explain many

important behaviors of momentum. In fact, momentum can be understood far

 more precisely if we study it on the right model.

 One nice model is the convex quadratic. This model is rich enough to

reproduce momentum’s local dynamics in real problems, and yet simple

enough to be understood in closed form. This balance gives us powerful

traction for understanding this algorithm.

---

 We begin with gradient descent. The algorithm has many virtues, but

speed is not one of them. It is simple — when optimizing a smooth

function

dt-math[block] {

 display: block;

}

fff, we make a small step in the gradient

dt-math[block] {

 display: block;

}

wk+1=wk−α∇f(wk).w^{k+1} = w^k-\alpha\nabla f(w^k).wk+1=wk−α∇f(wk).

 For a step-size small enough, gradient descent makes a monotonic

improvement at every iteration. It always converges, albeit to a local

minimum. And under a few weak curvature conditions it can even get there

 at an exponential rate.

 But the exponential decrease, though appealing in theory, can often be

 infuriatingly small. Things often begin quite well — with an

impressive, almost immediate decrease in the loss. But as the iterations

 progress, things start to slow down. You start to get a nagging feeling

 you’re not making as much progress as you should be. What has gone

wrong?

dt-math[block] {

 display: block;

}

fff

 which aren’t scaled properly. The landscapes are often described as

valleys, trenches, canals and ravines. The iterates either jump between

valleys, or approach the optimum in small, timid steps. Progress along

certain directions grind to a halt. In these unfortunate regions,

gradient descent fumbles.

dt-math[block] {

 display: block;

}

zk+1=βzk+∇f(wk)wk+1=wk−αzk+1

 \begin{aligned}

 z^{k+1}&=\beta z^{k}+\nabla f(w^{k})\\[0.4em]

 w^{k+1}&=w^{k}-\alpha z^{k+1}

 \end{aligned}

 zk+1wk+1=βzk+∇f(wk)=wk−αzk+1

 The change is innocent, and costs almost nothing. When

dt-math[block] {

 display: block;

}

β=0\beta = 0β=0 , we recover gradient descent. But for

dt-math[block] {

 display: block;

}

β=0.99\beta = 0.99β=0.99 (sometimes

dt-math[block] {

 display: block;

}

0.9990.9990.999,

 if things are really bad), this appears to be the boost we need. Our

iterations regain that speed and boldness it lost, speeding to the

optimum with a renewed energy.

 Optimizers call this minor miracle “acceleration”.

 The new algorithm may seem at first glance like a cheap hack. A simple

 trick to get around gradient descent’s more aberrant behavior — a

smoother for oscillations between steep canyons. But the truth, if

anything, is the other way round. It is gradient descent which is the

hack. First, momentum gives up to a quadratic speedup on many functions.

 1

 This is no small matter — this is similar to the speedup you get from

the Fast Fourier Transform, Quicksort, and Grover’s Algorithm. When the

universe gives you quadratic speedups, you should start to pay

attention.

 But there’s more. A lower bound, courtesy of Nesterov [5],

 states that momentum is, in a certain very narrow and technical sense,

optimal. Now, this doesn’t mean it is the best algorithm for all

functions in all circumstances. But it does satisfy some curiously

beautiful mathematical properties which scratch a very human itch for

perfection and closure. But more on that later. Let’s say this for

now — momentum is an algorithm for the book.

---

First Steps: Gradient Descent

-----------------------------

dt-math[block] {

 display: block;

}

f(w)=12wTAw−bTw,w∈Rn.

 f(w) = \tfrac{1}{2}w^TAw - b^Tw, \qquad w \in \mathbf{R}^n.

 f(w)=21wTAw−bTw,w∈Rn.

 Assume

dt-math[block] {

 display: block;

}

AAA is symmetric and invertible, then the optimal solution

dt-math[block] {

 display: block;

}

w⋆w^{\star}w⋆ occurs at

dt-math[block] {

 display: block;

}

w⋆=A−1b. w^{\star} = A^{-1}b.w⋆=A−1b.

dt-math[block] {

 display: block;

}

 etc) and captures all the key features of pathological curvature. And

more importantly, we can write an exact closed formula for gradient

descent on this function.

 This is how it goes. Since

dt-math[block] {

 display: block;

}

∇f(w)=Aw−b\nabla f(w)=Aw - b∇f(w)=Aw−b, the iterates are

dt-math[block] {

 display: block;

}

wk+1=wk−α(Awk−b).

 w^{k+1}=w^{k}- \alpha (Aw^{k} - b).

 wk+1=wk−α(Awk−b).

 Here’s the trick. There is a very natural space to view gradient

descent where all the dimensions act independently — the eigenvectors of

dt-math[block] {

 display: block;

}

AAA.

Optimum

Optimum

 deleteQueue.push(renderLoading(d3.select("#change\_of\_variables")))

 renderQueue.push(function(callback) {

 var U = givens(Math.PI/4)

 var Ut = numeric.transpose(U)

 // Render Foreground

 var left = d3.select("#mom1").style("border", "1px solid rgba(0, 0, 0, 0.2)")

 var c1 = genIterDiagram(quadf, [0,0], [[-3,3],[-3,3]])

 .width(340)

 .height(340)

 .iters(300)

 .alpha(0.018)

 .showSolution(false)

 .pathWidth(1)

 .circleRadius(1.5)

 .pointerScale(0.8)

 .showStartingPoint(false)

 .drag(function() {

 c2.control(c1.alpha(),

 c1.beta(),

 numeric.dot(U,c1.w0())) })

 (left)

 var right = d3.select("#mom2").style("border", "1px solid rgba(0, 0, 0, 0.2)")

 var c2 = genIterDiagram(eyef, [0,0], [[-3,3],[-3,3]])

 .width(340)

 .height(340)

 .iters(300)

 .alpha(0.018)

 .showSolution(false)

 .pathWidth(1)

 .circleRadius(1.5)

 .pointerScale(0.8)

 .showStartingPoint(false)

 .drag(function() {

 c1.control(c2.alpha(),

 c2.beta(),

 numeric.dot(Ut,c2.w0())) })

 (right)

 // Initialize

 c2.control(0.018,0,[-2.5,1])

 c1.control(0.018,0,numeric.dot(Ut,[-2.5,1]));

 callback(null);

});

 Every symmetric matrix

dt-math[block] {

 display: block;

}

AAA has an eigenvalue decomposition

dt-math[block] {

 display: block;

}

A=Q diag(λ1,…,λn) QT,Q=[q1,…,qn],

 A=Q diag(λ1,…,λn) QT,Q=[q1,…,qn],

 and, as per convention, we will assume that the

dt-math[block] {

 display: block;

}

λi\lambda\_iλi’s are sorted, from smallest

dt-math[block] {

 display: block;

}

λ1\lambda\_1λ1 to biggest

dt-math[block] {

 display: block;

}

λn\lambda\_nλn. If we perform a change of basis,

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xik+1=xik−αλixik=(1−αλi)xik=(1−αλi)k+1xi0

 \begin{aligned}

 x\_{i}^{k+1} & =x\_{i}^{k}-\alpha \lambda\_ix\_{i}^{k} \\[0.4em]

 &= (1-\alpha\lambda\_i)x^k\_i=(1-\alpha \lambda\_i)^{k+1}x^0\_i

 \end{aligned}

 Moving back to our original space

dt-math[block] {

 display: block;

}

www, we can see that

dt-math[block] {

 display: block;

}

wk−w⋆=Qxk=∑inxi0(1−αλi)kqi

 w^k - w^\star = Qx^k=\sum\_i^n x^0\_i(1-\alpha\lambda\_i)^k q\_i

 wk−w⋆=Qxk=i∑nxi0(1−αλi)kqi

 and there we have it — gradient descent in closed form.

### Decomposing the Error

 The above equation admits a simple interpretation. Each element of

dt-math[block] {

 display: block;

}

x0x^0x0 is the component of the error in the initial guess in the

dt-math[block] {

 display: block;

}

QQQ-basis. There are

dt-math[block] {

 display: block;

}

nnn

 such errors, and each of these errors follows its own, solitary path to

 the minimum, decreasing exponentially with a compounding rate of

dt-math[block] {

 display: block;

}

1−αλi1-\alpha\lambda\_i1−αλi. The closer that number is to

dt-math[block] {

 display: block;

}

111, the slower it converges.

 For most step-sizes, the eigenvectors with largest eigenvalues

converge the fastest. This triggers an explosion of progress in the

first few iterations, before things slow down as the smaller

eigenvectors’ struggles are revealed. By writing the contributions of

each eigenspace’s error to the loss

dt-math[block] {

 display: block;

}

f(wk)−f(w⋆)=∑(1−αλi)2kλi[xi0]2

 f(w^{k})-f(w^{\star})=\sum(1-\alpha\lambda\_{i})^{2k}\lambda\_{i}[x\_{i}^{0}]^2

 f(wk)−f(w⋆)=∑(1−αλi)2kλi[xi0]2

 we can visualize the contributions of each error component to the loss.

dt-math[block] {

 display: block;

}

λ1=0.01\lambda\_1=0.01λ1=0.01,

dt-math[block] {

 display: block;

}

λ2=0.1\lambda\_2=0.1λ2=0.1, and

dt-math[block] {

 display: block;

}

λ3=1\lambda\_3=1λ3=1 respectively.

 Step-size

 Optimal Step-size

012

 deleteQueue.push(renderLoading(d3.select("#milestones\_gd")))

 renderQueue.push(function(callback) {

 var graphDiv = d3.select("#obj")

 .style("width", 920 + "px")

 .style("height", 300 + "px")

 .style("top", "90px")

 .style("position", "relative")

 .style("margin-left", "auto")

 .style("margin-right", "auto")

 .attr("width", 920)

 .attr("height", 500)

 var svg = graphDiv.append("svg")

 .attr("width", 920)

 .attr("height", 300)

 .style("position","absolute")

 .style("left", "15px")

 var updateSliderGD = renderMilestones(svg, function() {});

 var alphaHTML = MathCache("alpha-equals");

 var slidera = sliderGen([250, 80])

 .ticks([0,1,200/(101),2])

 .change( function (i) {

 d3.select("#stepSizeMilestones")

 .html("Stepsize " + html )

 updateSliderGD(i,0.000)

 } )

 .ticktitles(function(d,i) { return [0,1,"",2][i] })

 .startxval(200/(101))

 .cRadius(7)

 .shifty(-12)

 .shifty(10)

 .margins(20,20)(d3.select("#sliderStep"))

 // renderDraggable(svg, [133.5, 23], [114.5, 90], 2, " ").attr("opacity", 0.1)

 // renderDraggable(svg, [133.5, 88], [115.5, 95], 2, " ").attr("opacity", 0.1)

 d3.select("#milestones\_gd\_optstep").on("click", slidera.init)

 svg.append("text")

 .attr("class", "katex morsd mathit")

 .style("font-size", "19px")

 .style("font-family","KaTeX\_Math")

 .attr("x", 105)

 .attr("y", 50)

 .attr("text-anchor", "end")

 .attr("fill", "gray")

 svg.append("text")

 .style("font-size", "13px")

 .attr("x", 0)

 .attr("y", 80)

 .attr("dy", 0)

 .attr("transform", "translate(110,0)")

 .attr("class", "caption")

 .attr("text-anchor", "end")

 .attr("fill", "gray")

 .text("At the initial point, the error in each component is equal.")

 svg.selectAll(".caption").call(wrap, 100)

 svg.append("text")

 .style("font-size", "13px")

 .attr("x", 420)

 .attr("y", 270)

 .attr("dy", 0)

 .attr("dx", -295)

 .attr("text-anchor", "start")

 .attr("fill", "gray")

 callback(null);

 });

### Choosing A Step-size

 The above analysis gives us immediate guidance as to how to set a step-size

dt-math[block] {

 display: block;

}

α\alphaα. In order to converge, each

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

0<αλi<2.0<\alpha\lambda\_i<2.0<αλi<2.

dt-math[block] {

 display: block;

}

λ1\lambda\_1λ1 or

dt-math[block] {

 display: block;

}

λn\lambda\_nλn:

dt-math[block] {

 display: block;

}

rate(α) = maxi∣1−αλi∣ = max{∣1−αλ1∣, ∣1−αλn∣}

 \begin{aligned}\text{rate}(\alpha) & ~=~

\max\_{i}\left|1-\alpha\lambda\_{i}\right|\\[0.9em] & ~=~

\max\left\{|1-\alpha\lambda\_{1}|,~ |1-\alpha\lambda\_{n}|\right\}

\end{aligned}

 rate(α) = imax∣1−αλi∣ = max{∣1−αλ1∣, ∣1−αλn∣}

 This overall rate is minimized when the rates for

dt-math[block] {

 display: block;

}

λ1\lambda\_1λ1 and

dt-math[block] {

 display: block;

}

λn\lambda\_nλn

 are the same — this mirrors our informal observation in the previous

section that the optimal step-size causes the first and last

eigenvectors to converge at the same rate. If we work this through we

get:

dt-math[block] {

 display: block;

}

optimal α = argminα rate(α) = 2λ1+λnoptimal rate = minα rate(α) = λn/λ1−1λn/λ1+1

 \begin{aligned}

 \text{optimal }\alpha ~=~{\mathop{\text{argmin}}\limits\_\alpha}

~\text{rate}(\alpha) & ~=~\frac{2}{\lambda\_{1}+\lambda\_{n}}\\[1.4em]

 \text{optimal rate} ~=~{\min\_\alpha} ~\text{rate}(\alpha) &

~=~\frac{\lambda\_{n}/\lambda\_{1}-1}{\lambda\_{n}/\lambda\_{1}+1}

 \end{aligned}

 Notice the ratio

dt-math[block] {

 display: block;

}

λn/λ1\lambda\_n/\lambda\_1λn/λ1

 determines the convergence rate of the problem. In fact, this ratio

appears often enough that we give it a name, and a symbol — the

condition number.

dt-math[block] {

 display: block;

}

condition number:=κ:=λnλ1

 \text{condition number} := \kappa :=\frac{\lambda\_n}{\lambda\_1}

 condition number:=κ:=λ1λn

dt-math[block] {

 display: block;

}

A−1bA^{-1}bA−1b is to perturbations in

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

κ=1\kappa = 1κ=1

 is ideal, giving convergence in one step (of course, the function is

trivial). Unfortunately the larger the ratio, the slower gradient

descent will be. The condition number is therefore a direct measure of

pathological curvature.

---

Example: Polynomial Regression

------------------------------

dt-math[block] {

 display: block;

}

nnn to be exact, one for each of the eigenvectors of

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

 Lets see how this plays out in polynomial regression. Given 1D data,

dt-math[block] {

 display: block;

}

ξi\xi\_iξi, our problem is to fit the model

dt-math[block] {

 display: block;

}

model(ξ)=w1p1(ξ)+⋯+wnpn(ξ)pi=ξ↦ξi−1

 model(ξ)=w1p1(ξ)+⋯+wnpn(ξ)pi=ξ↦ξi−1

 to our observations,

dt-math[block] {

 display: block;

}

did\_idi. This model, though nonlinear in the input

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

0p10 p\_1-2.00p1

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_1-2.00p2

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_12.00p3

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_12.00p4

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_12.00p5

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_1-2.00p6

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

model\text{model}model0+0+0+0+0+0=0

scrub values

 deleteQueue.push(renderLoading(d3.select("#poly0")))

 renderQueue.push(function(callback) {

 // Preprocess x, get eigendecomposition, etc

 var x = [-0.6, -0.55,-0.5,-0.45,-0.4,0.4,0.45,0.5,0.55,0.6]

 var D = vandermonde(x, 5)

 var Eigs = eigSym(numeric.dot(numeric.transpose(D),D))

 var U = Eigs.U

 var lambda = Eigs.lambda

 // Preprocess y

 var b = [-3/2,-4/2,-5/2,-3/2,-2/2,1/2,2/2,3/2,2/2,1/2]

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var step = 1.8/lambda[0]

 var iter = geniter(U, lambda, Dtb, step)

 var eigensum = d3.select("#poly0")

 var wi = [-2,-2,2,2,2,-2]

 function refit(b) {

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var Utsol = numeric.dot(sol,U)

 eigenControl.updateweights(Utsol)

 }

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 60,5 A 19.018 19.018 0 0 0 36,27",

 "text": "scrub values",

 "textOffset": [

 64,

 9

 ]

 }

 ]

 drawAnnotations(d3.select("#poly0f"), annotations)

 callback(null);

 });

 Because of the linearity, we can fit this model to our data

dt-math[block] {

 display: block;

}

ξi\xi\_iξi using linear regression on the model mismatch

dt-math[block] {

 display: block;

}

minimizew12∑i(model(ξi)−di)2  =  12∥Zw−d∥2

 minimizew21i∑(model(ξi)−di)2  =  21∥Zw−d∥2

 where

dt-math[block] {

 display: block;

}

Z=(1ξ1ξ12…ξ1n−11ξ2ξ22…ξ2n−1⋮⋮⋮⋱⋮1ξmξm2…ξmn−1).

 Z=\left(\begin{array}{ccccc}

 1 & \xi\_{1} & \xi\_{1}^{2} & \ldots & \xi\_{1}^{n-1}\\

 1 & \xi\_{2} & \xi\_{2}^{2} & \ldots & \xi\_{2}^{n-1}\\

 \vdots & \vdots & \vdots & \ddots & \vdots\\

 1 & \xi\_{m} & \xi\_{m}^{2} & \ldots & \xi\_{m}^{n-1}

 \end{array}\right).

dt-math[block] {

 display: block;

}

QQQ (the eigenvectors of

dt-math[block] {

 display: block;

}

ZTZZ^T ZZTZ). So let’s recast our regression problem in the basis of

dt-math[block] {

 display: block;

}

QQQ. First, we do a change of basis, by rotating

dt-math[block] {

 display: block;

}

www into

dt-math[block] {

 display: block;

}

QwQwQw, and counter-rotating our feature maps

dt-math[block] {

 display: block;

}

ppp into eigenspace,

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

model(ξ) = x1p¯1(ξ) + ⋯ + xnp¯n(ξ)p¯i=∑qijpj.

 model(ξ) = x1p¯1(ξ) + ⋯ + xnp¯n(ξ)p¯i=∑qijpj.

 This model is identical to the old one. But these new features

dt-math[block] {

 display: block;

}

p¯\bar{p}p¯

 (which I call “eigenfeatures”) and weights have the pleasing property

that each coordinate acts independently of the others. Now our

optimization problem breaks down, really, into

dt-math[block] {

 display: block;

}

nnn

 small 1D optimization problems. And each coordinate can be optimized

greedily and independently, one at a time in any order, to produce the

final, global, optimum. The eigenfeatures are also much more

informative:

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_10.458p¯1

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_10.903p¯2

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_16.13p¯3

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_112.4p¯4

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1128p¯5

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1261p¯6

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

 deleteQueue.push(renderLoading(d3.select("#poly1")))

 renderQueue.push(function(callback) {

 var inv = function(lambda) { return 1/lambda }

 // Preprocess x, get eigendecomposition, etc

 var x = [-0.6, -0.55,-0.5,-0.45,-0.4,0.4,0.45,0.5,0.55,0.6]

 var D = vandermonde(x, 5)

 var Eigs = eigSym(numeric.dot(numeric.transpose(D),D))

 var U = Eigs.U

 var lambda = Eigs.lambda

 // Preprocess y

 var b = [-3/2,-4/2,-5/2,-3/2,-2/2,1/2,2/2,3/2,2/2,1/2]

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var step = 1.8/lambda[0]

 var iter = geniter(U, lambda, Dtb, step)

 var eigensum = d3.select("#poly1")

 var wi = lambda.slice(0).map(scal)

 function refit(b) {

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var Utsol = numeric.dot(sol,U)

 eigenControl.updateweights(sol)

 }

 var eigenControl = renderEigenPanel(eigensum, U, x, b, wi, refit, true)

 var annotate = eigensum

 annotate.append("figcaption")

 .style("width", 230 + "px")

 .style("height", 150 + "px")

 .style("left", "0px")

 .style("position", "absolute")

 .style("padding", "10px")

 annotate.append("figcaption")

 .style("width", 230 + "px")

 .style("height", 150 + "px")

 .style("left", "260px")

 .style("position", "absolute")

 .style("padding", "10px")

 annotate.append("figcaption")

 .style("width", 230 + "px")

 .style("height", 150 + "px")

 .style("left", 530 + "px")

 .style("position", "absolute")

 .style("padding", "10px")

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 807,198 A 26.661 26.661 0 0 1 838,159",

 "text": "drag points to fit data",

 "textOffset": [

 799,

 214

 ]

 }]

 drawAnnotations(eigensum, annotations)

 callback(null);

 });

 The observations in the above diagram can be justified mathematically.

 From a statistical point of view, we would like a model which is, in

some sense, robust to noise. Our model cannot possibly be meaningful if

the slightest perturbation to the observations changes the entire model

dramatically. And the eigenfeatures, the principal components of the

data, give us exactly the decomposition we need to sort the features by

its sensitivity to perturbations in

dt-math[block] {

 display: block;

}

did\_idi’s.

 The most robust components appear in the front (with the largest

eigenvalues), and the most sensitive components in the back (with the

smallest eigenvalues).

 This measure of robustness, by a rather convenient coincidence, is

also a measure of how easily an eigenspace converges. And thus, the

“pathological directions” — the eigenspaces which converge the

slowest — are also those which are most sensitive to noise! So starting

at a simple initial point like

dt-math[block] {

 display: block;

}

000

 (by a gross abuse of language, let’s think of this as a prior), we

track the iterates till a desired level of complexity is reached. Let’s

see how this plays out in gradient descent.

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-0.389p¯1

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-2.47p¯2

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-0.481p¯3

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-0.540p¯4

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-0.0128p¯5

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_10.0543p¯6

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

 deleteQueue.push(renderLoading(d3.select("#poly2")))

 renderQueue.push(function(callback) {

 var inv = function(lambda) { return 1/lambda }

 // Preprocess x, get eigendecomposition, etc

 var x = [-0.6, -0.55,-0.5,-0.45,-0.4,0.4,0.45,0.5,0.55,0.6]

 var b = [-3/2,-4/2,-5/2,-3/2,-2/2,1/2,2/2,3/2,2/2,1/2]

 var D = vandermonde(x, 5)

 var Eigs = eigSym(numeric.dot(numeric.transpose(D),D))

 var U = Eigs.U

 var lambda = Eigs.lambda

 // Preprocess y

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var step = 1.8/lambda[0]

 var iter = geniter(U, lambda, Dtb, step)

 var eigensum = d3.select("#poly2")

 var wi = lambda.slice(0).map(scal)

 function refit(b) {

 var Dtb = numeric.dot(b,D)

 iter = geniter(U, lambda, Dtb, step)

 onChange(sliderControl.slidera.xval())

 }

 var eigenControl = renderEigenPanel(eigensum, U, x, b, wi, refit, true)

 var barlengths = getStepsConvergence(lambda, step).map(Math.log)

 var onChange = function(i) {

 eigenControl.updateweights(numeric.dot(U,iter(Math.floor(Math.exp(i-0.1)) )))

 }

 d3.select("#poly2").append("figcaption")

 .style("width", "120px")

 .style("position", "absolute")

 .style("left", "820px")

 .style("top","200px")

 sliderControl.slidera.init()

 // var figwidth = d3.select("#poly2").style("width")

 // var figheight = d3.select("#poly2").style("height")

 // var svgannotate = d3.select("#poly2")

 // .append("svg")

 // .style("width", figwidth)

 // .style("height", figheight)

 // .style("position", "absolute")

 // .style("top","0px")

 // .style("left","0px")

 // .style("pointer-events","none")

 // renderDraggable(svgannotate,

 // [139.88888549804688, 243.77951049804688],

 // [121.88888549804688, 200.77951049804688],

 // 5,

 // "We begin at x=w=0");

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 74,202 A 52.274 52.274 0 0 0 134,245",

 "text": "We begin at x=w=0",

 "textOffset": [

 21,

 198

 ]

 }

 ]

 drawAnnotations(d3.select("#poly2"), annotations)

 callback(null);

 });

 This effect is harnessed with the heuristic of early stopping : by

stopping the optimization early, you can often get better generalizing

results. Indeed, the effect of early stopping is very similar to that of

 more conventional methods of regularization, such as Tikhonov

Regression. Both methods try to suppress the components of the smallest

eigenvalues directly, though they employ different methods of spectral

decay.2

 But early stopping has a distinct advantage. Once the step-size is

chosen, there are no regularization parameters to fiddle with. Indeed,

in the course of a single optimization, we have the entire family of

models, from underfitted to overfitted, at our disposal. This gift, it

seems, doesn’t come at a price. A beautiful free lunch [7] indeed.

---

The Dynamics of Momentum

------------------------

 Let’s turn our attention back to momentum. Recall that the momentum update is

dt-math[block] {

 display: block;

}

zk+1=βzk+∇f(wk)wk+1=wk−αzk+1.

 \begin{aligned}

 z^{k+1}&=\beta z^{k}+\nabla f(w^{k})\\[0.4em]

 w^{k+1}&=w^{k}-\alpha z^{k+1}.

 \end{aligned}

 zk+1wk+1=βzk+∇f(wk)=wk−αzk+1.

 Since

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

zk+1=βzk+(Awk−b)wk+1=wk−αzk+1.

 \begin{aligned}

 z^{k+1}&=\beta z^{k}+ (Aw^{k}-b)\\[0.4em]

 w^{k+1}&=w^{k}-\alpha z^{k+1}.

 \end{aligned}

 zk+1wk+1=βzk+(Awk−b)=wk−αzk+1.

 Following [8], we go through the same motions, with the change of basis

dt-math[block] {

 display: block;

}

xk=Q(wk−w⋆)

 x^{k} = Q(w^{k} - w^\star)xk=Q(wk−w⋆) and

dt-math[block] {

 display: block;

}

yk=Qzk y^{k} = Qz^{k}yk=Qzk, to yield the update rule

dt-math[block] {

 display: block;

}

yik+1=βyik+λixikxik+1=xik−αyik+1.

 \begin{aligned}

 y\_{i}^{k+1}&=\beta y\_{i}^{k}+\lambda\_{i}x\_{i}^{k}\\[0.4em]

 x\_{i}^{k+1}&=x\_{i}^{k}-\alpha y\_{i}^{k+1}.

 \end{aligned}

 yik+1xik+1=βyik+λixik=xik−αyik+1.

 in which each component acts independently of the other components (though

dt-math[block] {

 display: block;

}

xikx^k\_ixik and

dt-math[block] {

 display: block;

}

yiky^k\_iyik are coupled). This lets us rewrite our iterates as

 3

dt-math[block] {

 display: block;

}

(yikxik)=Rk(yi0xi0)R=(βλi−αβ1−αλi).

 \left(\!\!\begin{array}{c}

 y\_{i}^{k}\\

 x\_{i}^{k}

 \end{array}\!\!\right)=R^k\left(\!\!\begin{array}{c}

 y\_{i}^{0}\\

 x\_{i}^{0}

 \end{array}\!\!\right)

 \qquad

 R = \left(\!\!\begin{array}{cc}

 \beta & \lambda\_{i}\\

 -\alpha\beta & 1-\alpha\lambda\_{i}

 \end{array}\!\!\right).

 (yikxik)=Rk(yi0xi0)R=(β−αβλi1−αλi).

 There are many ways of taking a matrix to the

dt-math[block] {

 display: block;

}

kthk^{th}kth power. But for the

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

RRR,

dt-math[block] {

 display: block;

}

σ1\sigma\_1σ1 and

dt-math[block] {

 display: block;

}

σ2\sigma\_2σ2.

dt-math[block] {

 display: block;

}

Rk={σ1kR1−σ2kR2σ1≠σ2σ1k(kR/σ1−(k−1)I)σ1=σ2,Rj=R−σjIσ1−σ2

\color{#AAA}{\color{black}{R^{k}}=\begin{cases}

\sigma\_{1}^{k}(kR/\sigma\_1-(k-1)I) & \sigma\_{1}=\sigma\_{2}

\end{cases},\qquad R\_{j}=\frac{R-\sigma\_{j}I}{\sigma\_{1}-\sigma\_{2}}}

 This formula is rather complicated, but the takeaway here is that it

plays the exact same role the individual convergence rates,

dt-math[block] {

 display: block;

}

1−αλi1-\alpha\lambda\_i1−αλi

 do in gradient descent. But instead of one geometric series, we have

two coupled series, which may have real or complex values. The

convergence rate is therefore the slowest of the two rates,

dt-math[block] {

 display: block;

}

max{∣σ1∣,∣σ2∣}\max\{|\sigma\_{1}|,|\sigma\_{2}|\}max{∣σ1∣,∣σ2∣}

4.

 By plotting this out, we see there are distinct regions of the

parameter space which reveal a rich taxonomy of convergence behavior [10]:

s1 =

 s2 =

 complex02410Momentum

dt-math[block] {

 display: block;

}

β=\beta=β=Step-size

dt-math[block] {

 display: block;

}

α=\alpha=α=

0.00.20.40.60.81.01.2

 Convergence Rate

 A plot of

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

2β2\sqrt{\beta}2√β is independent of

dt-math[block] {

 display: block;

}

α\alphaα and

dt-math[block] {

 display: block;

}

λi\lambda\_iλi.

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xik−xi∗x\_i^k - x\_i^\*xik−xi∗**1-Step Convergence**When

dt-math[block] {

 display: block;

}

α=1/λi\alpha = 1/\lambda\_iα=1/λi, and

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xik−xi∗x\_i^k - x\_i^\*xik−xi∗**Monotonic Oscillations**When

dt-math[block] {

 display: block;

}

α>1/λi\alpha > 1/\lambda\_iα>1/λi, the iterates flip between

dt-math[block] {

 display: block;

}

+++ and

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xik−xi∗x\_i^k - x\_i^\*xik−xi∗**Divergence**When

dt-math[block] {

 display: block;

}

 deleteQueue.push(renderLoading(d3.select("#momentum2D")))

 renderQueue.push(function(callback) {

 var defaults = [[0.0015, 0.9],

 [0.0015, 0.125],

 [0.01, 0.00001],

 [0.02, 0.05 ],

 [0.025, 0.235 ]]

 coor = render2DSliderGen(

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[0](xy[0], xy[1],bold)

 updateStemGraphs[0](a,b)

 },

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[1](xy[0], xy[1],bold)

 updateStemGraphs[1](a,b)

 },

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[2](xy[0], xy[1],bold)

 updateStemGraphs[2](a,b)

 },

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[3](xy[0], xy[1],bold)

 updateStemGraphs[3](a,b)

 },

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[4](xy[0], xy[1],bold)

 updateStemGraphs[4](a,b)

 }, defaults)(d3.select("#momentumCanvas"))

 var tax = renderTaxonomy(d3.select("#momentum2D"))

 var updatePaths = renderOverlay(d3.select("#momentumOverlay"), tax.div)

 var updateStemGraphs = tax.update

 colorMap(

 d3.select("#momentumAnnotation"),

 180,

 d3.scaleLinear().domain([0,1.2001]).range([0, 180])

 )

 var up = function (i, alpha, beta) {

 var xy = coor(alpha, beta)

 updatePaths[i](xy[0], xy[1], true)

 updateStemGraphs[i](alpha,beta)

 }

 for (var i = 0; i<5; i++) {

 up(i,defaults[i][0], defaults[i][1])

 }

 renderMath(document.getElementById("momentum2D"))

 callback(null);

 });

 For what values of

dt-math[block] {

 display: block;

}

α\alphaα and

dt-math[block] {

 display: block;

}

β\betaβ does momentum converge? Since we need both

dt-math[block] {

 display: block;

}

σ1\sigma\_1σ1 and

dt-math[block] {

 display: block;

}

σ2\sigma\_2σ2 to converge, our convergence criterion is now

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

 We recover the previous result for gradient descent when

dt-math[block] {

 display: block;

}

---

### The Critical Damping Coefficient

 The true magic happens, however, when we find the sweet spot of

dt-math[block] {

 display: block;

}

α\alphaα and

dt-math[block] {

 display: block;

}

β\betaβ. Let us try to first optimize over

dt-math[block] {

 display: block;

}

β\betaβ.

 Momentum admits an interesting physical interpretation when

dt-math[block] {

 display: block;

}

α\alphaα is [11]

 small: it is a discretization of a damped harmonic oscillator. Consider

 a physical simulation operating in discrete time (like a video game).

dt-math[block] {

 display: block;

}

yik+1y\_{i}^{k+1}yik+1

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

λixik\lambda\_{i}x\_{i}^{k}λixik

 and perturbed by an external force field

 We can think of

dt-math[block] {

 display: block;

}

−yik-y\_i^k−yik as **velocity**

dt-math[block] {

 display: block;

}

βyik\beta y\_{i}^{k}βyik

 which is dampened at each step

dt-math[block] {

 display: block;

}

xik+1x\_i^{k+1}xik+1

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

xik−αyik+1x\_i^k - \alpha y\_i^{k+1}xik−αyik+1

 And

dt-math[block] {

 display: block;

}

xxx is our particle’s **position**

 which is moved at each step by a small amount in the direction of the velocity

dt-math[block] {

 display: block;

}

yik+1y^{k+1}\_iyik+1.

 renderMath(document.getElementById("momentum\_annotations"))

dt-math[block] {

 display: block;

}

β = 10.9850.950.90.850.8

λ = 0 0.02 0.1 0.25VelocityDampingExternal ForcePosition**dt-math[block] {

 display: block;

}

β\betaβ: Horizontal Axis**

When

dt-math[block] {

 display: block;

}

λi=0\lambda\_i = 0λi=0 and

dt-math[block] {

 display: block;

}

β=1\beta=1β=1, the object moves at constant speed. As

dt-math[block] {

 display: block;

}

 display: block;

}

λ\lambdaλ: Vertical Axis**

 The external force causes the particle to return to the origin.

Combining damping and the force field, the particle behaves like a

damped harmonic oscillator, returning lazily to equlibrium.

Initial point x = 1, y = 1

 deleteQueue.push(renderLoading(d3.select("#phasediagram0")))

 renderQueue.push(function(callback) {

 phaseDiagram\_dec(d3.select("#phasediagram0div"))

 renderMath(d3.select("#phasediagram0div").node())

 var figure = d3.select("#phasediagram0")

 var figwidth = figure.style("width")

 var figheight = figure.style("height")

 var svg = figure.append("svg")

 .style("width", figwidth)

 .style("height", figheight)

 .style("position", "absolute")

 .style("top","0px")

 .style("left","0px")

 .style("pointer-events", "none")

 callback(null);

 renderMath(document.getElementById("phasediagram0"))

 });

 This system is best imagined as a weight suspended on a spring. We

pull the weight down by one unit, and we study the path it follows as it

 returns to equilibrium. In the analogy, the spring is the source of our

 external force

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xikx^k\_ixik and the speed

dt-math[block] {

 display: block;

}

yiky^k\_iyik are 0. The choice of

dt-math[block] {

 display: block;

}

β\betaβ crucially affects the rate of return to equilibrium.

reachesoptimum**Overdamping** When

dt-math[block] {

 display: block;

}

β\betaβ is too small (e.g. in Gradient Descent,

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

RRR are repeated, when

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

β\betaβ

 is too large we're under-damping. Here the resistance is too small, and

 spring oscillates up and down forever, missing the optimal value over

and over. 1.00.80.60.40.20.0Momentum β

 deleteQueue.push(renderLoading(d3.select("#phasediagram1")))

 renderQueue.push(function(callback) {

 phaseDiagram(d3.select("#phasediagram1"));

 renderMath(document.getElementById("phasediagram1"))

 callback(null);

 });

 The critical value of

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

iii) of

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

1−αλi1-\alpha\lambda\_i1−αλi! Alas, this only applies to the error in the

dt-math[block] {

 display: block;

}

ithi^{th}ith eigenspace, with

dt-math[block] {

 display: block;

}

α\alphaα fixed.

### Optimal parameters

 To get a global convergence rate, we must optimize over both

dt-math[block] {

 display: block;

}

α\alphaα and

dt-math[block] {

 display: block;

}

β\betaβ. This is a more complicated affair,6 but they work out to be

dt-math[block] {

 display: block;

}

α=(2λ1+λn)2β=(λn−λ1λn+λ1)2

 \alpha =

\left(\frac{2}{\sqrt{\lambda\_{1}}+\sqrt{\lambda\_{n}}}\right)^{2} \quad

\beta =

 Plug this into the convergence rate, and you get

dt-math[block] {

 display: block;

}

κ−1κ+1\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}√κ+1√κ−1

 Convergence rate, **Momentum**

dt-math[block] {

 display: block;

}

κ−1κ+1 \frac{\kappa-1}{\kappa+1}κ+1κ−1

 Convergence rate, **Gradient Descent**

 renderMath(document.getElementById("conv\_rate\_comparisons"))

 With barely a modicum of extra effort, we have essentially square

rooted the condition number! These gains, in principle, require explicit

 knowledge of

dt-math[block] {

 display: block;

}

λ1\lambda\_1λ1 and

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

111. So set

dt-math[block] {

 display: block;

}

β\betaβ as close to

dt-math[block] {

 display: block;

}

111 as you can, and then find the highest

dt-math[block] {

 display: block;

}

We can do the same decomposition here with momentum, with eigenvalues

dt-math[block] {

 display: block;

}

λ1=0.01\lambda\_1=0.01λ1=0.01,

dt-math[block] {

 display: block;

}

λ2=0.1\lambda\_2=0.1λ2=0.1, and

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

f(wk)−f(w⋆)f(w^k) - f(w^\star)f(wk)−f(w⋆)

 Note that the optimal parameters do not necessarily imply the fastest

convergence, though, only the fastest asymptotic convergence rate.

Step-size α =

Momentum β =

02401

020406080100120140Eigenvalue 123

Optimal parameters

 deleteQueue.push(renderLoading(d3.select("#milestonesMomentumFig")))

 renderQueue.push(function(callback) {

 var graphDiv = d3.select("#milestonesMomentum")

 .style("width", 920 + "px")

 .style("height", 300 + "px")

 .style("top", "170px")

 .style("position", "relative")

 .style("margin-left", "auto")

 .style("margin-right", "auto")

 .attr("width", 920)

 .attr("height", 500)

 var svg = graphDiv.append("svg").attr("width", 940).attr("height", 500)

 var update = renderMilestones(svg, function() {})

 var reset = slider2D(d3.select("#sliderStep2D"),

 function(x,y) { update(x,y) },

 1,

 100)

 // // Swoopy Annotator

 annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 798,98 A 27.97 27.97 0 0 0 760.9999389648438,70.9999771118164",

 "text": "Optimal parameters",

 "textOffset": [

 740,

 109

 ]

 }

 ]

 var sel = drawAnnotations(d3.select("#milestonesMomentumFig"), annotations)

 callback(null);

 });

 While the loss function of gradient descent had a graceful, monotonic

curve, optimization with momentum displays clear oscillations. These

ripples are not restricted to quadratics, and occur in all kinds of

functions in practice. They are not cause for alarm, but are an

indication that extra tuning of the hyperparameters is required.

---

Example: The Colorization Problem

---------------------------------

dt-math[block] {

 display: block;

}

GGG be the graph with vertices as pixels,

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

minimize\text{minimize} minimize

dt-math[block] {

 display: block;

}

 The **colorizer** pulls distinguished pixels towards 1

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

 The **smoother** spreads out the color

 renderMath(document.getElementById("colorizer\_equation"))

 The optimal solution to this problem is a vector of all

dt-math[block] {

 display: block;

}

111’s 7.

 An inspection of the gradient iteration reveals why we take a long time

 to get there. The gradient step, for each component, is some form of

weighted average of the current value and its neighbors:

dt-math[block] {

 display: block;

}

wik+1=wik−α∑j∈N(wik−wjk)−{α(wik−1)i∈D0i∉D

\alpha(w\_{i}^{k}-1) & i\in D\\

0 & i\notin D

\end{cases}

 wik+1=wik−αj∈N∑(wik−wjk)−{α(wik−1)0i∈Di∉D

 This kind of local averaging is effective at smoothing out local

variations in the pixels, but poor at taking advantage of global

structure. The updates are akin to a drop of ink, diffusing through

water. Movement towards equilibrium is made only through local

corrections and so, left undisturbed, its march towards the solution is

slow and laborious. Fortunately, momentum speeds things up

significantly.

dt-math[block] {

 display: block;

}

RnR^nRn.

 The smallest eigenvalues have low frequencies, hence gradient descent

corrects high frequency errors well but not low frequency ones.

 deleteQueue.push(renderLoading(d3.select("#flow")))

 renderQueue.push(function(callback) {

 d3.queue()

 .defer(d3.json, "assets/data/Sigma.json")

 .defer(d3.json, "assets/data/matrix.json")

 .defer(d3.json, "assets/data/Uval.json")

 .await(function(error, FlowSigma, M, FlowU) {

 if (error) {

 console.error("Error loading data files");

 }

 else {

 var reset = renderFlowWidget(d3.select("#flow"), FlowSigma, M, FlowU)

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 389,55 A 28.57 28.57 0 0 0 352,30",

 "text": "Optimal parameters",

 "textOffset": [

 341,

 65

 ]

 }

 ]

 var sel = drawAnnotations(d3.select("#flow"), annotations)

 }

 });

 callback(null);

 });

 In vectorized form, the colorization problem is

dt-math[block] {

 display: block;

}

minimize\text{minimize}minimize

 The **smoother**’s quadratic form is the **Graph Laplacian**

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

12xTLGx\frac{1}{2}x^{T}L\_{G}x21xTLGx

 And the colorizer is a small low rank correction with a linear term.

dt-math[block] {

 display: block;

}

eie\_iei is the

dt-math[block] {

 display: block;

}

ithi^{th}ith unit vector.

 The Laplacian matrix,

dt-math[block] {

 display: block;

}

LGL\_GLG

8,

 which dominates the behavior of the optimization problem, is a valuable

 bridge between linear algebra and graph theory. This is a rich field of

 study, but one fact is pertinent to our discussion here. The

conditioning of

dt-math[block] {

 display: block;

}

LGL\_GLG,

 here defined as the ratio of the second eigenvector to the last (the

first eigenvalue is always 0, with eigenvector equal to the matrix of

all 1′s), is directly connected to the connectivity of the graph.

Small world graphs, like expanders and dense graphs, have excellent conditioning

The conditioning of grids improves with its dimensionality.

And long, wiry graphs, like paths, condition poorly.

 deleteQueue.push(renderLoading(d3.select("#laplacianConditioning")))

 renderQueue.push(function(callback) {

 var g1 = d3.select("#graph").append("g")

 genExpander(g1)

 var g2 = d3.select("#graph").append("g").attr("transform", "translate(300,0)")

 genGrid(g2)

 var g3 = d3.select("#graph").append("g").attr("transform", "translate(600,0)")

 genPath(g3);

 callback(null);

 });

 These observations carry through to the colorization problem, and the

intuition behind it should be clear. Well connected graphs allow rapid

diffusion of information through the edges, while graphs with poor

connectivity do not. And this principle, taken to the extreme, furnishes

 a class of functions so hard to optimize they reveal the limits of

first order optimization.

---

 The Limits of Descent

-----------------------

 Let’s take a step back. We have, with a clever trick, improved the

convergence of gradient descent by a quadratic factor with the

introduction of a single auxiliary sequence. But is this the best we can

 do? Could we improve convergence even more with two sequences? Could

one perhaps choose the

dt-math[block] {

 display: block;

}

α\alphaα’s and

dt-math[block] {

 display: block;

}

 Unfortunately, while improvements to the momentum algorithm do exist,

they all run into a certain, critical, almost inescapable lower bound.

### Adventures in Algorithmic Space

 To understand the limits of what we can do, we must first formally

define the algorithmic space in which we are searching. Here’s one

possible definition. The observation we will make is that both gradient

descent and momentum can be “unrolled”. Indeed, since

dt-math[block] {

 display: block;

}

 \begin{array}{lll}

 w^{1} & \!= & \!w^{0} ~-~ \alpha\nabla f(w^{0})\\[0.35em]

 w^{2} & \!= & \!w^{1} ~-~ \alpha\nabla f(w^{1})\\[0.35em]

 & \!= & \!w^{0} ~-~ \alpha\nabla f(w^{0}) ~-~ \alpha\nabla f(w^{1})\\[0.35em]

 & ~ \!\vdots \\

 \end{array}

 we can write gradient descent as

dt-math[block] {

 display: block;

}

wk+1  =  w0 − α∑ik∇f(wi).

 w^{k+1} ~~=~~ w^{0} ~-~ \alpha\sum\_i^k\nabla f(w^{i}).

 wk+1  =  w0 − αi∑k∇f(wi).

 A similar trick can be done with momentum:

dt-math[block] {

 display: block;

}

wk+1  =  w0 + α∑ik(1−βk+1−i)1−β∇f(wi).

 wk+1  =  w0 + αi∑k1−β(1−βk+1−i)∇f(wi).

 In fact, all manner of first order algorithms, including the Conjugate

 Gradient algorithm, AdaMax, Averaged Gradient and more, can be written

(though not quite so neatly) in this unrolled form. Therefore the class

of algorithms for which

dt-math[block] {

 display: block;

}

wk+1  =  w0 + ∑ikγik∇f(wi) for some γik

 wk+1  =  w0 + i∑kγik∇f(wi) for some γik

 contains momentum, gradient descent and a whole bunch of other

algorithms you might dream up. This is what is assumed in Assumption

dt-math[block] {

 display: block;

}

wk+1  =  w0 + ∑ikΓik∇f(wi) for some diagonal matrix Γik.

 wk+1  =  w0 + i∑kΓik∇f(wi) for some diagonal matrix Γik.

 This class of methods covers most of the popular algorithms for

training neural networks, including ADAM and AdaGrad. We shall refer to

this class of methods as “Linear First Order Methods”, and we will show a

 single function all these methods ultimately fail on.

### The Resisting Oracle

 Earlier, when we talked about the colorizer problem, we observed that

wiry graphs cause bad conditioning in our optimization problem. Taking

this to its extreme, we can look at a graph consisting of a single

path — a function so badly conditioned that Nesterov called a variant of

 it “the worst function in the world”. The function follows the same

structure as the colorizer problem, and we shall call this the Convex

Rosenbrock,

dt-math[block] {

 display: block;

}

fn(w)f^n(w)fn(w)

dt-math[block] {

 display: block;

}

===

 with a colorizer of one node

dt-math[block] {

 display: block;

}

12(w1−1)2\frac{1}{2}\left(w\_{1}-1\right)^{2}21(w1−1)2

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

 strong couplings of adjacent nodes in the path,

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

2κ−1∥w∥2.\frac{2}{\kappa-1}\|w\|^{2}.κ−12∥w∥2.

 and a small regularization term.

 The optimal solution of this problem is

dt-math[block] {

 display: block;

}

wi⋆=(κ−1κ+1)i

 w\_{i}^{\star}=\left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^{i}

 wi⋆=(√κ+1√κ−1)i

 and the condition number of the problem

dt-math[block] {

 display: block;

}

fnf^nfn approaches

dt-math[block] {

 display: block;

}

κ\kappaκ as

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

w0=0w^0 = 0w0=0.

02401

Step-size α =

Momentum β =

Here we see the first 50 iterates of momentum on the Convex Rosenbrock for

dt-math[block] {

 display: block;

}

The

 remaining expanding space is the “light cone” of our iterate’s

influence. Momentum does very well here with the optimal parameters.

Iteration 01020304050

Error

0.00.20.40.60.81.0

Weights

0.00.20.40.60.81.0

Optimal parameters

 deleteQueue.push(renderLoading(d3.select("#rosenViz")))

 renderQueue.push(function(callback) {

 var alpha = 0.003

 var beta = 0.8

 var b = zeros(25); b[0] = 249.75

 var iter = geniterMomentum(RQ, RLambda, b, alpha, beta).iter

 var res = function(i) { return iter(i)[0] }

 var sampleSVG = d3.select("#iterates")

 .style("display", "block")

 .append("svg")

 .attr("width", 770)

 .attr("height", 700)

 var xstar = iter(10000000)

 var numIters = 52

 var pathstr = "M 8,0"

 for (var j = 0; j < 27; j++) {

 if (j != 26 && j > 0) {

 pathstr = pathstr + "L " + j\*8 + "," + (j-1)\*8 + "L " + j\*8 + "," + j\*8

 }

 if (j == 0) {

 //pathstr = pathstr + "L " + j\*8 + "," + j\*8

 }

 if (j == 26) {

 pathstr = pathstr + "L " + j\*8 + "," + (j-1)\*8

 }

 }

 pathstr = pathstr + "L "+ 26\*8 + ",0 L 8,0 "

 var Disps3 = []

 for (var j = 0; j < numIters; j++) {

 var disp = errorPlot.append("g")

 var denter = disp.selectAll("rect")

 .data(iter(j)[1])

 .enter()

 denter.append("rect")

 .style("fill", function(d,i) { return rosen(d) })

 .attr("height", 7.7)

 .attr("width", 7.7)

 .attr("x", function(d, i){return i\*8+ 10})

 .attr("y", function(d, i){return j\*8 })

 Disps3.push(disp)

 }

 var Disps4 = []

 for (var j = 0; j < numIters; j++) {

 var disp = errorPlot.append("g")

 var denter = disp.selectAll("rect")

 .data(iter(j)[1])

 .enter()

 denter.append("rect")

 .style("fill", function(d,i) { return jet(d) })

 .attr("height", 7.7)

 .attr("width", 7.7)

 .attr("x", function(d, i){return i\*8+ 10})

 .attr("y", function(d, i){return j\*8 })

 if ((j % 10 == 0) || j == 0) {

 disp.append("text")

 .attr("class", "figtext")

 .attr("text-anchor", "end")

 .attr("x", 0)

 .attr("y", function(d, i){return (j\*8 + 10) })

 .html( (j == 0) ? "Iteration 0" : "" + j)

 }

 Disps4.push(disp)

 }

 function update(alpha, beta) {

 for (var j = 0; j < numIters; j++) {

 var iterj = iter(j)

 }

 }

 update(0.0035749243028120182, 0.8095238095238095)

 colorMap( d3.select("#rosen\_colorbar1"),

 194,

 rosen,

 d3.scaleLinear().domain([0,1]).range([0, 194]) )

 colorMap( d3.select("#rosen\_colorbar2"),

 194,

 jet,

 d3.scaleLinear().domain([0,1]).range([0, 194]) );

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 817,95 A 23.869 23.869 0 0 0 792,72",

 "text": "Optimal parameters",

 "textOffset": [

 767,

 109

 ]

 }

 ]

 var sel = drawAnnotations(d3.select("#rosenViz"), annotations)

 callback(null);

 });

 The observations made in the above diagram are true for any Linear

First Order algorithm. Let us prove this. First observe that each

component of the gradient depends only on the values directly before and

 after it:

dt-math[block] {

 display: block;

}

∇f(x)i=2wi−wi−1−wi+1+4κ−1wi,i≠1.

 ∇f(x)i=2wi−wi−1−wi+1+κ−14wi,i≠1.

 Therefore the fact we start at 0 guarantees that that component must

remain stoically there till an element either before or after it turns

nonzero. And therefore, by induction, for any linear first order

algorithm,

dt-math[block] {

 display: block;

}

 \begin{array}{lllllllll}

 w^{0} & = & [~~0, & 0, & 0, & \ldots & 0,

& 0, & \ldots & 0~]\\[0.35em]

 w^{1} & = & [~w\_{1}^{1}, & 0, & 0, & \ldots

& 0, & 0, & \ldots & 0~]\\[0.35em]

 w^{2} & = & [~w\_{1}^{2}, & w\_{2}^{2}, & 0, &

\ldots & 0, & 0, & \ldots & 0~]\\[0.35em]

 & ~ \vdots \\

 w^{k} & = & [~w\_{1}^{k}, & w\_{2}^{k}, & w\_{3}^{k},

& \ldots & w\_{k}^{k}, & 0, & \ldots & 0~].\\

 \end{array}

dt-math[block] {

 display: block;

}

kkk steps to move from

dt-math[block] {

 display: block;

}

w0w\_0w0 to

dt-math[block] {

 display: block;

}

wkw\_kwk. We can therefore sum up the errors which cannot have changed yet9:

dt-math[block] {

 display: block;

}

∥wk−w⋆∥∞≥maxi≥k+1{∣wi⋆∣}=(κ−1κ+1)k+1=(κ−1κ+1)k∥w0−w⋆∥∞.

 \begin{aligned}

\|w^{k}-w^{\star}\|\_{\infty}&\geq\max\_{i\geq

 \end{aligned}

 As

dt-math[block] {

 display: block;

}

nnn gets large, the condition number of

dt-math[block] {

 display: block;

}

fnf^nfn approaches

dt-math[block] {

 display: block;

}

κ\kappaκ.

 And the gap therefore closes; the convergence rate that momentum

promises matches the best any linear first order algorithm can do. And

we arrive at the disappointing conclusion that on this problem, we

cannot do better.

 Like many such lower bounds, this result must not be taken literally,

but spiritually. It, perhaps, gives a sense of closure and finality to

our investigation. But this is not the final word on first order

optimization. This lower bound does not preclude the possibility, for

example, of reformulating the problem to change the condition number

itself! There is still much room for speedups, if you understand the

right places to look.

Momentum with Stochastic Gradients

----------------------------------

 There is a final point worth addressing. All the discussion above

assumes access to the true gradient — a luxury seldom afforded in modern

 machine learning. Computing the exact gradient requires a full pass

over all the data, the cost of which can be prohibitively expensive.

Instead, randomized approximations of the gradient, like minibatch

sampling, are often used as a plug-in replacement of

dt-math[block] {

 display: block;

}

∇f(w)\nabla f(w)∇f(w). We can write the approximation in two parts,

dt-math[block] {

 display: block;

}

∇f(w)\nabla f(w)∇f(w)

 the true gradient

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

error(w).\text{error}(w).error(w).

 and an approximation error.

If the estimator is unbiased e.g.

dt-math[block] {

 display: block;

}

E[error(w)]=0\mathbf{E}[\text{error}(w)] = 0E[error(w)]=0

 renderMath(document.getElementById("truegradientpluserror"))

 It is helpful to think of our approximate gradient as the injection of

 a special kind of noise into our iteration. And using the machinery

developed in the previous sections, we can deal with this extra term

directly. On a quadratic, the error term cleaves cleanly into a separate

 term, where 10

dt-math[block] {

 display: block;

}

(yikxik) \left(\begin{array}{c}

 y\_{i}^{k}\\

 x\_{i}^{k}

 \end{array}\right)(yikxik)

 the noisy iterates are a sum of

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

Rk(yi0xi0)R^{k}\left(\begin{array}{c}

 y\_{i}^{0}\\

 x\_{i}^{0}

 \end{array}\right)Rk(yi0xi0)

 the noiseless, deterministic iterates and

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

ϵik∑j=1kRk−j(1−α)\epsilon^k\_i \sum\_{j=1}^{k}R^{k-j}\left(\begin{array}{c}

1\\

-\alpha

\end{array}\right)ϵikj=1∑kRk−j(1−α)

 a decaying sum of the errors, where

dt-math[block] {

 display: block;

}

ϵk=Q⋅error(wk)\epsilon^k = Q \cdot \text{error}(w^k)ϵk=Q⋅error(wk).

 renderMath(document.getElementById("iteratespluserror"))

 The error term,

dt-math[block] {

 display: block;

}

ϵk\epsilon^kϵk, with its dependence on the

dt-math[block] {

 display: block;

}

wkw^kwk, is a fairly hairy object. Following [10],

 we model this as independent 0-mean Gaussian noise. In this simplified

model, the objective also breaks into two separable components, a sum of

 a deterministic error and a stochastic error

 11, visualized here.

We decompose the expected value of the objective value

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

Ef(w)−f(w⋆)\mathbf{E} f(w) - f(w^\star) Ef(w)−f(w⋆)

 The small black dots are a single run of stochastic gradient

Step-size α =

Momentum β =

As [1]

 observes, the optimization has two phases. In the initial transient

phase the magnitude of the noise is smaller than the magnitude of the

gradient, and Momentum still makes good progress. In the second,

stochastic phase, the noise overwhelms the gradient, and momentum is

less effective.

02401

020406080100120140Fine-tuning phaseTransient phase

 deleteQueue.push(renderLoading(d3.select("#stochastic")))

 renderQueue.push(function(callback) {

 var graphDiv = d3.select("#stochasticgd")

 .style("width", 920 + "px")

 .style("height", 300 + "px")

 .style("top", "170px")

 .style("position", "relative")

 .style("margin-left", "auto")

 .style("margin-right", "auto")

 .attr("width", 920)

 .attr("height", 500)

 var svg = graphDiv.append("svg").attr("width", 940).attr("height", 500)

 var update = renderStochasticMilestones(svg, function() {})

 slidera(0.5,0.5)

 });

 Note that there are a set of unfortunate tradeoffs which seem to pit

the two components of error against each other. Lowering the step-size,

for example, decreases the stochastic error, but also slows down the

rate of convergence. And increasing momentum, contrary to popular

belief, causes the errors to compound. Despite these undesirable

properties, stochastic gradient descent with momentum has still been

shown to have competitive performance on neural networks. As [1]

 has observed, the transient phase seems to matter more than the

fine-tuning phase in machine learning. And in fact, it has been recently

 suggested [12]

 that this noise is a good thing — it acts as a implicit regularizer,

which, like early stopping, prevents overfitting in the fine-tuning

phase of optimization.

---

Onwards and Downwards

---------------------

 The study of acceleration is seeing a small revival within the

optimization community. If the ideas in this article excite you, you may

 wish to read [13],

 which fully explores the idea of momentum as the discretization of a

certain differential equation. But other, less physical, interpretations

 exist. There is an algebraic interpretation of momentum in terms of

 connecting momentum to older methods, like the Ellipsoid method. And

 But like the proverbial blind men feeling an elephant, momentum seems

like something bigger than the sum of its parts. One day, hopefully

soon, the many perspectives will converge into a satisfying whole.