Why Momentum Really Works
=========================

Step-size α = 0.02

Momentum β = 0.99

 We often think of Momentum as a means of dampening oscillations and speeding up the iterations, leading to faster convergence. But it has other interesting behavior. It allows a larger range of step-sizes to be used, and creates its own oscillations. What is going on?

 dt-byline {
 font-size: 12px;
 line-height: 18px;
 display: block;
 border-top: 1px solid rgba(0, 0, 0, 0.1);
 border-bottom: 1px solid rgba(0, 0, 0, 0.1);
 color: rgba(0, 0, 0, 0.5);
 padding-top: 12px;
 padding-bottom: 12px;
 }
 dt-article.centered dt-byline {
 text-align: center;

 }
 dt-byline a,
 dt-article dt-byline a {
 text-decoration: none;
 border-bottom: none;
 }
 dt-article dt-byline a:hover {
 text-decoration: underline;
 border-bottom: none;
 }
 dt-byline .authors {
 text-align: left;
 }
 dt-byline .name {
 display: inline;
 text-transform: uppercase;
 }
 dt-byline .affiliation {
 display: inline;
 }
 dt-byline .date {
 display: block;
 text-align: left;
 }
 dt-byline .year, dt-byline .month {
 display: inline;
 }
 dt-byline .citation {
 display: block;
 text-align: left;
 }
 dt-byline .citation div {
 display: inline;
 }

 @media(min-width: 768px) {
 dt-byline {
 }
 }

 @media(min-width: 1080px) {
 dt-byline {
 border-bottom: none;
 margin-bottom: 70px;
 }

 dt-byline a:hover {
 color: rgba(0, 0, 0, 0.9);
 }

 dt-byline .authors {
 display: inline-block;
 }

 dt-byline .author {
 display: inline-block;
 margin-right: 12px;
 /\*padding-left: 20px;\*/
 /\*border-left: 1px solid #ddd;\*/
 }

 dt-byline .affiliation {
 display: block;
 }

 dt-byline .author:last-child {
 margin-right: 0;
 }

 dt-byline .name {
 display: block;
 }

 dt-byline .date {
 border-left: 1px solid rgba(0, 0, 0, 0.1);
 padding-left: 15px;
 margin-left: 15px;
 display: inline-block;
 }
 dt-byline .year, dt-byline .month {
 display: block;
 }

 dt-byline .citation {
 border-left: 1px solid rgba(0, 0, 0, 0.15);
 padding-left: 15px;
 margin-left: 15px;
 display: inline-block;
 }
 dt-byline .citation div {
 display: block;
 }
 }

[Gabriel Goh](http://gabgoh.github.io)
[UC Davis](http://math.ucdavis.edu)

April. 4
2017

[Citation:
Goh, 2017](#citation)

 // Render Foreground
 var iterControl = genIterDiagram(bananaf, [1,1/3], [[-2,2],[2/3 + 0.4,-2/3 + 0.4]])
 .alpha(0.003)
 .beta(0)
 (d3.select("#banana").style("position","relative"))

 var iterChange = iterControl.control
 var getw0 = iterControl.w0

 var StepRange = d3.scaleLinear().domain([0,100]).range([0,0.0062])
 var MomentumRange = d3.scaleLinear().domain([0,100]).range([0,0.98])

 var update = function (i,j) { iterChange(i, 0, getw0()) }

 var slidera = sliderGen([230, 40])
 .ticks([0,0.003,0.006])
 .ticktitles( function(d,i) { return ["0", "0.003", "0.006"][i]})
 .change( function (i) {
 d3.select("#sliderAlpha").selectAll(".figtext").html("Step-size α = " + getalpha().toPrecision(2) )
 iterChange(getalpha(), getbeta(), getw0() )
 } )
 .startxval(0.003)
 .cRadius(7)
 .shifty(-12)
 .margins(20,20)

 var sliderb = sliderGen([230, 40])
 .ticks([0,0.5,0.99])
 .change( function (i) {
 d3.select("#sliderBeta").selectAll(".figtext").html("Momentum β = " + getbeta().toPrecision(2) )
 iterChange(getalpha(), getbeta(), getw0() )
 } )
 .cRadius(7)
 .shifty(-12)
 .startxval(0.74)
 .margins(20,20)

 var getalpha = slidera( d3.select("#sliderAlpha")).xval
 var getbeta = sliderb( d3.select("#sliderBeta")).xval

 iterChange(getalpha(), getbeta(), getw0() )

 Here’s a popular story about momentum [1, 2, 3]: gradient descent is a man walking down a hill. He follows the steepest path downwards; his progress is slow, but steady. Momentum is a heavy ball rolling down the same hill. The added inertia acts both as a smoother and an accelerator, dampening oscillations and causing us to barrel through narrow valleys, small humps and local minima.

 This standard story isn’t wrong, but it fails to explain many important behaviors of momentum. In fact, momentum can be understood far more precisely if we study it on the right model.

 One nice model is the convex quadratic. This model is rich enough to reproduce momentum’s local dynamics in real problems, and yet simple enough to be understood in closed form. This balance gives us powerful traction for understanding this algorithm.

---

 We begin with gradient descent. The algorithm has many virtues, but speed is not one of them. It is simple — when optimizing a smooth function

dt-math[block] {
 display: block;
}

fff, we make a small step in the gradient

dt-math[block] {
 display: block;
}

wk+1=wk−α∇f(wk).w^{k+1} = w^k-\alpha\nabla f(w^k).wk+1=wk−α∇f(wk).

 For a step-size small enough, gradient descent makes a monotonic improvement at every iteration. It always converges, albeit to a local minimum. And under a few weak curvature conditions it can even get there at an exponential rate.

 But the exponential decrease, though appealing in theory, can often be infuriatingly small. Things often begin quite well — with an impressive, almost immediate decrease in the loss. But as the iterations progress, things start to slow down. You start to get a nagging feeling you’re not making as much progress as you should be. What has gone wrong?

 The problem could be the optimizer’s old nemesis, pathological curvature. Pathological curvature is, simply put, regions of

dt-math[block] {
 display: block;
}

fff which aren’t scaled properly. The landscapes are often described as valleys, trenches, canals and ravines. The iterates either jump between valleys, or approach the optimum in small, timid steps. Progress along certain directions grind to a halt. In these unfortunate regions, gradient descent fumbles.

 Momentum proposes the following tweak to gradient descent. We give gradient descent a short-term memory:

dt-math[block] {
 display: block;
}

zk+1=βzk+∇f(wk)wk+1=wk−αzk+1
 \begin{aligned}
 z^{k+1}&=\beta z^{k}+\nabla f(w^{k})\\[0.4em]
 w^{k+1}&=w^{k}-\alpha z^{k+1}
 \end{aligned}
 zk+1wk+1=βzk+∇f(wk)=wk−αzk+1

 The change is innocent, and costs almost nothing. When

dt-math[block] {
 display: block;
}

β=0\beta = 0β=0 , we recover gradient descent. But for

dt-math[block] {
 display: block;
}

β=0.99\beta = 0.99β=0.99 (sometimes

dt-math[block] {
 display: block;
}

0.9990.9990.999, if things are really bad), this appears to be the boost we need. Our iterations regain that speed and boldness it lost, speeding to the optimum with a renewed energy.

 Optimizers call this minor miracle “acceleration”.

 The new algorithm may seem at first glance like a cheap hack. A simple trick to get around gradient descent’s more aberrant behavior — a smoother for oscillations between steep canyons. But the truth, if anything, is the other way round. It is gradient descent which is the hack. First, momentum gives up to a quadratic speedup on many functions. 1 This is no small matter — this is similar to the speedup you get from the Fast Fourier Transform, Quicksort, and Grover’s Algorithm. When the universe gives you quadratic speedups, you should start to pay attention.

 But there’s more. A lower bound, courtesy of Nesterov [5], states that momentum is, in a certain very narrow and technical sense, optimal. Now, this doesn’t mean it is the best algorithm for all functions in all circumstances. But it does satisfy some curiously beautiful mathematical properties which scratch a very human itch for perfection and closure. But more on that later. Let’s say this for now — momentum is an algorithm for the book.

---

First Steps: Gradient Descent
-----------------------------

 We begin by studying gradient descent on the simplest model possible which isn’t trivial — the convex quadratic,

dt-math[block] {
 display: block;
}

f(w)=12wTAw−bTw,w∈Rn.
 f(w) = \tfrac{1}{2}w^TAw - b^Tw, \qquad w \in \mathbf{R}^n.
 f(w)=21wTAw−bTw,w∈Rn.

 Assume

dt-math[block] {
 display: block;
}

AAA is symmetric and invertible, then the optimal solution

dt-math[block] {
 display: block;
}

w⋆w^{\star}w⋆ occurs at

dt-math[block] {
 display: block;
}

w⋆=A−1b. w^{\star} = A^{-1}b.w⋆=A−1b.

 Simple as this model may be, it is rich enough to approximate many functions (think of

dt-math[block] {
 display: block;
}

AAA as your favorite model of curvature — the Hessian, Fisher Information Matrix [6], etc) and captures all the key features of pathological curvature. And more importantly, we can write an exact closed formula for gradient descent on this function.

 This is how it goes. Since

dt-math[block] {
 display: block;
}

∇f(w)=Aw−b\nabla f(w)=Aw - b∇f(w)=Aw−b, the iterates are

dt-math[block] {
 display: block;
}

wk+1=wk−α(Awk−b).
 w^{k+1}=w^{k}- \alpha (Aw^{k} - b).
 wk+1=wk−α(Awk−b).

 Here’s the trick. There is a very natural space to view gradient descent where all the dimensions act independently — the eigenvectors of

dt-math[block] {
 display: block;
}

AAA.

 deleteQueue.push(renderLoading(d3.select("#change\_of\_variables")))
 renderQueue.push(function(callback) {
 var U = givens(Math.PI/4)
 var Ut = numeric.transpose(U)
 // Render Foreground
 var left = d3.select("#mom1").style("border", "1px solid rgba(0, 0, 0, 0.2)")

 var c1 = genIterDiagram(quadf, [0,0], [[-3,3],[-3,3]])
 .width(340)
 .height(340)
 .iters(300)
 .alpha(0.018)
 .showSolution(false)
 .pathWidth(1)
 .circleRadius(1.5)
 .pointerScale(0.8)
 .showStartingPoint(false)
 .drag(function() {
 c2.control(c1.alpha(),
 c1.beta(),
 numeric.dot(U,c1.w0())) })
 (left)

 var right = d3.select("#mom2").style("border", "1px solid rgba(0, 0, 0, 0.2)")
 var c2 = genIterDiagram(eyef, [0,0], [[-3,3],[-3,3]])
 .width(340)
 .height(340)
 .iters(300)
 .alpha(0.018)
 .showSolution(false)
 .pathWidth(1)
 .circleRadius(1.5)
 .pointerScale(0.8)
 .showStartingPoint(false)
 .drag(function() {
 c1.control(c2.alpha(),
 c2.beta(),
 numeric.dot(Ut,c2.w0())) })
 (right)

 // Initialize
 c2.control(0.018,0,[-2.5,1])
 c1.control(0.018,0,numeric.dot(Ut,[-2.5,1]));
 callback(null);
});

 Every symmetric matrix

dt-math[block] {
 display: block;
}

AAA has an eigenvalue decomposition

dt-math[block] {
 display: block;
}

A=Q diag(λ1,…,λn) QT,Q=[q1,…,qn],
 A=Q\ \text{diag}(\lambda\_{1},\ldots,\lambda\_{n})\ Q^{T},\qquad Q = [q\_1,\ldots,q\_n],
 A=Q diag(λ1,…,λn) QT,Q=[q1,…,qn],

 and, as per convention, we will assume that the

dt-math[block] {
 display: block;
}

λi\lambda\_iλi’s are sorted, from smallest

dt-math[block] {
 display: block;
}

λ1\lambda\_1λ1 to biggest

dt-math[block] {
 display: block;
}

λn\lambda\_nλn. If we perform a change of basis,

dt-math[block] {
 display: block;
}

xk=QT(wk−w⋆)x^{k} = Q^T(w^{k} - w^\star)xk=QT(wk−w⋆), the iterations break apart, becoming:

dt-math[block] {
 display: block;
}

xik+1=xik−αλixik=(1−αλi)xik=(1−αλi)k+1xi0
 \begin{aligned}
 x\_{i}^{k+1} & =x\_{i}^{k}-\alpha \lambda\_ix\_{i}^{k} \\[0.4em]
 &= (1-\alpha\lambda\_i)x^k\_i=(1-\alpha \lambda\_i)^{k+1}x^0\_i
 \end{aligned}
 xik+1=xik−αλixik=(1−αλi)xik=(1−αλi)k+1xi0

 Moving back to our original space

dt-math[block] {
 display: block;
}

www, we can see that

dt-math[block] {
 display: block;
}

wk−w⋆=Qxk=∑inxi0(1−αλi)kqi
 w^k - w^\star = Qx^k=\sum\_i^n x^0\_i(1-\alpha\lambda\_i)^k q\_i
 wk−w⋆=Qxk=i∑nxi0(1−αλi)kqi

 and there we have it — gradient descent in closed form.

### Decomposing the Error

 The above equation admits a simple interpretation. Each element of

dt-math[block] {
 display: block;
}

x0x^0x0 is the component of the error in the initial guess in the

dt-math[block] {
 display: block;
}

QQQ-basis. There are

dt-math[block] {
 display: block;
}

nnn such errors, and each of these errors follows its own, solitary path to the minimum, decreasing exponentially with a compounding rate of

dt-math[block] {
 display: block;
}

1−αλi1-\alpha\lambda\_i1−αλi. The closer that number is to

dt-math[block] {
 display: block;
}

111, the slower it converges.

 For most step-sizes, the eigenvectors with largest eigenvalues converge the fastest. This triggers an explosion of progress in the first few iterations, before things slow down as the smaller eigenvectors’ struggles are revealed. By writing the contributions of each eigenspace’s error to the loss

dt-math[block] {
 display: block;
}

f(wk)−f(w⋆)=∑(1−αλi)2kλi[xi0]2
 f(w^{k})-f(w^{\star})=\sum(1-\alpha\lambda\_{i})^{2k}\lambda\_{i}[x\_{i}^{0}]^2
 f(wk)−f(w⋆)=∑(1−αλi)2kλi[xi0]2
 we can visualize the contributions of each error component to the loss.

Optimization can be seen as combination of several component problems, shown here as  1  2  3 with eigenvalues

dt-math[block] {
 display: block;
}

λ1=0.01\lambda\_1=0.01λ1=0.01,

dt-math[block] {
 display: block;
}

λ2=0.1\lambda\_2=0.1λ2=0.1, and

dt-math[block] {
 display: block;
}

λ3=1\lambda\_3=1λ3=1 respectively.

 Step-size

 Optimal Step-size

 deleteQueue.push(renderLoading(d3.select("#milestones\_gd")))
 renderQueue.push(function(callback) {
 var graphDiv = d3.select("#obj")
 .style("width", 920 + "px")
 .style("height", 300 + "px")
 .style("top", "90px")
 .style("position", "relative")
 .style("margin-left", "auto")
 .style("margin-right", "auto")
 .attr("width", 920)
 .attr("height", 500)

 var svg = graphDiv.append("svg")
 .attr("width", 920)
 .attr("height", 300)
 .style("position","absolute")
 .style("left", "15px")

 var updateSliderGD = renderMilestones(svg, function() {});

 var alphaHTML = MathCache("alpha-equals");

 var slidera = sliderGen([250, 80])
 .ticks([0,1,200/(101),2])
 .change( function (i) {
 var html = alphaHTML + '' + i.toPrecision(4) + "";
 d3.select("#stepSizeMilestones")
 .html("Stepsize " + html )
 updateSliderGD(i,0.000)
 } )
 .ticktitles(function(d,i) { return [0,1,"",2][i] })
 .startxval(200/(101))
 .cRadius(7)
 .shifty(-12)
 .shifty(10)
 .margins(20,20)(d3.select("#sliderStep"))

 // renderDraggable(svg, [133.5, 23], [114.5, 90], 2, " ").attr("opacity", 0.1)
 // renderDraggable(svg, [133.5, 88], [115.5, 95], 2, " ").attr("opacity", 0.1)
 // renderDraggable(svg, [132.5, 154], [114.5, 100], 2, " ").attr("opacity", 0.1)

 d3.select("#milestones\_gd\_optstep").on("click", slidera.init)

 svg.append("text")
 .attr("class", "katex morsd mathit")
 .style("font-size", "19px")
 .style("font-family","KaTeX\_Math")
 .attr("x", 105)
 .attr("y", 50)
 .attr("text-anchor", "end")
 .attr("fill", "gray")
 .html("f(w<tspan baseline-shift = \"super\" font-size = \"15\">k</tspan>) - f(w<tspan baseline-shift = \"super\" font-size = \"15\">\*</tspan>)")

 svg.append("text")
 .style("font-size", "13px")
 .attr("x", 0)
 .attr("y", 80)
 .attr("dy", 0)
 .attr("transform", "translate(110,0)")
 .attr("class", "caption")
 .attr("text-anchor", "end")
 .attr("fill", "gray")
 .text("At the initial point, the error in each component is equal.")

 svg.selectAll(".caption").call(wrap, 100)

 svg.append("text")
 .style("font-size", "13px")
 .attr("x", 420)
 .attr("y", 270)
 .attr("dy", 0)
 .attr("dx", -295)
 .attr("text-anchor", "start")
 .attr("fill", "gray")
 .text("At the optimum, the rates of convergence of the largest and smallest eigenvalues equalize.")

 callback(null);
 });

### Choosing A Step-size

 The above analysis gives us immediate guidance as to how to set a step-size

dt-math[block] {
 display: block;
}

α\alphaα. In order to converge, each

dt-math[block] {
 display: block;
}

∣1−αλi∣|1-\alpha \lambda\_i|∣1−αλi∣ must be strictly less than 1. All workable step-sizes, therefore, fall in the interval

dt-math[block] {
 display: block;
}

0<αλi<2.0<\alpha\lambda\_i<2.0<αλi<2.

 The overall convergence rate is determined by the slowest error component, which must be either

dt-math[block] {
 display: block;
}

λ1\lambda\_1λ1 or

dt-math[block] {
 display: block;
}

λn\lambda\_nλn:

dt-math[block] {
 display: block;
}

rate(α) = maxi∣1−αλi∣ = max{∣1−αλ1∣, ∣1−αλn∣}
 \begin{aligned}\text{rate}(\alpha) & ~=~ \max\_{i}\left|1-\alpha\lambda\_{i}\right|\\[0.9em] & ~=~ \max\left\{|1-\alpha\lambda\_{1}|,~ |1-\alpha\lambda\_{n}|\right\} \end{aligned}
 rate(α) = imax∣1−αλi∣ = max{∣1−αλ1∣, ∣1−αλn∣}

 This overall rate is minimized when the rates for

dt-math[block] {
 display: block;
}

λ1\lambda\_1λ1 and

dt-math[block] {
 display: block;
}

λn\lambda\_nλn are the same — this mirrors our informal observation in the previous section that the optimal step-size causes the first and last eigenvectors to converge at the same rate. If we work this through we get:

dt-math[block] {
 display: block;
}

optimal α = argminα rate(α) = 2λ1+λnoptimal rate = minα rate(α) = λn/λ1−1λn/λ1+1
 \begin{aligned}
 \text{optimal }\alpha ~=~{\mathop{\text{argmin}}\limits\_\alpha} ~\text{rate}(\alpha) & ~=~\frac{2}{\lambda\_{1}+\lambda\_{n}}\\[1.4em]
 \text{optimal rate} ~=~{\min\_\alpha} ~\text{rate}(\alpha) & ~=~\frac{\lambda\_{n}/\lambda\_{1}-1}{\lambda\_{n}/\lambda\_{1}+1}
 \end{aligned}
 optimal α = αargmin rate(α)optimal rate = αmin rate(α) = λ1+λn2 = λn/λ1+1λn/λ1−1

 Notice the ratio

dt-math[block] {
 display: block;
}

λn/λ1\lambda\_n/\lambda\_1λn/λ1 determines the convergence rate of the problem. In fact, this ratio appears often enough that we give it a name, and a symbol — the condition number.

dt-math[block] {
 display: block;
}

condition number:=κ:=λnλ1
 \text{condition number} := \kappa :=\frac{\lambda\_n}{\lambda\_1}
 condition number:=κ:=λ1λn
 The condition number means many things. It is a measure of how close to singular a matrix is. It is a measure of how robust

dt-math[block] {
 display: block;
}

A−1bA^{-1}bA−1b is to perturbations in

dt-math[block] {
 display: block;
}

bbb. And, in this context, the condition number gives us a measure of how poorly gradient descent will perform. A ratio of

dt-math[block] {
 display: block;
}

κ=1\kappa = 1κ=1 is ideal, giving convergence in one step (of course, the function is trivial). Unfortunately the larger the ratio, the slower gradient descent will be. The condition number is therefore a direct measure of pathological curvature.

---

Example: Polynomial Regression
------------------------------

 The above analysis reveals an insight: all errors are not made equal. Indeed, there are different kinds of errors,

dt-math[block] {
 display: block;
}

nnn to be exact, one for each of the eigenvectors of

dt-math[block] {
 display: block;
}

AAA. And gradient descent is better at correcting some kinds of errors than others. But what do the eigenvectors of

dt-math[block] {
 display: block;
}

AAA mean? Surprisingly, in many applications they admit a very concrete interpretation.

 Lets see how this plays out in polynomial regression. Given 1D data,

dt-math[block] {
 display: block;
}

ξi\xi\_iξi, our problem is to fit the model

dt-math[block] {
 display: block;
}

model(ξ)=w1p1(ξ)+⋯+wnpn(ξ)pi=ξ↦ξi−1
\text{model}(\xi)=w\_{1}p\_{1}(\xi)+\cdots+w\_{n}p\_{n}(\xi)\qquad p\_{i}=\xi\mapsto\xi^{i-1}
 model(ξ)=w1p1(ξ)+⋯+wnpn(ξ)pi=ξ↦ξi−1

 to our observations,

dt-math[block] {
 display: block;
}

did\_idi. This model, though nonlinear in the input

dt-math[block] {
 display: block;
}

ξ\xiξ, is linear in the weights, and therefore we can write the model as a linear combination of monomials, like:

 deleteQueue.push(renderLoading(d3.select("#poly0")))
 renderQueue.push(function(callback) {

 // Preprocess x, get eigendecomposition, etc
 var x = [-0.6, -0.55,-0.5,-0.45,-0.4,0.4,0.45,0.5,0.55,0.6]
 var D = vandermonde(x, 5)
 var Eigs = eigSym(numeric.dot(numeric.transpose(D),D))
 var U = Eigs.U
 var lambda = Eigs.lambda

 // Preprocess y
 var b = [-3/2,-4/2,-5/2,-3/2,-2/2,1/2,2/2,3/2,2/2,1/2]
 var Dtb = numeric.dot(b,D)
 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var step = 1.8/lambda[0]
 var iter = geniter(U, lambda, Dtb, step)

 var eigensum = d3.select("#poly0")

 var wi = [-2,-2,2,2,2,-2]

 function refit(b) {
 var Dtb = numeric.dot(b,D)
 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))
 var Utsol = numeric.dot(sol,U)
 eigenControl.updateweights(Utsol)
 }

 var eigenControl = renderEigenPanel(eigensum, numeric.identity(6), x, b, wi, refit);

 // Swoopy Annotator
 var annotations = [
 {
 "x": 0,
 "y": 0,
 "path": "M 60,5 A 19.018 19.018 0 0 0 36,27",
 "text": "scrub values",
 "textOffset": [
 64,
 9
 ]
 }
 ]

 drawAnnotations(d3.select("#poly0f"), annotations)

 callback(null);
 });

 Because of the linearity, we can fit this model to our data

dt-math[block] {
 display: block;
}

ξi\xi\_iξi using linear regression on the model mismatch

dt-math[block] {
 display: block;
}

minimizew12∑i(model(ξi)−di)2  =  12∥Zw−d∥2
 \text{minimize}\_w \qquad\tfrac{1}{2}\sum\_i (\text{model}(\xi\_{i})-d\_{i})^{2} ~~=~~ \tfrac{1}{2}\|Zw - d\|^2
 minimizew21i∑(model(ξi)−di)2  =  21∥Zw−d∥2
 where

dt-math[block] {
 display: block;
}

Z=(1ξ1ξ12…ξ1n−11ξ2ξ22…ξ2n−1⋮⋮⋮⋱⋮1ξmξm2…ξmn−1).
 Z=\left(\begin{array}{ccccc}
 1 & \xi\_{1} & \xi\_{1}^{2} & \ldots & \xi\_{1}^{n-1}\\
 1 & \xi\_{2} & \xi\_{2}^{2} & \ldots & \xi\_{2}^{n-1}\\
 \vdots & \vdots & \vdots & \ddots & \vdots\\
 1 & \xi\_{m} & \xi\_{m}^{2} & \ldots & \xi\_{m}^{n-1}
 \end{array}\right).
 Z=⎝⎜⎜⎛11⋮1ξ1ξ2⋮ξmξ12ξ22⋮ξm2……⋱…ξ1n−1ξ2n−1⋮ξmn−1⎠⎟⎟⎞.

 The path of convergence, as we know, is elucidated when we view the iterates in the space of

dt-math[block] {
 display: block;
}

QQQ (the eigenvectors of

dt-math[block] {
 display: block;
}

ZTZZ^T ZZTZ). So let’s recast our regression problem in the basis of

dt-math[block] {
 display: block;
}

QQQ. First, we do a change of basis, by rotating

dt-math[block] {
 display: block;
}

www into

dt-math[block] {
 display: block;
}

QwQwQw, and counter-rotating our feature maps

dt-math[block] {
 display: block;
}

ppp into eigenspace,

dt-math[block] {
 display: block;
}

p¯\bar{p}p¯. We can now conceptualize the same regression as one over a different polynomial basis, with the model

dt-math[block] {
 display: block;
}

model(ξ) = x1p¯1(ξ) + ⋯ + xnp¯n(ξ)p¯i=∑qijpj.
 \text{model}(\xi)~=~x\_{1}\bar{p}\_{1}(\xi)~+~\cdots~+~x\_{n}\bar{p}\_{n}(\xi)\qquad \bar{p}\_{i}=\sum q\_{ij}p\_j.
 model(ξ) = x1p¯1(ξ) + ⋯ + xnp¯n(ξ)p¯i=∑qijpj.

 This model is identical to the old one. But these new features

dt-math[block] {
 display: block;
}

p¯\bar{p}p¯ (which I call “eigenfeatures”) and weights have the pleasing property that each coordinate acts independently of the others. Now our optimization problem breaks down, really, into

dt-math[block] {
 display: block;
}

nnn small 1D optimization problems. And each coordinate can be optimized greedily and independently, one at a time in any order, to produce the final, global, optimum. The eigenfeatures are also much more informative:

 deleteQueue.push(renderLoading(d3.select("#poly1")))
 renderQueue.push(function(callback) {
 var inv = function(lambda) { return 1/lambda }
 var scal = function(lambda) { return lambda < 1e-10 ? -100 : 1.5/Math.sqrt(lambda) }

 // Preprocess x, get eigendecomposition, etc
 var x = [-0.6, -0.55,-0.5,-0.45,-0.4,0.4,0.45,0.5,0.55,0.6]
 var D = vandermonde(x, 5)
 var Eigs = eigSym(numeric.dot(numeric.transpose(D),D))
 var U = Eigs.U
 var lambda = Eigs.lambda

 // Preprocess y
 var b = [-3/2,-4/2,-5/2,-3/2,-2/2,1/2,2/2,3/2,2/2,1/2]
 var Dtb = numeric.dot(b,D)
 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var step = 1.8/lambda[0]
 var iter = geniter(U, lambda, Dtb, step)

 var eigensum = d3.select("#poly1")

 var wi = lambda.slice(0).map(scal)

 function refit(b) {
 var Dtb = numeric.dot(b,D)
 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))
 var Utsol = numeric.dot(sol,U)
 eigenControl.updateweights(sol)
 }

 var eigenControl = renderEigenPanel(eigensum, U, x, b, wi, refit, true)

 var annotate = eigensum

 annotate.append("figcaption")
 .style("width", 230 + "px")
 .style("height", 150 + "px")
 .style("left", "0px")
 .style("position", "absolute")
 .style("padding", "10px")
 .html("The data comes in 2 clusters. The first 2 eigenfeatures capture variations between the clusters. ")

 annotate.append("figcaption")
 .style("width", 230 + "px")
 .style("height", 150 + "px")
 .style("left", "260px")
 .style("position", "absolute")
 .style("padding", "10px")
 .html("Next there are smooth variations within clusters, peaks within clusters,")

 annotate.append("figcaption")
 .style("width", 230 + "px")
 .style("height", 150 + "px")
 .style("left", 530 + "px")
 .style("position", "absolute")
 .style("padding", "10px")
 .html("and finally, jagged polynomials which differ wildly on neighboring points. ");

 // Swoopy Annotator
 var annotations = [
 {
 "x": 0,
 "y": 0,
 "path": "M 807,198 A 26.661 26.661 0 0 1 838,159",
 "text": "drag points to fit data",
 "textOffset": [
 799,
 214
 ]
 }]

 drawAnnotations(eigensum, annotations)
 callback(null);
 });

 The observations in the above diagram can be justified mathematically. From a statistical point of view, we would like a model which is, in some sense, robust to noise. Our model cannot possibly be meaningful if the slightest perturbation to the observations changes the entire model dramatically. And the eigenfeatures, the principal components of the data, give us exactly the decomposition we need to sort the features by its sensitivity to perturbations in

dt-math[block] {
 display: block;
}

did\_idi’s. The most robust components appear in the front (with the largest eigenvalues), and the most sensitive components in the back (with the smallest eigenvalues).

 This measure of robustness, by a rather convenient coincidence, is also a measure of how easily an eigenspace converges. And thus, the “pathological directions” — the eigenspaces which converge the slowest — are also those which are most sensitive to noise! So starting at a simple initial point like

dt-math[block] {
 display: block;
}

000 (by a gross abuse of language, let’s think of this as a prior), we track the iterates till a desired level of complexity is reached. Let’s see how this plays out in gradient descent.

 deleteQueue.push(renderLoading(d3.select("#poly2")))
 renderQueue.push(function(callback) {
 var inv = function(lambda) { return 1/lambda }
 var scal = function(lambda) { return lambda < 1e-10 ? -100 : 1.5/Math.sqrt(lambda) }

 // Preprocess x, get eigendecomposition, etc
 var x = [-0.6, -0.55,-0.5,-0.45,-0.4,0.4,0.45,0.5,0.55,0.6]
 var b = [-3/2,-4/2,-5/2,-3/2,-2/2,1/2,2/2,3/2,2/2,1/2]

 var D = vandermonde(x, 5)
 var Eigs = eigSym(numeric.dot(numeric.transpose(D),D))
 var U = Eigs.U
 var lambda = Eigs.lambda

 // Preprocess y
 var Dtb = numeric.dot(b,D)
 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var step = 1.8/lambda[0]
 var iter = geniter(U, lambda, Dtb, step)

 var eigensum = d3.select("#poly2")

 var wi = lambda.slice(0).map(scal)

 function refit(b) {
 var Dtb = numeric.dot(b,D)
 iter = geniter(U, lambda, Dtb, step)
 onChange(sliderControl.slidera.xval())
 }

 var eigenControl = renderEigenPanel(eigensum, U, x, b, wi, refit, true)

 var barlengths = getStepsConvergence(lambda, step).map(Math.log)

 var onChange = function(i) {
 eigenControl.updateweights(numeric.dot(U,iter(Math.floor(Math.exp(i-0.1)) )))
 }

 var sliderControl = sliderBarGen(barlengths, function(i) {return Math.exp(i-0.1)}).update(onChange)(d3.select("#poly2"))

 d3.select("#poly2").append("figcaption")
 .style("width", "120px")
 .style("position", "absolute")
 .style("left", "820px")
 .style("top","200px")
 .html("When an eigenspace has converged to three significant digits, the bar greys out. Drag the observations to change fit.")

 sliderControl.slidera.init()

 // var figwidth = d3.select("#poly2").style("width")
 // var figheight = d3.select("#poly2").style("height")
 // var svgannotate = d3.select("#poly2")
 // .append("svg")
 // .style("width", figwidth)
 // .style("height", figheight)
 // .style("position", "absolute")
 // .style("top","0px")
 // .style("left","0px")
 // .style("pointer-events","none")

 // renderDraggable(svgannotate,
 // [139.88888549804688, 243.77951049804688],
 // [121.88888549804688, 200.77951049804688],
 // 5,
 // "We begin at x=w=0");

 // Swoopy Annotator
 var annotations = [
 {
 "x": 0,
 "y": 0,
 "path": "M 74,202 A 52.274 52.274 0 0 0 134,245",
 "text": "We begin at x=w=0",
 "textOffset": [
 21,
 198
 ]
 }
 ]

 drawAnnotations(d3.select("#poly2"), annotations)

 callback(null);
 });

 This effect is harnessed with the heuristic of early stopping : by stopping the optimization early, you can often get better generalizing results. Indeed, the effect of early stopping is very similar to that of more conventional methods of regularization, such as Tikhonov Regression. Both methods try to suppress the components of the smallest eigenvalues directly, though they employ different methods of spectral decay.2 But early stopping has a distinct advantage. Once the step-size is chosen, there are no regularization parameters to fiddle with. Indeed, in the course of a single optimization, we have the entire family of models, from underfitted to overfitted, at our disposal. This gift, it seems, doesn’t come at a price. A beautiful free lunch [7] indeed.

---

The Dynamics of Momentum
------------------------

 Let’s turn our attention back to momentum. Recall that the momentum update is

dt-math[block] {
 display: block;
}

zk+1=βzk+∇f(wk)wk+1=wk−αzk+1.
 \begin{aligned}
 z^{k+1}&=\beta z^{k}+\nabla f(w^{k})\\[0.4em]
 w^{k+1}&=w^{k}-\alpha z^{k+1}.
 \end{aligned}
 zk+1wk+1=βzk+∇f(wk)=wk−αzk+1.

 Since

dt-math[block] {
 display: block;
}

∇f(wk)=Awk−b\nabla f(w^k) = Aw^k - b∇f(wk)=Awk−b, the update on the quadratic is

dt-math[block] {
 display: block;
}

zk+1=βzk+(Awk−b)wk+1=wk−αzk+1.
 \begin{aligned}
 z^{k+1}&=\beta z^{k}+ (Aw^{k}-b)\\[0.4em]
 w^{k+1}&=w^{k}-\alpha z^{k+1}.
 \end{aligned}
 zk+1wk+1=βzk+(Awk−b)=wk−αzk+1.

 Following [8], we go through the same motions, with the change of basis

dt-math[block] {
 display: block;
}

xk=Q(wk−w⋆)
 x^{k} = Q(w^{k} - w^\star)xk=Q(wk−w⋆) and

dt-math[block] {
 display: block;
}

yk=Qzk y^{k} = Qz^{k}yk=Qzk, to yield the update rule

dt-math[block] {
 display: block;
}

yik+1=βyik+λixikxik+1=xik−αyik+1.
 \begin{aligned}
 y\_{i}^{k+1}&=\beta y\_{i}^{k}+\lambda\_{i}x\_{i}^{k}\\[0.4em]
 x\_{i}^{k+1}&=x\_{i}^{k}-\alpha y\_{i}^{k+1}.
 \end{aligned}
 yik+1xik+1=βyik+λixik=xik−αyik+1.

 in which each component acts independently of the other components (though

dt-math[block] {
 display: block;
}

xikx^k\_ixik and

dt-math[block] {
 display: block;
}

yiky^k\_iyik are coupled). This lets us rewrite our iterates as

 3

dt-math[block] {
 display: block;
}

(yikxik)=Rk(yi0xi0)R=(βλi−αβ1−αλi).
 \left(\!\!\begin{array}{c}
 y\_{i}^{k}\\
 x\_{i}^{k}
 \end{array}\!\!\right)=R^k\left(\!\!\begin{array}{c}
 y\_{i}^{0}\\
 x\_{i}^{0}
 \end{array}\!\!\right)
 \qquad
 R = \left(\!\!\begin{array}{cc}
 \beta & \lambda\_{i}\\
 -\alpha\beta & 1-\alpha\lambda\_{i}
 \end{array}\!\!\right).
 (yikxik)=Rk(yi0xi0)R=(β−αβλi1−αλi).
 There are many ways of taking a matrix to the

dt-math[block] {
 display: block;
}

kthk^{th}kth power. But for the

dt-math[block] {
 display: block;
}

2×22 \times 22×2 case there is an elegant and little known formula [9] in terms of the eigenvalues of

dt-math[block] {
 display: block;
}

RRR,

dt-math[block] {
 display: block;
}

σ1\sigma\_1σ1 and

dt-math[block] {
 display: block;
}

σ2\sigma\_2σ2.

dt-math[block] {
 display: block;
}

Rk={σ1kR1−σ2kR2σ1≠σ2σ1k(kR/σ1−(k−1)I)σ1=σ2,Rj=R−σjIσ1−σ2
\color{#AAA}{\color{black}{R^{k}}=\begin{cases}
\color{black}{\sigma\_{1}^{k}}R\_{1}-\color{black}{\sigma\_{2}^{k}}R\_{2} & \sigma\_{1}\neq\sigma\_{2}\\
\sigma\_{1}^{k}(kR/\sigma\_1-(k-1)I) & \sigma\_{1}=\sigma\_{2}
\end{cases},\qquad R\_{j}=\frac{R-\sigma\_{j}I}{\sigma\_{1}-\sigma\_{2}}}
 Rk={σ1kR1−σ2kR2σ1k(kR/σ1−(k−1)I)σ1≠σ2σ1=σ2,Rj=σ1−σ2R−σjI

 This formula is rather complicated, but the takeaway here is that it plays the exact same role the individual convergence rates,

dt-math[block] {
 display: block;
}

1−αλi1-\alpha\lambda\_i1−αλi do in gradient descent. But instead of one geometric series, we have two coupled series, which may have real or complex values. The convergence rate is therefore the slowest of the two rates,

dt-math[block] {
 display: block;
}

max{∣σ1∣,∣σ2∣}\max\{|\sigma\_{1}|,|\sigma\_{2}|\}max{∣σ1∣,∣σ2∣}
4. By plotting this out, we see there are distinct regions of the parameter space which reveal a rich taxonomy of convergence behavior [10]:

 Convergence Rate

 A plot of

dt-math[block] {
 display: block;
}

max{∣σ1∣,∣σ2∣}\max\{|\sigma\_1|, |\sigma\_2|\}max{∣σ1∣,∣σ2∣} reveals distinct regions, each with its own style of convergence.

 deleteQueue.push(renderLoading(d3.select("#momentum2D")))
 renderQueue.push(function(callback) {
 var defaults = [[0.0015, 0.9],
 [0.0015, 0.125],
 [0.01, 0.00001],
 [0.02, 0.05 ],
 [0.025, 0.235 ]]

 coor = render2DSliderGen(
 function(a,b,bold) {
 var xy = coor(a,b)
 updatePaths[0](xy[0], xy[1],bold)
 updateStemGraphs[0](a,b)
 },
 function(a,b,bold) {
 var xy = coor(a,b)
 updatePaths[1](xy[0], xy[1],bold)
 updateStemGraphs[1](a,b)
 },
 function(a,b,bold) {
 var xy = coor(a,b)
 updatePaths[2](xy[0], xy[1],bold)
 updateStemGraphs[2](a,b)
 },
 function(a,b,bold) {
 var xy = coor(a,b)
 updatePaths[3](xy[0], xy[1],bold)
 updateStemGraphs[3](a,b)
 },
 function(a,b,bold) {
 var xy = coor(a,b)
 updatePaths[4](xy[0], xy[1],bold)
 updateStemGraphs[4](a,b)
 }, defaults)(d3.select("#momentumCanvas"))

 var tax = renderTaxonomy(d3.select("#momentum2D"))

 var updatePaths = renderOverlay(d3.select("#momentumOverlay"), tax.div)
 var updateStemGraphs = tax.update

 colorMap(
 d3.select("#momentumAnnotation"),
 180,
 d3.scaleLinear().domain([0,0.3,0.5,0.7,1,1.01]).range(colorbrewer.YlGnBu[5].concat(["black"])),
 d3.scaleLinear().domain([0,1.2001]).range([0, 180])
 )

 var up = function (i, alpha, beta) {
 var xy = coor(alpha, beta)
 updatePaths[i](xy[0], xy[1], true)
 updateStemGraphs[i](alpha,beta)
 }

 for (var i = 0; i<5; i++) {
 up(i,defaults[i][0], defaults[i][1])
 }

 renderMath(document.getElementById("momentum2D"))

 callback(null);
 });

 For what values of

dt-math[block] {
 display: block;
}

α\alphaα and

dt-math[block] {
 display: block;
}

β\betaβ does momentum converge? Since we need both

dt-math[block] {
 display: block;
}

σ1\sigma\_1σ1 and

dt-math[block] {
 display: block;
}

σ2\sigma\_2σ2 to converge, our convergence criterion is now

dt-math[block] {
 display: block;
}

max{∣σ1∣,∣σ2∣}<1\max\{|\sigma\_{1}|,|\sigma\_{2}|\} < 1max{∣σ1∣,∣σ2∣}<1. The range of available step-sizes work out 5 to be

dt-math[block] {
 display: block;
}

0<αλi<2+2βfor0≤β<10<\alpha\lambda\_{i}<2+2\beta \qquad \text{for} \qquad 0 \leq \beta < 10<αλi<2+2βfor0≤β<1

 We recover the previous result for gradient descent when

dt-math[block] {
 display: block;
}

β=0\beta = 0β=0. But notice an immediate boon we get. Momentum allows us to crank up the step-size up by a factor of 2 before diverging.

---

### The Critical Damping Coefficient

 The true magic happens, however, when we find the sweet spot of

dt-math[block] {
 display: block;
}

α\alphaα and

dt-math[block] {
 display: block;
}

β\betaβ. Let us try to first optimize over

dt-math[block] {
 display: block;
}

β\betaβ.

 Momentum admits an interesting physical interpretation when

dt-math[block] {
 display: block;
}

α\alphaα is [11] small: it is a discretization of a damped harmonic oscillator. Consider a physical simulation operating in discrete time (like a video game).

dt-math[block] {
 display: block;
}

yik+1y\_{i}^{k+1}yik+1

dt-math[block] {
 display: block;
}

===

dt-math[block] {
 display: block;
}

+++

dt-math[block] {
 display: block;
}

λixik\lambda\_{i}x\_{i}^{k}λixik

 and perturbed by an external force field

 We can think of

dt-math[block] {
 display: block;
}

−yik-y\_i^k−yik as **velocity**

dt-math[block] {
 display: block;
}

βyik\beta y\_{i}^{k}βyik

 which is dampened at each step

dt-math[block] {
 display: block;
}

xik+1x\_i^{k+1}xik+1

dt-math[block] {
 display: block;
}

===

dt-math[block] {
 display: block;
}

xik−αyik+1x\_i^k - \alpha y\_i^{k+1}xik−αyik+1

 And

dt-math[block] {
 display: block;
}

xxx is our particle’s **position**

 which is moved at each step by a small amount in the direction of the velocity

dt-math[block] {
 display: block;
}

yik+1y^{k+1}\_iyik+1.

 renderMath(document.getElementById("momentum\_annotations"))

 We can break this equation apart to see how each component affects the dynamics of the system. Here we plot, for

dt-math[block] {
 display: block;
}

150150150 iterates, the particle’s velocity (the horizontal axis) against its position (the vertical axis), in a phase diagram.

 deleteQueue.push(renderLoading(d3.select("#phasediagram0")))
 renderQueue.push(function(callback) {
 phaseDiagram\_dec(d3.select("#phasediagram0div"))
 renderMath(d3.select("#phasediagram0div").node())

 var figure = d3.select("#phasediagram0")
 var figwidth = figure.style("width")
 var figheight = figure.style("height")

 var svg = figure.append("svg")
 .style("width", figwidth)
 .style("height", figheight)
 .style("position", "absolute")
 .style("top","0px")
 .style("left","0px")
 .style("pointer-events", "none")

 renderDraggable(svg, [178.5, 66], [199.5, 66], 4, "Initial point <tspan x=\"0\" dy=\"1.2em\">x = 1, y = 1</tspan>").attr("opacity", 0.5)
 // renderDraggable(svg, [185, 116], [234, 174], 4, "constant speed").attr("opacity", 0.5)
 // renderDraggable(svg, [470, 86], [445, 71], 4, "slows to a halt").attr("opacity", 0.5)
 // renderDraggable(svg, [740, 64], [718, 97], 4, "all energy lost in 1 step").attr("opacity", 0.5)
 // renderDraggable(svg, [219, 269], [246, 327], 4, "particle returns slowly").attr("opacity", 0.5)
 // renderDraggable(svg, [250, 433], [287, 468], 4, "particle returns quickly").attr("opacity", 0.5)
 callback(null);
 renderMath(document.getElementById("phasediagram0"))

 });

 This system is best imagined as a weight suspended on a spring. We pull the weight down by one unit, and we study the path it follows as it returns to equilibrium. In the analogy, the spring is the source of our external force

dt-math[block] {
 display: block;
}

λixik\lambda\_ix^k\_iλixik, and equilibrium is the state when both the position

dt-math[block] {
 display: block;
}

xikx^k\_ixik and the speed

dt-math[block] {
 display: block;
}

yiky^k\_iyik are 0. The choice of

dt-math[block] {
 display: block;
}

β\betaβ crucially affects the rate of return to equilibrium.

 deleteQueue.push(renderLoading(d3.select("#phasediagram1")))
 renderQueue.push(function(callback) {
 phaseDiagram(d3.select("#phasediagram1"));
 renderMath(document.getElementById("phasediagram1"))
 callback(null);
 });

 The critical value of

dt-math[block] {
 display: block;
}

β=(1−αλi)2\beta = (1 - \sqrt{\alpha \lambda\_i})^2β=(1−√αλi)2 gives us a convergence rate (in eigenspace

dt-math[block] {
 display: block;
}

iii) of

dt-math[block] {
 display: block;
}

1−αλi.1 - \sqrt{\alpha\lambda\_i}.1−√αλi. A square root improvement over gradient descent,

dt-math[block] {
 display: block;
}

1−αλi1-\alpha\lambda\_i1−αλi! Alas, this only applies to the error in the

dt-math[block] {
 display: block;
}

ithi^{th}ith eigenspace, with

dt-math[block] {
 display: block;
}

α\alphaα fixed.

### Optimal parameters

 To get a global convergence rate, we must optimize over both

dt-math[block] {
 display: block;
}

α\alphaα and

dt-math[block] {
 display: block;
}

β\betaβ. This is a more complicated affair,6 but they work out to be

dt-math[block] {
 display: block;
}

α=(2λ1+λn)2β=(λn−λ1λn+λ1)2
 \alpha = \left(\frac{2}{\sqrt{\lambda\_{1}}+\sqrt{\lambda\_{n}}}\right)^{2} \quad \beta = \left(\frac{\sqrt{\lambda\_{n}}-\sqrt{\lambda\_{1}}}{\sqrt{\lambda\_{n}}+\sqrt{\lambda\_{1}}}\right)^{2}
 α=(√λ1+√λn2)2β=(√λn+√λ1√λn−√λ1)2
 Plug this into the convergence rate, and you get

dt-math[block] {
 display: block;
}

κ−1κ+1\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}√κ+1√κ−1

 Convergence rate, **Momentum**

dt-math[block] {
 display: block;
}

κ−1κ+1 \frac{\kappa-1}{\kappa+1}κ+1κ−1

 Convergence rate, **Gradient Descent**

 renderMath(document.getElementById("conv\_rate\_comparisons"))

 With barely a modicum of extra effort, we have essentially square rooted the condition number! These gains, in principle, require explicit knowledge of

dt-math[block] {
 display: block;
}

λ1\lambda\_1λ1 and

dt-math[block] {
 display: block;
}

λn\lambda\_nλn. But the formulas reveal a simple guideline. When the problem’s conditioning is poor, the optimal

dt-math[block] {
 display: block;
}

α\alphaα is approximately twice that of gradient descent, and the momentum term is close to

dt-math[block] {
 display: block;
}

111. So set

dt-math[block] {
 display: block;
}

β\betaβ as close to

dt-math[block] {
 display: block;
}

111 as you can, and then find the highest

dt-math[block] {
 display: block;
}

α\alphaα which still converges. Being at the knife’s edge of divergence, like in gradient descent, is a good place to be.

We can do the same decomposition here with momentum, with eigenvalues

dt-math[block] {
 display: block;
}

λ1=0.01\lambda\_1=0.01λ1=0.01,

dt-math[block] {
 display: block;
}

λ2=0.1\lambda\_2=0.1λ2=0.1, and

dt-math[block] {
 display: block;
}

λ3=1\lambda\_3=1λ3=1. Though the decrease is no longer monotonic, but significantly faster.

dt-math[block] {
 display: block;
}

f(wk)−f(w⋆)f(w^k) - f(w^\star)f(wk)−f(w⋆)
 Note that the optimal parameters do not necessarily imply the fastest convergence, though, only the fastest asymptotic convergence rate.
Step-size α =
Momentum β =

 deleteQueue.push(renderLoading(d3.select("#milestonesMomentumFig")))
 renderQueue.push(function(callback) {
 var graphDiv = d3.select("#milestonesMomentum")
 .style("width", 920 + "px")
 .style("height", 300 + "px")
 .style("top", "170px")
 .style("position", "relative")
 .style("margin-left", "auto")
 .style("margin-right", "auto")
 .attr("width", 920)
 .attr("height", 500)

 var svg = graphDiv.append("svg").attr("width", 940).attr("height", 500)

 var update = renderMilestones(svg, function() {})
 var reset = slider2D(d3.select("#sliderStep2D"),
 function(x,y) { update(x,y) },
 1,
 100)

 // // Swoopy Annotator
 annotations = [
 {
 "x": 0,
 "y": 0,
 "path": "M 798,98 A 27.97 27.97 0 0 0 760.9999389648438,70.9999771118164",
 "text": "Optimal parameters",
 "textOffset": [
 740,
 109
 ]
 }
 ]
 var sel = drawAnnotations(d3.select("#milestonesMomentumFig"), annotations)
 sel.selectAll("text").style("cursor", "pointer").style("pointer-events","all").on("click", reset)
 callback(null);
 });

 While the loss function of gradient descent had a graceful, monotonic curve, optimization with momentum displays clear oscillations. These ripples are not restricted to quadratics, and occur in all kinds of functions in practice. They are not cause for alarm, but are an indication that extra tuning of the hyperparameters is required.

---

Example: The Colorization Problem
---------------------------------

 Let’s look at how momentum accelerates convergence with a concrete example. On a grid of pixels let

dt-math[block] {
 display: block;
}

GGG be the graph with vertices as pixels,

dt-math[block] {
 display: block;
}

EEE be the set of edges connecting each pixel to its four neighboring pixels, and

dt-math[block] {
 display: block;
}

DDD be a small set of a few distinguished vertices. Consider the problem of minimizing

dt-math[block] {
 display: block;
}

minimize\text{minimize} minimize

dt-math[block] {
 display: block;
}

12∑i∈D(wi−1)2\qquad \frac{1}{2} \sum\_{i\in D} (w\_i - 1)^2 21i∈D∑(wi−1)2

 The **colorizer** pulls distinguished pixels towards 1

dt-math[block] {
 display: block;
}

+++

dt-math[block] {
 display: block;
}

12∑i,j∈E(wi−wj)2.\frac{1}{2} \sum\_{i,j\in E} (w\_i - w\_j)^2.21i,j∈E∑(wi−wj)2.

 The **smoother** spreads out the color

 renderMath(document.getElementById("colorizer\_equation"))

 The optimal solution to this problem is a vector of all

dt-math[block] {
 display: block;
}

111’s 7. An inspection of the gradient iteration reveals why we take a long time to get there. The gradient step, for each component, is some form of weighted average of the current value and its neighbors:

dt-math[block] {
 display: block;
}

wik+1=wik−α∑j∈N(wik−wjk)−{α(wik−1)i∈D0i∉D
w\_{i}^{k+1}=w\_{i}^{k}-\alpha\sum\_{j\in N}(w\_{i}^{k}-w\_{j}^{k})-\begin{cases}
\alpha(w\_{i}^{k}-1) & i\in D\\
0 & i\notin D
\end{cases}
 wik+1=wik−αj∈N∑(wik−wjk)−{α(wik−1)0i∈Di∉D

 This kind of local averaging is effective at smoothing out local variations in the pixels, but poor at taking advantage of global structure. The updates are akin to a drop of ink, diffusing through water. Movement towards equilibrium is made only through local corrections and so, left undisturbed, its march towards the solution is slow and laborious. Fortunately, momentum speeds things up significantly.

 The eigenvectors of the colorization problem form a generalized Fourier basis for

dt-math[block] {
 display: block;
}

RnR^nRn. The smallest eigenvalues have low frequencies, hence gradient descent corrects high frequency errors well but not low frequency ones.

 deleteQueue.push(renderLoading(d3.select("#flow")))
 renderQueue.push(function(callback) {
 d3.queue()
 .defer(d3.json, "assets/data/Sigma.json")
 .defer(d3.json, "assets/data/matrix.json")
 .defer(d3.json, "assets/data/Uval.json")
 .await(function(error, FlowSigma, M, FlowU) {
 if (error) {
 console.error("Error loading data files");
 }
 else {
 var reset = renderFlowWidget(d3.select("#flow"), FlowSigma, M, FlowU)
 // Swoopy Annotator
 var annotations = [
 {
 "x": 0,
 "y": 0,
 "path": "M 389,55 A 28.57 28.57 0 0 0 352,30",
 "text": "Optimal parameters",
 "textOffset": [
 341,
 65
 ]
 }
 ]
 var sel = drawAnnotations(d3.select("#flow"), annotations)
 sel.selectAll("text").style("cursor", "pointer").style("pointer-events","all").on("click", reset)
 }
 });
 callback(null);
 });

 In vectorized form, the colorization problem is

dt-math[block] {
 display: block;
}

minimize\text{minimize}minimize

 The **smoother**’s quadratic form is the **Graph Laplacian**

dt-math[block] {
 display: block;
}

12∑i∈D(xTeieiTx−eiTx)\frac{1}{2}\sum\_{i\in D}\left(x^{T}e\_{i}e\_{i}^{T}x-e\_{i}^{T}x\right)21i∈D∑(xTeieiTx−eiTx)

dt-math[block] {
 display: block;
}

+++

dt-math[block] {
 display: block;
}

12xTLGx\frac{1}{2}x^{T}L\_{G}x21xTLGx

 And the colorizer is a small low rank correction with a linear term.

dt-math[block] {
 display: block;
}

eie\_iei is the

dt-math[block] {
 display: block;
}

ithi^{th}ith unit vector.

 The Laplacian matrix,

dt-math[block] {
 display: block;
}

LGL\_GLG
8, which dominates the behavior of the optimization problem, is a valuable bridge between linear algebra and graph theory. This is a rich field of study, but one fact is pertinent to our discussion here. The conditioning of

dt-math[block] {
 display: block;
}

LGL\_GLG, here defined as the ratio of the second eigenvector to the last (the first eigenvalue is always 0, with eigenvector equal to the matrix of all 1′s), is directly connected to the connectivity of the graph.

Small world graphs, like expanders and dense graphs, have excellent conditioning
The conditioning of grids improves with its dimensionality.
And long, wiry graphs, like paths, condition poorly.

 deleteQueue.push(renderLoading(d3.select("#laplacianConditioning")))
 renderQueue.push(function(callback) {
 var g1 = d3.select("#graph").append("g")
 genExpander(g1)

 var g2 = d3.select("#graph").append("g").attr("transform", "translate(300,0)")
 genGrid(g2)

 var g3 = d3.select("#graph").append("g").attr("transform", "translate(600,0)")
 genPath(g3);
 callback(null);
 });

 These observations carry through to the colorization problem, and the intuition behind it should be clear. Well connected graphs allow rapid diffusion of information through the edges, while graphs with poor connectivity do not. And this principle, taken to the extreme, furnishes a class of functions so hard to optimize they reveal the limits of first order optimization.

---

 The Limits of Descent
-----------------------

 Let’s take a step back. We have, with a clever trick, improved the convergence of gradient descent by a quadratic factor with the introduction of a single auxiliary sequence. But is this the best we can do? Could we improve convergence even more with two sequences? Could one perhaps choose the

dt-math[block] {
 display: block;
}

α\alphaα’s and

dt-math[block] {
 display: block;
}

β\betaβ’s intelligently and adaptively? It is tempting to ride this wave of optimism - to the cube root and beyond!

 Unfortunately, while improvements to the momentum algorithm do exist, they all run into a certain, critical, almost inescapable lower bound.

### Adventures in Algorithmic Space

 To understand the limits of what we can do, we must first formally define the algorithmic space in which we are searching. Here’s one possible definition. The observation we will make is that both gradient descent and momentum can be “unrolled”. Indeed, since

dt-math[block] {
 display: block;
}

w1=w0 − α∇f(w0)w2=w1 − α∇f(w1)=w0 − α∇f(w0) − α∇f(w1) ⋮wk+1=w0 − α∇f(w0) −    ⋯⋯    − α∇f(wk)
 \begin{array}{lll}
 w^{1} & \!= & \!w^{0} ~-~ \alpha\nabla f(w^{0})\\[0.35em]
 w^{2} & \!= & \!w^{1} ~-~ \alpha\nabla f(w^{1})\\[0.35em]
 & \!= & \!w^{0} ~-~ \alpha\nabla f(w^{0}) ~-~ \alpha\nabla f(w^{1})\\[0.35em]
 & ~ \!\vdots \\

 w^{k+1} & \!= & \!w^{0} ~-~ \alpha\nabla f(w^{0}) ~-~~~~ \cdots\cdots ~~~~-~ \alpha\nabla f(w^{k})
 \end{array}
 w1w2wk+1=== ⋮=w0 − α∇f(w0)w1 − α∇f(w1)w0 − α∇f(w0) − α∇f(w1)w0 − α∇f(w0) −    ⋯⋯    − α∇f(wk)
 we can write gradient descent as

dt-math[block] {
 display: block;
}

wk+1  =  w0 − α∑ik∇f(wi).
 w^{k+1} ~~=~~ w^{0} ~-~ \alpha\sum\_i^k\nabla f(w^{i}).
 wk+1  =  w0 − αi∑k∇f(wi).

 A similar trick can be done with momentum:

dt-math[block] {
 display: block;
}

wk+1  =  w0 + α∑ik(1−βk+1−i)1−β∇f(wi).
 w^{k+1} ~~=~~ w^{0} ~+~ \alpha\sum\_i^k\frac{(1-\beta^{k+1-i})}{1-\beta}\nabla f(w^i).
 wk+1  =  w0 + αi∑k1−β(1−βk+1−i)∇f(wi).

 In fact, all manner of first order algorithms, including the Conjugate Gradient algorithm, AdaMax, Averaged Gradient and more, can be written (though not quite so neatly) in this unrolled form. Therefore the class of algorithms for which

dt-math[block] {
 display: block;
}

wk+1  =  w0 + ∑ikγik∇f(wi) for some γik
 w^{k+1} ~~=~~ w^{0} ~+~ \sum\_{i}^{k}\gamma\_{i}^{k}\nabla f(w^{i}) \qquad \text{ for some } \gamma\_{i}^{k}
 wk+1  =  w0 + i∑kγik∇f(wi) for some γik

 contains momentum, gradient descent and a whole bunch of other algorithms you might dream up. This is what is assumed in Assumption 2.1.4 [5] of Nesterov. But let’s push this even further, and expand this class to allow different step-sizes for different directions.

dt-math[block] {
 display: block;
}

wk+1  =  w0 + ∑ikΓik∇f(wi) for some diagonal matrix Γik.
 w^{k+1} ~~=~~ w^{0} ~+~ \sum\_{i}^{k}\Gamma\_{i}^{k}\nabla f(w^{i}) \quad \text{ for some diagonal matrix } \Gamma\_{i}^{k} .
 wk+1  =  w0 + i∑kΓik∇f(wi) for some diagonal matrix Γik.

 This class of methods covers most of the popular algorithms for training neural networks, including ADAM and AdaGrad. We shall refer to this class of methods as “Linear First Order Methods”, and we will show a single function all these methods ultimately fail on.

### The Resisting Oracle

 Earlier, when we talked about the colorizer problem, we observed that wiry graphs cause bad conditioning in our optimization problem. Taking this to its extreme, we can look at a graph consisting of a single path — a function so badly conditioned that Nesterov called a variant of it “the worst function in the world”. The function follows the same structure as the colorizer problem, and we shall call this the Convex Rosenbrock,

dt-math[block] {
 display: block;
}

fn(w)f^n(w)fn(w)

dt-math[block] {
 display: block;
}

===

 with a colorizer of one node

dt-math[block] {
 display: block;
}

12(w1−1)2\frac{1}{2}\left(w\_{1}-1\right)^{2}21(w1−1)2

dt-math[block] {
 display: block;
}

+++

dt-math[block] {
 display: block;
}

12∑i=1n(wi−wi+1)2\frac{1}{2}\sum\_{i=1}^{n}(w\_{i}-w\_{i+1})^{2}21i=1∑n(wi−wi+1)2

 strong couplings of adjacent nodes in the path,

dt-math[block] {
 display: block;
}

+++

dt-math[block] {
 display: block;
}

2κ−1∥w∥2.\frac{2}{\kappa-1}\|w\|^{2}.κ−12∥w∥2.

 and a small regularization term.

 The optimal solution of this problem is

dt-math[block] {
 display: block;
}

wi⋆=(κ−1κ+1)i
 w\_{i}^{\star}=\left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^{i}
 wi⋆=(√κ+1√κ−1)i

 and the condition number of the problem

dt-math[block] {
 display: block;
}

fnf^nfn approaches

dt-math[block] {
 display: block;
}

κ\kappaκ as

dt-math[block] {
 display: block;
}

nnn goes to infinity. Now observe the behavior of the momentum algorithm on this function, starting from

dt-math[block] {
 display: block;
}

w0=0w^0 = 0w0=0.

Step-size α =
Momentum β =
Here we see the first 50 iterates of momentum on the Convex Rosenbrock for

dt-math[block] {
 display: block;
}

n=25n=25n=25. The behavior here is similar to that of any Linear First Order Algorithm.

This triangle is a “dead zone” of our iterates. The iterates are always 0, no matter what the parameters.
The remaining expanding space is the “light cone” of our iterate’s influence. Momentum does very well here with the optimal parameters.

Error

Weights

 deleteQueue.push(renderLoading(d3.select("#rosenViz")))
 renderQueue.push(function(callback) {
 var RQ = [[0.033430859446525574,0.06637421995401382,0.09834969788789749,0.1288910210132599,0.1575528085231781,0.18391713500022888,0.20759953558444977,0.22825466096401215,0.24558131396770477,0.25932684540748596,0.2692908048629761,0.2753278911113739,0.2773500978946686,0.2753278911113739,0.2692908048629761,0.25932684540748596,0.24558131396770477,0.22825466096401215,0.20759953558444977,0.18391713500022888,0.1575528085231781,0.1288910210132599,0.09834969788789749,0.06637421995401382,0.033430859446525574],[-0.06637421995401382,-0.1288910210132599,-0.18391713500022888,-0.22825466096401215,-0.25932684540748596,-0.2753278911113739,-0.2753278911113739,-0.25932684540748596,-0.22825466096401215,-0.18391713500022888,-0.1288910210132599,-0.06637421995401382,-1.917815576756455e-14,0.06637421995401382,0.1288910210132599,0.18391713500022888,0.22825466096401215,0.25932684540748596,0.2753278911113739,0.2753278911113739,0.25932684540748596,0.22825466096401215,0.18391713500022888,0.1288910210132599,0.06637421995401382],[0.09834969788789749,0.18391713500022888,0.24558131396770477,0.2753278911113739,0.2692908048629761,0.22825466096401215,0.1575528085231781,0.06637421995401382,-0.033430859446525574,-0.1288910210132599,-0.20759953558444977,-0.25932684540748596,-0.2773500978946686,-0.25932684540748596,-0.20759953558444977,-0.1288910210132599,-0.033430859446525574,0.06637421995401382,0.1575528085231781,0.22825466096401215,0.2692908048629761,0.2753278911113739,0.24558131396770477,0.18391713500022888,0.09834969788789749],[0.1288910210132599,0.22825466096401215,0.2753278911113739,0.25932684540748596,0.18391713500022888,0.06637421995401382,-0.06637421995401382,-0.18391713500022888,-0.25932684540748596,-0.2753278911113739,-0.22825466096401215,-0.1288910210132599,-1.2834394835694717e-14,0.1288910210132599,0.22825466096401215,0.2753278911113739,0.25932684540748596,0.18391713500022888,0.06637421995401382,-0.06637421995401382,-0.18391713500022888,-0.25932684540748596,-0.2753278911113739,-0.22825466096401215,-0.1288910210132599],[-0.1575528085231781,-0.25932684540748596,-0.2692908048629761,-0.18391713500022888,-0.033430859446525574,0.1288910210132599,0.24558131396770477,0.2753278911113739,0.20759953558444977,0.06637421995401382,-0.09834969788789749,-0.22825466096401215,-0.2773500978946686,-0.22825466096401215,-0.09834969788789749,0.06637421995401382,0.20759953558444977,0.2753278911113739,0.24558131396770477,0.1288910210132599,-0.033430859446525574,-0.18391713500022888,-0.2692908048629761,-0.25932684540748596,-0.1575528085231781],[-0.18391713500022888,-0.2753278911113739,-0.22825466096401215,-0.06637421995401382,0.1288910210132599,0.25932684540748596,0.25932684540748596,0.1288910210132599,-0.06637421995401382,-0.22825466096401215,-0.2753278911113739,-0.18391713500022888,-4.583645525716709e-15,0.18391713500022888,0.2753278911113739,0.22825466096401215,0.06637421995401382,-0.1288910210132599,-0.25932684540748596,-0.25932684540748596,-0.1288910210132599,0.06637421995401382,0.22825466096401215,0.2753278911113739,0.18391713500022888],[-0.20759953558444977,-0.2753278911113739,-0.1575528085231781,0.06637421995401382,0.24558131396770477,0.25932684540748596,0.09834969788789749,-0.1288910210132599,-0.2692908048629761,-0.22825466096401215,-0.033430859446525574,0.18391713500022888,0.2773500978946686,0.18391713500022888,-0.033430859446525574,-0.22825466096401215,-0.2692908048629761,-0.1288910210132599,0.09834969788789749,0.25932684540748596,0.24558131396770477,0.06637421995401382,-0.1575528085231781,-0.2753278911113739,-0.20759953558444977],[-0.22825466096401215,-0.25932684540748596,-0.06637421995401382,0.18391713500022888,0.2753278911113739,0.1288910210132599,-0.1288910210132599,-0.2753278911113739,-0.18391713500022888,0.06637421995401382,0.25932684540748596,0.22825466096401215,4.337912797388764e-15,-0.22825466096401215,-0.25932684540748596,-0.06637421995401382,0.18391713500022888,0.2753278911113739,0.1288910210132599,-0.1288910210132599,-0.2753278911113739,-0.18391713500022888,0.06637421995401382,0.25932684540748596,0.22825466096401215],[-0.24558131396770477,-0.22825466096401215,0.033430859446525574,0.25932684540748596,0.20759953558444977,-0.06637421995401382,-0.2692908048629761,-0.18391713500022888,0.09834969788789749,0.2753278911113739,0.1575528085231781,-0.1288910210132599,-0.2773500978946686,-0.1288910210132599,0.1575528085231781,0.2753278911113739,0.09834969788789749,-0.18391713500022888,-0.2692908048629761,-0.06637421995401382,0.20759953558444977,0.25932684540748596,0.033430859446525574,-0.22825466096401215,-0.24558131396770477],[-0.25932684540748596,-0.18391713500022888,0.1288910210132599,0.2753278911113739,0.06637421995401382,-0.22825466096401215,-0.22825466096401215,0.06637421995401382,0.2753278911113739,0.1288910210132599,-0.18391713500022888,-0.25932684540748596,-2.6265288907570752e-15,0.25932684540748596,0.18391713500022888,-0.1288910210132599,-0.2753278911113739,-0.06637421995401382,0.22825466096401215,0.22825466096401215,-0.06637421995401382,-0.2753278911113739,-0.1288910210132599,0.18391713500022888,0.25932684540748596],[0.2692908048629761,0.1288910210132599,-0.20759953558444977,-0.22825466096401215,0.09834969788789749,0.2753278911113739,0.033430859446525574,-0.25932684540748596,-0.1575528085231781,0.18391713500022888,0.24558131396770477,-0.06637421995401382,-0.2773500978946686,-0.06637421995401382,0.24558131396770477,0.18391713500022888,-0.1575528085231781,-0.25932684540748596,0.033430859446525574,0.2753278911113739,0.09834969788789749,-0.22825466096401215,-0.20759953558444977,0.1288910210132599,0.2692908048629761],[0.2753278911113739,0.06637421995401382,-0.25932684540748596,-0.1288910210132599,0.22825466096401215,0.18391713500022888,-0.18391713500022888,-0.22825466096401215,0.1288910210132599,0.25932684540748596,-0.06637421995401382,-0.2753278911113739,-2.224606886845463e-15,0.2753278911113739,0.06637421995401382,-0.25932684540748596,-0.1288910210132599,0.22825466096401215,0.18391713500022888,-0.18391713500022888,-0.22825466096401215,0.1288910210132599,0.25932684540748596,-0.06637421995401382,-0.2753278911113739],[0.2773500978946686,1.2625047254649028e-16,-0.2773500978946686,-6.312523627324514e-16,0.2773500978946686,8.83753307825432e-16,-0.2773500978946686,-8.83753307825432e-16,0.2773500978946686,1.5781309068311285e-15,-0.2773500978946686,-4.734392720493385e-16,0.2773500978946686,2.0200075607438445e-15,-0.2773500978946686,-1.7990691808479273e-15,0.2773500978946686,1.6728188141805554e-15,-0.2773500978946686,-1.4518804342846382e-15,0.2773500978946686,-3.4718879950284827e-16,-0.2773500978946686,1.2625047254649028e-15,0.2773500978946686],[-0.2753278911113739,0.06637421995401382,0.25932684540748596,-0.1288910210132599,-0.22825466096401215,0.18391713500022888,0.18391713500022888,-0.22825466096401215,-0.1288910210132599,0.25932684540748596,0.06637421995401382,-0.2753278911113739,-1.0026396759284933e-15,0.2753278911113739,-0.06637421995401382,-0.25932684540748596,0.1288910210132599,0.22825466096401215,-0.18391713500022888,-0.18391713500022888,0.22825466096401215,0.1288910210132599,-0.25932684540748596,-0.06637421995401382,0.2753278911113739],[0.2692908048629761,-0.1288910210132599,-0.20759953558444977,0.22825466096401215,0.09834969788789749,-0.2753278911113739,0.033430859446525574,0.25932684540748596,-0.1575528085231781,-0.18391713500022888,0.24558131396770477,0.06637421995401382,-0.2773500978946686,0.06637421995401382,0.24558131396770477,-0.18391713500022888,-0.1575528085231781,0.25932684540748596,0.033430859446525574,-0.2753278911113739,0.09834969788789749,0.22825466096401215,-0.20759953558444977,-0.1288910210132599,0.2692908048629761],[-0.25932684540748596,0.18391713500022888,0.1288910210132599,-0.2753278911113739,0.06637421995401382,0.22825466096401215,-0.22825466096401215,-0.06637421995401382,0.2753278911113739,-0.1288910210132599,-0.18391713500022888,0.25932684540748596,1.770693656524247e-16,-0.25932684540748596,0.18391713500022888,0.1288910210132599,-0.2753278911113739,0.06637421995401382,0.22825466096401215,-0.22825466096401215,-0.06637421995401382,0.2753278911113739,-0.1288910210132599,-0.18391713500022888,0.25932684540748596],[0.24558131396770477,-0.22825466096401215,-0.033430859446525574,0.25932684540748596,-0.20759953558444977,-0.06637421995401382,0.2692908048629761,-0.18391713500022888,-0.09834969788789749,0.2753278911113739,-0.1575528085231781,-0.1288910210132599,0.2773500978946686,-0.1288910210132599,-0.1575528085231781,0.2753278911113739,-0.09834969788789749,-0.18391713500022888,0.2692908048629761,-0.06637421995401382,-0.20759953558444977,0.25932684540748596,-0.033430859446525574,-0.22825466096401215,0.24558131396770477],[0.22825466096401215,-0.25932684540748596,0.06637421995401382,0.18391713500022888,-0.2753278911113739,0.1288910210132599,0.1288910210132599,-0.2753278911113739,0.18391713500022888,0.06637421995401382,-0.25932684540748596,0.22825466096401215,9.870699398085384e-16,-0.22825466096401215,0.25932684540748596,-0.06637421995401382,-0.18391713500022888,0.2753278911113739,-0.1288910210132599,-0.1288910210132599,0.2753278911113739,-0.18391713500022888,-0.06637421995401382,0.25932684540748596,-0.22825466096401215],[-0.20759953558444977,0.2753278911113739,-0.1575528085231781,-0.06637421995401382,0.24558131396770477,-0.25932684540748596,0.09834969788789749,0.1288910210132599,-0.2692908048629761,0.22825466096401215,-0.033430859446525574,-0.18391713500022888,0.2773500978946686,-0.18391713500022888,-0.033430859446525574,0.22825466096401215,-0.2692908048629761,0.1288910210132599,0.09834969788789749,-0.25932684540748596,0.24558131396770477,-0.06637421995401382,-0.1575528085231781,0.2753278911113739,-0.20759953558444977],[-0.18391713500022888,0.2753278911113739,-0.22825466096401215,0.06637421995401382,0.1288910210132599,-0.25932684540748596,0.25932684540748596,-0.1288910210132599,-0.06637421995401382,0.22825466096401215,-0.2753278911113739,0.18391713500022888,8.371955066048889e-16,-0.18391713500022888,0.2753278911113739,-0.22825466096401215,0.06637421995401382,0.1288910210132599,-0.25932684540748596,0.25932684540748596,-0.1288910210132599,-0.06637421995401382,0.22825466096401215,-0.2753278911113739,0.18391713500022888],[-0.1575528085231781,0.25932684540748596,-0.2692908048629761,0.18391713500022888,-0.033430859446525574,-0.1288910210132599,0.24558131396770477,-0.2753278911113739,0.20759953558444977,-0.06637421995401382,-0.09834969788789749,0.22825466096401215,-0.2773500978946686,0.22825466096401215,-0.09834969788789749,-0.06637421995401382,0.20759953558444977,-0.2753278911113739,0.24558131396770477,-0.1288910210132599,-0.033430859446525574,0.18391713500022888,-0.2692908048629761,0.25932684540748596,-0.1575528085231781],[-0.1288910210132599,0.22825466096401215,-0.2753278911113739,0.25932684540748596,-0.18391713500022888,0.06637421995401382,0.06637421995401382,-0.18391713500022888,0.25932684540748596,-0.2753278911113739,0.22825466096401215,-0.1288910210132599,-3.0802549003271684e-16,0.1288910210132599,-0.22825466096401215,0.2753278911113739,-0.25932684540748596,0.18391713500022888,-0.06637421995401382,-0.06637421995401382,0.18391713500022888,-0.25932684540748596,0.2753278911113739,-0.22825466096401215,0.1288910210132599],[-0.09834969788789749,0.18391713500022888,-0.24558131396770477,0.2753278911113739,-0.2692908048629761,0.22825466096401215,-0.1575528085231781,0.06637421995401382,0.033430859446525574,-0.1288910210132599,0.20759953558444977,-0.25932684540748596,0.2773500978946686,-0.25932684540748596,0.20759953558444977,-0.1288910210132599,0.033430859446525574,0.06637421995401382,-0.1575528085231781,0.22825466096401215,-0.2692908048629761,0.2753278911113739,-0.24558131396770477,0.18391713500022888,-0.09834969788789749],[-0.06637421995401382,0.1288910210132599,-0.18391713500022888,0.22825466096401215,-0.25932684540748596,0.2753278911113739,-0.2753278911113739,0.25932684540748596,-0.22825466096401215,0.18391713500022888,-0.1288910210132599,0.06637421995401382,6.798085926082532e-17,-0.06637421995401382,0.1288910210132599,-0.18391713500022888,0.22825466096401215,-0.25932684540748596,0.2753278911113739,-0.2753278911113739,0.25932684540748596,-0.22825466096401215,0.18391713500022888,-0.1288910210132599,0.06637421995401382],[0.033430859446525574,-0.06637421995401382,0.09834969788789749,-0.1288910210132599,0.1575528085231781,-0.18391713500022888,0.20759953558444977,-0.22825466096401215,0.24558131396770477,-0.25932684540748596,0.2692908048629761,-0.2753278911113739,0.2773500978946686,-0.2753278911113739,0.2692908048629761,-0.25932684540748596,0.24558131396770477,-0.22825466096401215,0.20759953558444977,-0.18391713500022888,0.1575528085231781,-0.1288910210132599,0.09834969788789749,-0.06637421995401382,0.033430859446525574]]

 RLambda = [4.6419172286987305,15.514562606811523,33.45938491821289,58.21471405029297,89.4195556640625,126.61888122558594,169.27023315429688,216.7516632080078,268.37078857421875,323.3748474121094,380.9618225097656,440.29193115234375,500.5,560.7080688476562,620.0381469726562,677.6251220703125,732.6292114257812,784.2483520507812,831.7297973632812,874.381103515625,911.5804443359375,942.7852783203125,967.5405883789062,985.4854125976562,996.3580932617188]

 var alpha = 0.003
 var beta = 0.8
 var b = zeros(25); b[0] = 249.75
 var iter = geniterMomentum(RQ, RLambda, b, alpha, beta).iter
 var res = function(i) { return iter(i)[0] }

 var sampleSVG = d3.select("#iterates")
 .style("display", "block")
 .append("svg")
 .attr("width", 770)
 .attr("height", 700)

 var rosen = d3.scaleLinear().domain([0,0.2,0.5,0.6,0.8]).range(colorbrewer.YlGnBu[5]);
 var jet = d3.scaleLinear().domain([-0,0.1,0.2,0.5,1.0,8]).range(colorbrewer.YlGnBu[5]);
 var contrast = d3.scaleLinear().domain([-12,0,12]).range(colorbrewer.YlGnBu[3]);

 var xstar = iter(10000000)
 var numIters = 52

 var pathstr = "M 8,0"

 for (var j = 0; j < 27; j++) {
 if (j != 26 && j > 0) {
 pathstr = pathstr + "L " + j\*8 + "," + (j-1)\*8 + "L " + j\*8 + "," + j\*8
 }
 if (j == 0) {
 //pathstr = pathstr + "L " + j\*8 + "," + j\*8
 }
 if (j == 26) {
 pathstr = pathstr + "L " + j\*8 + "," + (j-1)\*8
 }
 }
 pathstr = pathstr + "L "+ 26\*8 + ",0 L 8,0 "

 var Disps3 = []
 var errorPlot = sampleSVG.append("g").attr("transform", "translate(" + 440 + ",0)")
 for (var j = 0; j < numIters; j++) {
 var disp = errorPlot.append("g")

 var denter = disp.selectAll("rect")
 .data(iter(j)[1])
 .enter()

 denter.append("rect")
 .style("fill", function(d,i) { return rosen(d) })
 .attr("height", 7.7)
 .attr("width", 7.7)
 .attr("x", function(d, i){return i\*8+ 10})
 .attr("y", function(d, i){return j\*8 })
 Disps3.push(disp)
 }

 errorPlot.append("g").append("path").attr("d", pathstr).style("stroke","black").style("fill","none").style("stroke-width","1px").attr("transform", "translate(2,0.5)")

 var Disps4 = []
 var errorPlot = sampleSVG.append("g").attr("transform", "translate(" + 220 + ",0)")
 for (var j = 0; j < numIters; j++) {
 var disp = errorPlot.append("g")

 var denter = disp.selectAll("rect")
 .data(iter(j)[1])
 .enter()

 denter.append("rect")
 .style("fill", function(d,i) { return jet(d) })
 .attr("height", 7.7)
 .attr("width", 7.7)
 .attr("x", function(d, i){return i\*8+ 10})
 .attr("y", function(d, i){return j\*8 })

 if ((j % 10 == 0) || j == 0) {
 disp.append("text")
 .attr("class", "figtext")
 .attr("text-anchor", "end")
 .attr("x", 0)
 .attr("y", function(d, i){return (j\*8 + 10) })
 .html( (j == 0) ? "Iteration 0" : "" + j)
 }

 Disps4.push(disp)
 }

 errorPlot.append("g").append("path").attr("d", pathstr).style("stroke","black").style("fill","none").style("stroke-width","1px").attr("transform", "translate(2,0.5)")

 function update(alpha, beta) {
 var iter = geniterMomentum(RQ, RLambda, b, Math.max(alpha, 0.00001), Math.min(Math.max(beta, 0.00001), 0.999)).iter
 for (var j = 0; j < numIters; j++) {
 var iterj = iter(j)
 Disps3[j].selectAll("rect").data(iterj[1]).merge(Disps3[j]).style("fill", function(d,i) { return rosen(Math.abs(d - xstar[1][i]) ) })
 Disps4[j].selectAll("rect").data(iterj[1]).merge(Disps4[j]).style("fill", function(d,i) { return jet(d) })

 }
 }

 var reset = slider2D(d3.select("#sliderz"), function(x,y) { update(x/RLambda[RLambda.length - 1],y) }, 4.6, 996.35)

 update(0.0035749243028120182, 0.8095238095238095)

 colorMap( d3.select("#rosen\_colorbar1"),
 194,
 rosen,
 d3.scaleLinear().domain([0,1]).range([0, 194]) )

 colorMap( d3.select("#rosen\_colorbar2"),
 194,
 jet,
 d3.scaleLinear().domain([0,1]).range([0, 194]) );

 // Swoopy Annotator
 var annotations = [
 {
 "x": 0,
 "y": 0,
 "path": "M 817,95 A 23.869 23.869 0 0 0 792,72",
 "text": "Optimal parameters",
 "textOffset": [
 767,
 109
 ]
 }
 ]
 var sel = drawAnnotations(d3.select("#rosenViz"), annotations)
 sel.selectAll("text").style("cursor", "pointer").style("pointer-events","all").on("click", reset)

 callback(null);
 });

 The observations made in the above diagram are true for any Linear First Order algorithm. Let us prove this. First observe that each component of the gradient depends only on the values directly before and after it:

dt-math[block] {
 display: block;
}

∇f(x)i=2wi−wi−1−wi+1+4κ−1wi,i≠1.
\nabla f(x)\_{i}=2w\_{i}-w\_{i-1}-w\_{i+1} +\frac{4}{\kappa-1} w\_{i}, \qquad i \neq 1.
 ∇f(x)i=2wi−wi−1−wi+1+κ−14wi,i≠1.

 Therefore the fact we start at 0 guarantees that that component must remain stoically there till an element either before or after it turns nonzero. And therefore, by induction, for any linear first order algorithm,

dt-math[block] {
 display: block;
}

w0=[  0,0,0,…0,0,…0 ]w1=[ w11,0,0,…0,0,…0 ]w2=[ w12,w22,0,…0,0,…0 ] ⋮wk=[ w1k,w2k,w3k,…wkk,0,…0 ].
 \begin{array}{lllllllll}
 w^{0} & = & [~~0, & 0, & 0, & \ldots & 0, & 0, & \ldots & 0~]\\[0.35em]
 w^{1} & = & [~w\_{1}^{1}, & 0, & 0, & \ldots & 0, & 0, & \ldots & 0~]\\[0.35em]
 w^{2} & = & [~w\_{1}^{2}, & w\_{2}^{2}, & 0, & \ldots & 0, & 0, & \ldots & 0~]\\[0.35em]
 & ~ \vdots \\
 w^{k} & = & [~w\_{1}^{k}, & w\_{2}^{k}, & w\_{3}^{k}, & \ldots & w\_{k}^{k}, & 0, & \ldots & 0~].\\
 \end{array}
 w0w1w2wk=== ⋮=[  0,[ w11,[ w12,[ w1k,0,0,w22,w2k,0,0,0,w3k,…………0,0,0,wkk,0,0,0,0,…………0 ]0 ]0 ]0 ].

 Think of this restriction as a “speed of light” of information transfer. Error signals will take at least

dt-math[block] {
 display: block;
}

kkk steps to move from

dt-math[block] {
 display: block;
}

w0w\_0w0 to

dt-math[block] {
 display: block;
}

wkw\_kwk. We can therefore sum up the errors which cannot have changed yet9:

dt-math[block] {
 display: block;
}

∥wk−w⋆∥∞≥maxi≥k+1{∣wi⋆∣}=(κ−1κ+1)k+1=(κ−1κ+1)k∥w0−w⋆∥∞.
 \begin{aligned}
\|w^{k}-w^{\star}\|\_{\infty}&\geq\max\_{i\geq k+1}\{|w\_{i}^{\star}|\}\\[0.9em]&=\left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^{k+1}\\[0.9em]&=\left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^{k}\|w^{0}-w^{\star}\|\_{\infty}.
 \end{aligned}
 ∥wk−w⋆∥∞≥i≥k+1max{∣wi⋆∣}=(√κ+1√κ−1)k+1=(√κ+1√κ−1)k∥w0−w⋆∥∞.

 As

dt-math[block] {
 display: block;
}

nnn gets large, the condition number of

dt-math[block] {
 display: block;
}

fnf^nfn approaches

dt-math[block] {
 display: block;
}

κ\kappaκ. And the gap therefore closes; the convergence rate that momentum promises matches the best any linear first order algorithm can do. And we arrive at the disappointing conclusion that on this problem, we cannot do better.

 Like many such lower bounds, this result must not be taken literally, but spiritually. It, perhaps, gives a sense of closure and finality to our investigation. But this is not the final word on first order optimization. This lower bound does not preclude the possibility, for example, of reformulating the problem to change the condition number itself! There is still much room for speedups, if you understand the right places to look.

Momentum with Stochastic Gradients
----------------------------------

 There is a final point worth addressing. All the discussion above assumes access to the true gradient — a luxury seldom afforded in modern machine learning. Computing the exact gradient requires a full pass over all the data, the cost of which can be prohibitively expensive. Instead, randomized approximations of the gradient, like minibatch sampling, are often used as a plug-in replacement of

dt-math[block] {
 display: block;
}

∇f(w)\nabla f(w)∇f(w). We can write the approximation in two parts,

dt-math[block] {
 display: block;
}

∇f(w)\nabla f(w)∇f(w)

 the true gradient

dt-math[block] {
 display: block;
}

+++

dt-math[block] {
 display: block;
}

error(w).\text{error}(w).error(w).

 and an approximation error.
If the estimator is unbiased e.g.

dt-math[block] {
 display: block;
}

E[error(w)]=0\mathbf{E}[\text{error}(w)] = 0E[error(w)]=0

 renderMath(document.getElementById("truegradientpluserror"))

 It is helpful to think of our approximate gradient as the injection of a special kind of noise into our iteration. And using the machinery developed in the previous sections, we can deal with this extra term directly. On a quadratic, the error term cleaves cleanly into a separate term, where 10

dt-math[block] {
 display: block;
}

(yikxik) \left(\begin{array}{c}
 y\_{i}^{k}\\
 x\_{i}^{k}
 \end{array}\right)(yikxik)

 the noisy iterates are a sum of

dt-math[block] {
 display: block;
}

===

dt-math[block] {
 display: block;
}

Rk(yi0xi0)R^{k}\left(\begin{array}{c}
 y\_{i}^{0}\\
 x\_{i}^{0}
 \end{array}\right)Rk(yi0xi0)

 the noiseless, deterministic iterates and

dt-math[block] {
 display: block;
}

+++

dt-math[block] {
 display: block;
}

ϵik∑j=1kRk−j(1−α)\epsilon^k\_i \sum\_{j=1}^{k}R^{k-j}\left(\begin{array}{c}
1\\
-\alpha
\end{array}\right)ϵikj=1∑kRk−j(1−α)

 a decaying sum of the errors, where

dt-math[block] {
 display: block;
}

ϵk=Q⋅error(wk)\epsilon^k = Q \cdot \text{error}(w^k)ϵk=Q⋅error(wk).

 renderMath(document.getElementById("iteratespluserror"))

 The error term,

dt-math[block] {
 display: block;
}

ϵk\epsilon^kϵk, with its dependence on the

dt-math[block] {
 display: block;
}

wkw^kwk, is a fairly hairy object. Following [10], we model this as independent 0-mean Gaussian noise. In this simplified model, the objective also breaks into two separable components, a sum of a deterministic error and a stochastic error
 11, visualized here.

We decompose the expected value of the objective value

dt-math[block] {
 display: block;
}

Ef(w)−f(w⋆)\mathbf{E} f(w) - f(w^\star)Ef(w)−f(w⋆) into a deterministic part  and a stochastic part .

dt-math[block] {
 display: block;
}

Ef(w)−f(w⋆)\mathbf{E} f(w) - f(w^\star) Ef(w)−f(w⋆)
 The small black dots are a single run of stochastic gradient
Step-size α =
Momentum β =
As [1] observes, the optimization has two phases. In the initial transient phase the magnitude of the noise is smaller than the magnitude of the gradient, and Momentum still makes good progress. In the second, stochastic phase, the noise overwhelms the gradient, and momentum is less effective.

 deleteQueue.push(renderLoading(d3.select("#stochastic")))
 renderQueue.push(function(callback) {
 var graphDiv = d3.select("#stochasticgd")
 .style("width", 920 + "px")
 .style("height", 300 + "px")
 .style("top", "170px")
 .style("position", "relative")
 .style("margin-left", "auto")
 .style("margin-right", "auto")
 .attr("width", 920)
 .attr("height", 500)

 var svg = graphDiv.append("svg").attr("width", 940).attr("height", 500)

 var update = renderStochasticMilestones(svg, function() {})
 var slidera = slider2D(d3.select("#sliderStep2DD"), function(x,y) { update(x,y) }, 1, 100)
 slidera(0.5,0.5)
 });

 Note that there are a set of unfortunate tradeoffs which seem to pit the two components of error against each other. Lowering the step-size, for example, decreases the stochastic error, but also slows down the rate of convergence. And increasing momentum, contrary to popular belief, causes the errors to compound. Despite these undesirable properties, stochastic gradient descent with momentum has still been shown to have competitive performance on neural networks. As [1] has observed, the transient phase seems to matter more than the fine-tuning phase in machine learning. And in fact, it has been recently suggested [12] that this noise is a good thing — it acts as a implicit regularizer, which, like early stopping, prevents overfitting in the fine-tuning phase of optimization.

---

Onwards and Downwards
---------------------

 The study of acceleration is seeing a small revival within the optimization community. If the ideas in this article excite you, you may wish to read [13], which fully explores the idea of momentum as the discretization of a certain differential equation. But other, less physical, interpretations exist. There is an algebraic interpretation of momentum in terms of approximating polynomials [3, 14]. Geometric interpretations are emerging [15, 16], connecting momentum to older methods, like the Ellipsoid method. And finally, there are interpretations relating momentum to duality [17], perhaps providing a clue as how to accelerate second order methods and Quasi Newton (for a first step, see [18]). But like the proverbial blind men feeling an elephant, momentum seems like something bigger than the sum of its parts. One day, hopefully soon, the many perspectives will converge into a satisfying whole.

Why Momentum Really Works

Summary: We often think of optimization with momentum as a ball rolling down a hill. This isn't wrong, but there is much more to the story.


 .mountain-illustration {
 margin-top: 20px;
 margin-bottom: 10px;
 width: 180%;
 margin-left: -40%;
 }
 @media (min-width: 768px) {
 .mountain-illustration {
 max-width: 1200px;
 margin-left: auto;
 width: 100%;
 }
 }
 dt-article > h1:first-of-type {
 margin-top: 0;
 }

Achieving a research-level understanding of most topics is like climbing a mountain. Aspiring researchers must struggle to understand vast bodies of work that came before them, to learn techniques, and to gain intuition. Upon reaching the top, the new researcher begins doing novel work, throwing new stones onto the top of the mountain and making it a little taller for whoever comes next.

Mathematics is a striking example of this. For centuries, countless minds have climbed the mountain range of mathematics and laid new boulders at the top. Over time, different peaks formed, built on top of particularly beautiful results. Now the peaks of mathematics are so numerous and steep that no person can climb them all. Even with a lifetime of dedicated effort, a mathematician may only enjoy some of their vistas.

People expect the climb to be hard. It reflects the tremendous progress and cumulative effort that’s gone into mathematics. The climb is seen as an intellectual pilgrimage, the labor a rite of passage. But the climb could be massively easier. It’s entirely possible to build paths and staircases into these mountains.That is, really outstanding tutorials, reviews, textbooks, and so on. The climb isn’t something to be proud of.

The climb isn’t progress: the climb is a mountain of debt.

The Debt
--------

Programmers talk about technical debt: there are ways to write software that are faster in the short run but problematic in the long run. Managers talk about institutional debt: institutions can grow quickly at the cost of bad practices creeping in. Both are easy to accumulate but hard to get rid of.

Research can also have debt. It comes in several forms:

* **Poor Exposition** – Often, there is no good explanation of important ideas and one has to struggle to understand them. This problem is so pervasive that we take it for granted and don’t appreciate how much better things could be.
* **Undigested Ideas** – Most ideas start off rough and hard to understand. They become radically easier as we polish them, developing the right analogies, language, and ways of thinking.
* **Bad abstractions and notation** – Abstractions and notation are the user interface of research, shaping how we think and communicate. Unfortunately, we often get stuck with the first formalisms to develop even when they’re bad. For example, an object with extra electrons is negative, and pi is wrong.

* **Noise** – Being a researcher is like standing in the middle of a construction site. Countless papers scream for your attention and there’s no easy way to filter or summarize them.Because most work is explained poorly, it takes a lot of energy to understand each piece of work. For many papers, one wants a simple one sentence explanation of it, but needs to fight with it to get that sentence. Because the simplest way to get the attention of interested parties is to get everyone’s attention, we get flooded with work. Because we incentivize people being “prolific,” we get flooded with a lot of work… We think noise is the main way experts experience research debt.

The insidious thing about research debt is that it’s normal. Everyone takes it for granted, and doesn’t realize that things could be different. For example, it’s normal to give very mediocre explanations of research, and people perceive that to be the ceiling of explanation quality. On the rare occasions that truly excellent explanations come along, people see them as one-off miracles rather than a sign that we could systematically be doing better.

Interpretive Labor
------------------

There’s a tradeoff between the energy put into explaining an idea, and the energy needed to understand it. On one extreme, the explainer can painstakingly craft a beautiful explanation, leading their audience to understanding without even realizing it could have been difficult. On the other extreme, the explainer can do the absolute minimum and abandon their audience to struggle. This energy is called [interpretive labor](https://acesounderglass.com/2015/06/09/interpretive-labor/) .

Many explanations are not one-to-one. People give lectures, write books, or communicate online. In these one-to-many cases, each member of the audience pays the cost of understanding, even though the cost of explaining stays the same.More formally, if *N* people are trying to understand each other, it takes each one *O(1)* effort to write an explanation of their ideas but *O(N)* effort to understand each of the other *N-1* people’s ideas. The result is that energy cost looks like *O(a + bN)* where *a* and *b* are coefficients for the trade off between energy on the explanation side and energy on the understanding side. That is a is the energy spent on explaining and b is the corresponding effort needed to understand. This is similar to ideas in The Mythical Man-Month As a result, the cost of understanding has a multiplier in the interpretive labor tradeoff — sometimes a huge multiplier.For example, Christopher’s average blog post is read by over 100,000 people; if he can save each reader just one second, he’s saved humanity 30 hours.

![](assets/publish-one-many-crop.jpg)

In research, we often have a group of researchers all trying to understand each other. Just like before, the cost of explaining stays constant as the group grows, but the cost of understanding increases with each new member. At some size, the effort to understand everyone else becomes too much. As a defense mechanism, people specialize, focusing on a narrower area of interest. The maintainable size of the field is controlled by how its members trade off the energy between communicating and understanding.

Research debt is the accumulation of missing interpretive labor. It’s extremely natural for young ideas to go through a stage of debt, like early prototypes in engineering. The problem is that we often stop at that point. Young ideas aren’t ending points for us to put in a paper and abandon. When we let things stop there the debt piles up. It becomes harder to understand and build on each other’s work and the field fragments.

Clear Thinking
--------------

It’s worth being clear that research debt isn’t just about ideas not being explained well. It’s a lack of digesting ideas – or, at least, a lack of the public version of ideas being digested.Often, some individuals have a much more developed version of an idea than is publicly shared. There are a lot of reasons for not sharing it (in particular, they’re often not traditionally publishable). It’s a communal messiness of thought.

Developing good abstractions, notations, visualizations, and so forth, is improving the user interfaces for ideas. This helps both with understanding ideas for the first time and with thinking clearly about them. Conversely, if we can’t explain an idea well, that’s often a sign that we don’t understand it as well as we could.

It shouldn’t be that surprising that these two largely go hand in hand. Part of thinking is having a conversation with ourselves.

Research Distillation
---------------------

Research distillation is the opposite of research debt. It can be incredibly satisfying, combining deep scientific understanding, empathy, and design to do justice to our research and lay bare beautiful insights.

Distillation is also hard. It’s tempting to think of explaining an idea as just putting a layer of polish on it, but good explanations often involve transforming the idea. This kind of refinement of an idea can take just as much effort and deep understanding as the initial discovery.

This leaves us with no easy way out. We can’t solve research debt by having one person write a textbook: their energy is spread too thin to polish every idea from scratch. We can’t outsource distillation to less skilled non-experts: refining and explaining ideas requires creativity and deep understanding, just as much as novel research.

Research distillation doesn’t have to be you, but it does have to be us.

Where are the Distillers?
-------------------------

Like the theoretician, the experimentalist or the research engineer, the research distiller is an integral role for a healthy research community. Right now, almost no one is filling it.

Why do researchers not work on distillation? One possibility is perverse incentives, like wanting your work to look difficult. Those certainly exist, but we don’t think they’re the main factor.
 There are a lot of perverse incentives that push against explaining things well, sharing data, and so forth. This is especially true when the work you are doing isn’t that interesting or isn’t reproducible and you want to obscure that. Or if you have a lot of competitors and don’t want them to catch up.
However, our experience is that most good researchers don’t seem that motivated by these kind of factors. Instead, the main issue is that it isn’t worthwhile for them to divert energy from pursuing results to distill things. Perhaps things are different in other fields, or I’m not cynical enough.
 Another possibility is that they don’t enjoy research distillation. Again, we don’t think that’s what’s going on.

Lots of people want to work on research distillation. Unfortunately, it’s very difficult to do so, because we don’t support them.There is a strange kind of informal support for people working on research distillation. Christopher has personally benefitted a great deal from this. But it’s unreliable and not widely advertised, which makes it hard to build a career on.

An aspiring research distiller lacks many things that are easy to take for granted: a career path, places to learn, examples and role models. Underlying this is a deeper issue: their work isn’t seen as a real research contribution. We need to fix this.

An Ecosystem for Distillation
-----------------------------

If you are excited to distill ideas, seek clarity, and build beautiful explanations, we are letting you down. You have something precious to contribute and we aren’t supporting you the way we should.

The [Distill Ecosystem](http://distill.pub/about) is an attempt to better support this kind of work. Right now, it has three parts:

* The Distill Journal – A venue to give traditional validation to non-traditional contributions.
* The Distill Prize – A $10,000 prize to acknowledge outstanding explanations of machine learning.
* Distill Infrastructure – Tools for making beautiful interactive essays.

This is just a start: there’s a lot more that needs to be done. A complete ecosystem for this kind of work needs several other components, including places where one can learn these skills and reliable sources of employment doing this kind of work. We’re optimistic that will come with time.

---

Further Reading
---------------

![](assets/headstand.jpg)

* **Visual Mathematics:**

 Several mathematicians have made remarkable efforts to visually explain certain topics. Needham’s tour-de-force *Visual Complex Analysis* is particularly striking, but there are several lovely examples of new clarity being brought to a topic by visually reformulating it  and, on a smaller scale, [a plethora of visual proofs](http://mathoverflow.net/questions/8846/proofs-without-words).
* **Explorable Explanations:**

 There’s a loose community exploring how the interactive medium enabled by computers can be used to communicate and think in ways previously impossible. These ideas start, as many ideas in computing do, with work done by Douglas Engelbart and Alan Kay.

More recently, [Explorable Explanations](http://explorableexplanations.com/) have started to reimagine what an essay can be in this new medium. This started with Bret Victor’s foundational essay and has been further developed by amazing examples (eg. ).

There are also explorations of how we can augment our ability to think in this new medium, bringing previously inaccessible ideas within reach. Once again, Bret Victor has wonderful ideas , as does Michael Nielsen .
* **Research Distribution:**

 Over last few decades, there’s been a big push for research to be freely available online. This includes the formation of [arXiv.org](https://arxiv.org) and [PLOS](https://www.plos.org/), and [journal editorial boards resigning](http://oad.simmons.edu/oadwiki/Journal_declarations_of_independence) to start open-access journals.

 Increasingly, the challenge is filtering accessible content. Karpathy’s [ArXiv Sanity](http://www.arxiv-sanity.com/) is a lovely tool for this. Crowd curation by online communities also helps a great deal.
* **Open-Notebook Science:**
 Open notebook science, like Dror Bar-Natan’s [Academic Pensieve](http://drorbn.net/AcademicPensieve/), and massively-collaborative research projects like the [Polymath Project](https://polymathprojects.org/), separate the sharing of results from formal publishing.
 This seems really important. Traditionally, if one doesn’t turn research into a paper, it’s basically as though you didn’t do it. This creates a strong incentive for all research to be dressed up as an important paper, increasing noise.
* **Discussion of Debt and Distillation:**
 A number of mathematicians have discussed what we call research distillation. Some great comments and references are collected in this [MathOverflow thread](http://mathoverflow.net/questions/43690/whats-a-mathematician-to-do) — I’d draw particular attention to Thurston’s account of accidentally killing a field by drowning it in research debt in section 6 of .

 Other nice thoughts in this space include the idea of an “open exposition problem” , Grothendieck’s “rising sea” approach to mathematics , and recent calls to value conceptual progress in CS more .

Research Debt

Summary: Science is a human activity. When we fail to distill and explain research, we accumulate a kind of debt...

The Tau Manifesto

Interpretive Labor

A visual explanation of Jensen's inequality

A Research Center for Augmenting Human Intellect

Personal Dynamic Media

Explorable explanations

Visualizing Algorithms

How to Fold a Julia Fractal

Back to the Future of Handwriting Recognition

Media for thinking the unthinkable

Learnable programming

Thought as a Technology

Toward an Exploratory Medium for Mathematics

On Proof and Progress in Mathematics

A beginner’s guide to forcing

Statement on conceptual contributions in theory