Why Momentum Really Works

Why Momentum Really Works

=========================

OptimumSolutionStarting Point

Step-size α = 0.02

00.0030.006

Momentum β = 0.99

<PHONE>.990

 We often think of Momentum as a means of dampening oscillations

and speeding up the iterations, leading to faster convergence. But it

has other interesting behavior. It allows a larger range of step-sizes

to be used, and creates its own oscillations. What is going on?

 dt-byline {

 font-size: 12px;

 line-height: 18px;

 display: block;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 color: rgba(0, 0, 0, 0.5);

 padding-top: 12px;

 padding-bottom: 12px;

 }

 dt-article.centered dt-byline {

 text-align: center;

 }

 dt-byline a,

 dt-article dt-byline a {

 text-decoration: none;

 border-bottom: none;

 }

 dt-article dt-byline a:hover {

 text-decoration: underline;

 border-bottom: none;

 }

 dt-byline .authors {

 text-align: left;

 }

 dt-byline .name {

 display: inline;

 text-transform: uppercase;

 }

 dt-byline .affiliation {

 display: inline;

 }

 dt-byline .date {

 display: block;

 text-align: left;

 }

 dt-byline .year, dt-byline .month {

 display: inline;

 }

 dt-byline .citation {

 display: block;

 text-align: left;

 }

 dt-byline .citation div {

 display: inline;

 }

 @media(min-width: 768px) {

 dt-byline {

 }

 }

 @media(min-width: 1080px) {

 dt-byline {

 border-bottom: none;

 margin-bottom: 70px;

 }

 dt-byline a:hover {

 color: rgba(0, 0, 0, 0.9);

 }

 dt-byline .authors {

 display: inline-block;

 }

 dt-byline .author {

 display: inline-block;

 margin-right: 12px;

 /\*padding-left: 20px;\*/

 /\*border-left: 1px solid #ddd;\*/

 }

 dt-byline .affiliation {

 display: block;

 }

 dt-byline .author:last-child {

 margin-right: 0;

 }

 dt-byline .name {

 display: block;

 }

 dt-byline .date {

 border-left: 1px solid rgba(0, 0, 0, 0.1);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .year, dt-byline .month {

 display: block;

 }

 dt-byline .citation {

 border-left: 1px solid rgba(0, 0, 0, 0.15);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .citation div {

 display: block;

 }

 }

[Gabriel Goh](http://gabgoh.github.io/)

[UC Davis](http://math.ucdavis.edu/)

April. 4

2017

[Citation:

Goh, 2017](#citation)

 // Render Foreground

 .alpha(0.003)

 .beta(0)

 (d3.select("#banana").style("position","relative"))

 var iterChange = iterControl.control

 var getw0 = iterControl.w0

 var StepRange = d3.scaleLinear().domain([0,100]).range([0,0.0062])

 var MomentumRange = d3.scaleLinear().domain([0,100]).range([0,0.98])

 var update = function (i,j) { iterChange(i, 0, getw0()) }

 var slidera = sliderGen([230, 40])

 .ticks([0,0.003,0.006])

 .ticktitles( function(d,i) { return ["0", "0.003", "0.006"][i]})

 .change( function (i) {

 iterChange(getalpha(), getbeta(), getw0() )

 } )

 .startxval(0.003)

 .cRadius(7)

 .shifty(-12)

 .margins(20,20)

 var sliderb = sliderGen([230, 40])

 .ticks([0,0.5,0.99])

 .change( function (i) {

 iterChange(getalpha(), getbeta(), getw0() )

 } )

 .cRadius(7)

 .shifty(-12)

 .startxval(0.74)

 .margins(20,20)

 var getalpha = slidera( d3.select("#sliderAlpha")).xval

 var getbeta = sliderb( d3.select("#sliderBeta")).xval

 iterChange(getalpha(), getbeta(), getw0() )

 Here's a popular story about momentum [1, 2, 3]:

 gradient descent is a man walking down a hill. He follows the steepest

path downwards; his progress is slow, but steady. Momentum is a heavy

ball rolling down the same hill. The added inertia acts both as a

smoother and an accelerator, dampening oscillations and causing us to

barrel through narrow valleys, small humps and local minima.

 This standard story isn't wrong, but it fails to explain many

important behaviors of momentum. In fact, momentum can be understood far

 more precisely if we study it on the right model.

 One nice model is the convex quadratic. This model is rich enough to

reproduce momentum's local dynamics in real problems, and yet simple

enough to be understood in closed form. This balance gives us powerful

traction for understanding this algorithm.

---

 We begin with gradient descent. The algorithm has many virtues, but

speed is not one of them. It is simple — when optimizing a smooth

function

dt-math[block] {

 display: block;

}

fff, we make a small step in the gradient

dt-math[block] {

 display: block;

}

wk+1=wk−α∇f(wk).w^{k+1} = w^k-\alpha\nabla f(w^k).wk+1=wk−α∇f(wk).

 For a step-size small enough, gradient descent makes a monotonic

improvement at every iteration. It always converges, albeit to a local

minimum. And under a few weak curvature conditions it can even get there

 at an exponential rate.

 But the exponential decrease, though appealing in theory, can often be

 infuriatingly small. Things often begin quite well — with an

impressive, almost immediate decrease in the loss. But as the iterations

 progress, things start to slow down. You start to get a nagging feeling

 you're not making as much progress as you should be. What has gone

wrong?

dt-math[block] {

 display: block;

}

fff

 which aren't scaled properly. The landscapes are often described as

valleys, trenches, canals and ravines. The iterates either jump between

valleys, or approach the optimum in small, timid steps. Progress along

certain directions grind to a halt. In these unfortunate regions,

gradient descent fumbles.

dt-math[block] {

 display: block;

}

zk+1=βzk+∇f(wk)wk+1=wk−αzk+1

 \begin{aligned}

 z^{k+1}&=\beta z^{k}+\nabla f(w^{k})\\[0.4em]

 w^{k+1}&=w^{k}-\alpha z^{k+1}

 \end{aligned}

 zk+1wk+1=βzk+∇f(wk)=wk−αzk+1

 The change is innocent, and costs almost nothing. When

dt-math[block] {

 display: block;

}

β=0\beta = 0β=0 , we recover gradient descent. But for

dt-math[block] {

 display: block;

}

β=0.99\beta = 0.99β=0.99 (sometimes

dt-math[block] {

 display: block;

}

0.9990.9990.999,

 if things are really bad), this appears to be the boost we need. Our

iterations regain that speed and boldness it lost, speeding to the

optimum with a renewed energy.

 Optimizers call this minor miracle "acceleration".

 The new algorithm may seem at first glance like a cheap hack. A simple

 trick to get around gradient descent's more aberrant behavior — a

smoother for oscillations between steep canyons. But the truth, if

anything, is the other way round. It is gradient descent which is the

hack. First, momentum gives up to a quadratic speedup on many functions.

 1

 This is no small matter — this is similar to the speedup you get from

the Fast Fourier Transform, Quicksort, and Grover's Algorithm. When the

universe gives you quadratic speedups, you should start to pay

attention.

 But there's more. A lower bound, courtesy of Nesterov [5],

 states that momentum is, in a certain very narrow and technical sense,

optimal. Now, this doesn't mean it is the best algorithm for all

functions in all circumstances. But it does satisfy some curiously

beautiful mathematical properties which scratch a very human itch for

perfection and closure. But more on that later. Let's say this for

now — momentum is an algorithm for the book.

---

First Steps: Gradient Descent

-----------------------------

dt-math[block] {

 display: block;

}

f(w)=12wTAw−bTw,w∈Rn.

 f(w) = \tfrac{1}{2}w^TAw - b^Tw, \qquad w \in \mathbf{R}^n.

 f(w)=21wTAw−bTw,w∈Rn.

 Assume

dt-math[block] {

 display: block;

}

AAA is symmetric and invertible, then the optimal solution

dt-math[block] {

 display: block;

}

w⋆w^{\star}w⋆ occurs at

dt-math[block] {

 display: block;

}

w⋆=A−1b. w^{\star} = A^{-1}b.w⋆=A−1b.

dt-math[block] {

 display: block;

}

 etc) and captures all the key features of pathological curvature. And

more importantly, we can write an exact closed formula for gradient

descent on this function.

 This is how it goes. Since

dt-math[block] {

 display: block;

}

∇f(w)=Aw−b\nabla f(w)=Aw - b∇f(w)=Aw−b, the iterates are

dt-math[block] {

 display: block;

}

wk+1=wk−α(Awk−b).

 w^{k+1}=w^{k}- \alpha (Aw^{k} - b).

 wk+1=wk−α(Awk−b).

 Here's the trick. There is a very natural space to view gradient

descent where all the dimensions act independently — the eigenvectors of

dt-math[block] {

 display: block;

}

AAA.

Optimum

Optimum

 deleteQueue.push(renderLoading(d3.select("#change\_of\_variables")))

 renderQueue.push(function(callback) {

 var U = givens(Math.PI/4)

 var Ut = numeric.transpose(U)

 // Render Foreground

 var left = d3.select("#mom1").style("border", "1px solid rgba(0, 0, 0, 0.2)")

 var c1 = genIterDiagram(quadf, [0,0], [[-3,3],[-3,3]])

 .width(340)

 .height(340)

 .iters(300)

 .alpha(0.018)

 .showSolution(false)

 .pathWidth(1)

 .circleRadius(1.5)

 .pointerScale(0.8)

 .showStartingPoint(false)

 .drag(function() {

 c2.control(c1.alpha(),

 c1.beta(),

 numeric.dot(U,c1.w0())) })

 (left)

 var right = d3.select("#mom2").style("border", "1px solid rgba(0, 0, 0, 0.2)")

 var c2 = genIterDiagram(eyef, [0,0], [[-3,3],[-3,3]])

 .width(340)

 .height(340)

 .iters(300)

 .alpha(0.018)

 .showSolution(false)

 .pathWidth(1)

 .circleRadius(1.5)

 .pointerScale(0.8)

 .showStartingPoint(false)

 .drag(function() {

 c1.control(c2.alpha(),

 c2.beta(),

 numeric.dot(Ut,c2.w0())) })

 (right)

 // Initialize

 c2.control(0.018,0,[-2.5,1])

 c1.control(0.018,0,numeric.dot(Ut,[-2.5,1]));

 callback(null);

});

 Every symmetric matrix

dt-math[block] {

 display: block;

}

AAA has an eigenvalue decomposition

dt-math[block] {

 display: block;

}

A=Q diag(λ1,…,λn) QT,Q=[q1,…,qn],

 A=Q diag(λ1,…,λn) QT,Q=[q1,…,qn],

 and, as per convention, we will assume that the

dt-math[block] {

 display: block;

}

λi\lambda\_iλi's are sorted, from smallest

dt-math[block] {

 display: block;

}

λ1\lambda\_1λ1 to biggest

dt-math[block] {

 display: block;

}

λn\lambda\_nλn. If we perform a change of basis,

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xik+1=xik−αλixik=(1−αλi)xik=(1−αλi)k+1xi0

 \begin{aligned}

 x\_{i}^{k+1} & =x\_{i}^{k}-\alpha \lambda\_ix\_{i}^{k} \\[0.4em]

 &= (1-\alpha\lambda\_i)x^k\_i=(1-\alpha \lambda\_i)^{k+1}x^0\_i

 \end{aligned}

 Moving back to our original space

dt-math[block] {

 display: block;

}

www, we can see that

dt-math[block] {

 display: block;

}

wk−w⋆=Qxk=∑inxi0(1−αλi)kqi

 w^k - w^\star = Qx^k=\sum\_i^n x^0\_i(1-\alpha\lambda\_i)^k q\_i

 wk−w⋆=Qxk=i∑nxi0(1−αλi)kqi

 and there we have it — gradient descent in closed form.

### Decomposing the Error

 The above equation admits a simple interpretation. Each element of

dt-math[block] {

 display: block;

}

x0x^0x0 is the component of the error in the initial guess in the

dt-math[block] {

 display: block;

}

QQQ-basis. There are

dt-math[block] {

 display: block;

}

nnn

 such errors, and each of these errors follows its own, solitary path to

 the minimum, decreasing exponentially with a compounding rate of

dt-math[block] {

 display: block;

}

1−αλi1-\alpha\lambda\_i1−αλi. The closer that number is to

dt-math[block] {

 display: block;

}

111, the slower it converges.

 For most step-sizes, the eigenvectors with largest eigenvalues

converge the fastest. This triggers an explosion of progress in the

first few iterations, before things slow down as the smaller

eigenvectors' struggles are revealed. By writing the contributions of

each eigenspace's error to the loss

dt-math[block] {

 display: block;

}

f(wk)−f(w⋆)=∑(1−αλi)2kλi[xi0]2

 f(w^{k})-f(w^{\star})=\sum(1-\alpha\lambda\_{i})^{2k}\lambda\_{i}[x\_{i}^{0}]^2

 f(wk)−f(w⋆)=∑(1−αλi)2kλi[xi0]2

 we can visualize the contributions of each error component to the loss.

dt-math[block] {

 display: block;

}

λ1=0.01\lambda\_1=0.01λ1=0.01,

dt-math[block] {

 display: block;

}

λ2=0.1\lambda\_2=0.1λ2=0.1, and

dt-math[block] {

 display: block;

}

λ3=1\lambda\_3=1λ3=1 respectively.

 Step-size

 Optimal Step-size

012

 deleteQueue.push(renderLoading(d3.select("#milestones\_gd")))

 renderQueue.push(function(callback) {

 var graphDiv = d3.select("#obj")

 .style("width", 920 + "px")

 .style("height", 300 + "px")

 .style("top", "90px")

 .style("position", "relative")

 .style("margin-left", "auto")

 .style("margin-right", "auto")

 .attr("width", 920)

 .attr("height", 500)

 var svg = graphDiv.append("svg")

 .attr("width", 920)

 .attr("height", 300)

 .style("position","absolute")

 .style("left", "15px")

 var updateSliderGD = renderMilestones(svg, function() {});

 var alphaHTML = MathCache("alpha-equals");

 var slidera = sliderGen([250, 80])

 .ticks([0,1,200/(101),2])

 .change( function (i) {

 d3.select("#stepSizeMilestones")

 .html("Stepsize " + html )

 updateSliderGD(i,0.000)

 } )

 .ticktitles(function(d,i) { return [0,1,"",2][i] })

 .startxval(200/(101))

 .cRadius(7)

 .shifty(-12)

 .shifty(10)

 .margins(20,20)(d3.select("#sliderStep"))

 // renderDraggable(svg, [133.5, 23], [114.5, 90], 2, " ").attr("opacity", 0.1)

 // renderDraggable(svg, [133.5, 88], [115.5, 95], 2, " ").attr("opacity", 0.1)

 d3.select("#milestones\_gd\_optstep").on("click", slidera.init)

 svg.append("text")

 .attr("class", "katex morsd mathit")

 .style("font-size", "19px")

 .style("font-family","KaTeX\_Math")

 .attr("x", 105)

 .attr("y", 50)

 .attr("text-anchor", "end")

 .attr("fill", "gray")

 svg.append("text")

 .style("font-size", "13px")

 .attr("x", 0)

 .attr("y", 80)

 .attr("dy", 0)

 .attr("transform", "translate(110,0)")

 .attr("class", "caption")

 .attr("text-anchor", "end")

 .attr("fill", "gray")

 .text("At the initial point, the error in each component is equal.")

 svg.selectAll(".caption").call(wrap, 100)

 svg.append("text")

 .style("font-size", "13px")

 .attr("x", 420)

 .attr("y", 270)

 .attr("dy", 0)

 .attr("dx", -295)

 .attr("text-anchor", "start")

 .attr("fill", "gray")

 callback(null);

 });

### Choosing A Step-size

 The above analysis gives us immediate guidance as to how to set a step-size

dt-math[block] {

 display: block;

}

α\alphaα. In order to converge, each

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

0<αλi<2.0<\alpha\lambda\_i<2.0<αλi<2.

dt-math[block] {

 display: block;

}

λ1\lambda\_1λ1 or

dt-math[block] {

 display: block;

}

λn\lambda\_nλn:

dt-math[block] {

 display: block;

}

rate(α) = maxi∣1−αλi∣ = max{∣1−αλ1∣, ∣1−αλn∣}

 \begin{aligned}\text{rate}(\alpha) & ~=~

\max\_{i}\left|1-\alpha\lambda\_{i}\right|\\[0.9em] & ~=~

\max\left\{|1-\alpha\lambda\_{1}|,~ |1-\alpha\lambda\_{n}|\right\}

\end{aligned}

 rate(α) = imax∣1−αλi∣ = max{∣1−αλ1∣, ∣1−αλn∣}

 This overall rate is minimized when the rates for

dt-math[block] {

 display: block;

}

λ1\lambda\_1λ1 and

dt-math[block] {

 display: block;

}

λn\lambda\_nλn

 are the same — this mirrors our informal observation in the previous

section that the optimal step-size causes the first and last

eigenvectors to converge at the same rate. If we work this through we

get:

dt-math[block] {

 display: block;

}

optimal α = argminα rate(α) = 2λ1+λnoptimal rate = minα rate(α) = λn/λ1−1λn/λ1+1

 \begin{aligned}

 \text{optimal }\alpha ~=~{\mathop{\text{argmin}}\limits\_\alpha}

~\text{rate}(\alpha) & ~=~\frac{2}{\lambda\_{1}+\lambda\_{n}}\\[1.4em]

 \text{optimal rate} ~=~{\min\_\alpha} ~\text{rate}(\alpha) &

~=~\frac{\lambda\_{n}/\lambda\_{1}-1}{\lambda\_{n}/\lambda\_{1}+1}

 \end{aligned}

 Notice the ratio

dt-math[block] {

 display: block;

}

λn/λ1\lambda\_n/\lambda\_1λn/λ1

 determines the convergence rate of the problem. In fact, this ratio

appears often enough that we give it a name, and a symbol — the

condition number.

dt-math[block] {

 display: block;

}

condition number:=κ:=λnλ1

 \text{condition number} := \kappa :=\frac{\lambda\_n}{\lambda\_1}

 condition number:=κ:=λ1λn

dt-math[block] {

 display: block;

}

A−1bA^{-1}bA−1b is to perturbations in

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

κ=1\kappa = 1κ=1

 is ideal, giving convergence in one step (of course, the function is

trivial). Unfortunately the larger the ratio, the slower gradient

descent will be. The condition number is therefore a direct measure of

pathological curvature.

---

Example: Polynomial Regression

------------------------------

dt-math[block] {

 display: block;

}

nnn to be exact, one for each of the eigenvectors of

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

 Lets see how this plays out in polynomial regression. Given 1D data,

dt-math[block] {

 display: block;

}

ξi\xi\_iξi, our problem is to fit the model

dt-math[block] {

 display: block;

}

model(ξ)=w1p1(ξ)+⋯+wnpn(ξ)pi=ξ↦ξi−1

 model(ξ)=w1p1(ξ)+⋯+wnpn(ξ)pi=ξ↦ξi−1

 to our observations,

dt-math[block] {

 display: block;

}

did\_idi. This model, though nonlinear in the input

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

0p10 p\_1-2.00p1

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_1-2.00p2

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_12.00p3

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_12.00p4

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_12.00p5

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p10 p\_1-2.00p6

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

model\text{model}model0+0+0+0+0+0=0

scrub values

 deleteQueue.push(renderLoading(d3.select("#poly0")))

 renderQueue.push(function(callback) {

 // Preprocess x, get eigendecomposition, etc

 var x = [-0.6, -0.55,-0.5,-0.45,-0.4,0.4,0.45,0.5,0.55,0.6]

 var D = vandermonde(x, 5)

 var Eigs = eigSym(numeric.dot(numeric.transpose(D),D))

 var U = Eigs.U

 var lambda = Eigs.lambda

 // Preprocess y

 var b = [-3/2,-4/2,-5/2,-3/2,-2/2,1/2,2/2,3/2,2/2,1/2]

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var step = 1.8/lambda[0]

 var iter = geniter(U, lambda, Dtb, step)

 var eigensum = d3.select("#poly0")

 var wi = [-2,-2,2,2,2,-2]

 function refit(b) {

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var Utsol = numeric.dot(sol,U)

 eigenControl.updateweights(Utsol)

 }

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 60,5 A 19.018 19.018 0 0 0 36,27",

 "text": "scrub values",

 "textOffset": [

 64,

 9

 ]

 }

 ]

 drawAnnotations(d3.select("#poly0f"), annotations)

 callback(null);

 });

 Because of the linearity, we can fit this model to our data

dt-math[block] {

 display: block;

}

ξi\xi\_iξi using linear regression on the model mismatch

dt-math[block] {

 display: block;

}

minimizew12∑i(model(ξi)−di)2  =  12∥Zw−d∥2

 minimizew21i∑(model(ξi)−di)2  =  21∥Zw−d∥2

 where

dt-math[block] {

 display: block;

}

Z=(1ξ1ξ12…ξ1n−11ξ2ξ22…ξ2n−1⋮⋮⋮⋱⋮1ξmξm2…ξmn−1).

 Z=\left(\begin{array}{ccccc}

 1 & \xi\_{1} & \xi\_{1}^{2} & \ldots & \xi\_{1}^{n-1}\\

 1 & \xi\_{2} & \xi\_{2}^{2} & \ldots & \xi\_{2}^{n-1}\\

 \vdots & \vdots & \vdots & \ddots & \vdots\\

 1 & \xi\_{m} & \xi\_{m}^{2} & \ldots & \xi\_{m}^{n-1}

 \end{array}\right).

dt-math[block] {

 display: block;

}

QQQ (the eigenvectors of

dt-math[block] {

 display: block;

}

ZTZZ^T ZZTZ). So let's recast our regression problem in the basis of

dt-math[block] {

 display: block;

}

QQQ. First, we do a change of basis, by rotating

dt-math[block] {

 display: block;

}

www into

dt-math[block] {

 display: block;

}

QwQwQw, and counter-rotating our feature maps

dt-math[block] {

 display: block;

}

ppp into eigenspace,

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

model(ξ) = x1p¯1(ξ) + ⋯ + xnp¯n(ξ)p¯i=∑qijpj.

 model(ξ) = x1p¯1(ξ) + ⋯ + xnp¯n(ξ)p¯i=∑qijpj.

 This model is identical to the old one. But these new features

dt-math[block] {

 display: block;

}

p¯\bar{p}p¯

 (which I call "eigenfeatures") and weights have the pleasing property

that each coordinate acts independently of the others. Now our

optimization problem breaks down, really, into

dt-math[block] {

 display: block;

}

nnn

 small 1D optimization problems. And each coordinate can be optimized

greedily and independently, one at a time in any order, to produce the

final, global, optimum. The eigenfeatures are also much more

informative:

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_10.458p¯1

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_10.903p¯2

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_16.13p¯3

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_112.4p¯4

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1128p¯5

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1261p¯6

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

 deleteQueue.push(renderLoading(d3.select("#poly1")))

 renderQueue.push(function(callback) {

 var inv = function(lambda) { return 1/lambda }

 // Preprocess x, get eigendecomposition, etc

 var x = [-0.6, -0.55,-0.5,-0.45,-0.4,0.4,0.45,0.5,0.55,0.6]

 var D = vandermonde(x, 5)

 var Eigs = eigSym(numeric.dot(numeric.transpose(D),D))

 var U = Eigs.U

 var lambda = Eigs.lambda

 // Preprocess y

 var b = [-3/2,-4/2,-5/2,-3/2,-2/2,1/2,2/2,3/2,2/2,1/2]

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var step = 1.8/lambda[0]

 var iter = geniter(U, lambda, Dtb, step)

 var eigensum = d3.select("#poly1")

 var wi = lambda.slice(0).map(scal)

 function refit(b) {

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var Utsol = numeric.dot(sol,U)

 eigenControl.updateweights(sol)

 }

 var eigenControl = renderEigenPanel(eigensum, U, x, b, wi, refit, true)

 var annotate = eigensum

 annotate.append("figcaption")

 .style("width", 230 + "px")

 .style("height", 150 + "px")

 .style("left", "0px")

 .style("position", "absolute")

 .style("padding", "10px")

 annotate.append("figcaption")

 .style("width", 230 + "px")

 .style("height", 150 + "px")

 .style("left", "260px")

 .style("position", "absolute")

 .style("padding", "10px")

 annotate.append("figcaption")

 .style("width", 230 + "px")

 .style("height", 150 + "px")

 .style("left", 530 + "px")

 .style("position", "absolute")

 .style("padding", "10px")

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 807,198 A 26.661 26.661 0 0 1 838,159",

 "text": "drag points to fit data",

 "textOffset": [

 799,

 214

 ]

 }]

 drawAnnotations(eigensum, annotations)

 callback(null);

 });

 The observations in the above diagram can be justified mathematically.

 From a statistical point of view, we would like a model which is, in

some sense, robust to noise. Our model cannot possibly be meaningful if

the slightest perturbation to the observations changes the entire model

dramatically. And the eigenfeatures, the principal components of the

data, give us exactly the decomposition we need to sort the features by

its sensitivity to perturbations in

dt-math[block] {

 display: block;

}

did\_idi's.

 The most robust components appear in the front (with the largest

eigenvalues), and the most sensitive components in the back (with the

smallest eigenvalues).

 This measure of robustness, by a rather convenient coincidence, is

also a measure of how easily an eigenspace converges. And thus, the

"pathological directions" — the eigenspaces which converge the

slowest — are also those which are most sensitive to noise! So starting

at a simple initial point like

dt-math[block] {

 display: block;

}

000

 (by a gross abuse of language, let's think of this as a prior), we

track the iterates till a desired level of complexity is reached. Let's

see how this plays out in gradient descent.

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-0.389p¯1

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-2.47p¯2

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-0.481p¯3

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-0.540p¯4

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_1-0.0128p¯5

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

0p¯10 \bar{p}\_10.0543p¯6

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

 deleteQueue.push(renderLoading(d3.select("#poly2")))

 renderQueue.push(function(callback) {

 var inv = function(lambda) { return 1/lambda }

 // Preprocess x, get eigendecomposition, etc

 var x = [-0.6, -0.55,-0.5,-0.45,-0.4,0.4,0.45,0.5,0.55,0.6]

 var b = [-3/2,-4/2,-5/2,-3/2,-2/2,1/2,2/2,3/2,2/2,1/2]

 var D = vandermonde(x, 5)

 var Eigs = eigSym(numeric.dot(numeric.transpose(D),D))

 var U = Eigs.U

 var lambda = Eigs.lambda

 // Preprocess y

 var Dtb = numeric.dot(b,D)

 var sol = numeric.mul(numeric.dot(U, Dtb), lambda.map(inv))

 var step = 1.8/lambda[0]

 var iter = geniter(U, lambda, Dtb, step)

 var eigensum = d3.select("#poly2")

 var wi = lambda.slice(0).map(scal)

 function refit(b) {

 var Dtb = numeric.dot(b,D)

 iter = geniter(U, lambda, Dtb, step)

 onChange(sliderControl.slidera.xval())

 }

 var eigenControl = renderEigenPanel(eigensum, U, x, b, wi, refit, true)

 var barlengths = getStepsConvergence(lambda, step).map(Math.log)

 var onChange = function(i) {

 eigenControl.updateweights(numeric.dot(U,iter(Math.floor(Math.exp(i-0.1)) )))

 }

 d3.select("#poly2").append("figcaption")

 .style("width", "120px")

 .style("position", "absolute")

 .style("left", "820px")

 .style("top","200px")

 sliderControl.slidera.init()

 // var figwidth = d3.select("#poly2").style("width")

 // var figheight = d3.select("#poly2").style("height")

 // var svgannotate = d3.select("#poly2")

 // .append("svg")

 // .style("width", figwidth)

 // .style("height", figheight)

 // .style("position", "absolute")

 // .style("top","0px")

 // .style("left","0px")

 // .style("pointer-events","none")

 // renderDraggable(svgannotate,

 // [139.88888549804688, 243.77951049804688],

 // [121.88888549804688, 200.77951049804688],

 // 5,

 // "We begin at x=w=0");

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 74,202 A 52.274 52.274 0 0 0 134,245",

 "text": "We begin at x=w=0",

 "textOffset": [

 21,

 198

 ]

 }

 ]

 drawAnnotations(d3.select("#poly2"), annotations)

 callback(null);

 });

 This effect is harnessed with the heuristic of early stopping : by

stopping the optimization early, you can often get better generalizing

results. Indeed, the effect of early stopping is very similar to that of

 more conventional methods of regularization, such as Tikhonov

Regression. Both methods try to suppress the components of the smallest

eigenvalues directly, though they employ different methods of spectral

decay.2

 But early stopping has a distinct advantage. Once the step-size is

chosen, there are no regularization parameters to fiddle with. Indeed,

in the course of a single optimization, we have the entire family of

models, from underfitted to overfitted, at our disposal. This gift, it

seems, doesn't come at a price. A beautiful free lunch [7] indeed.

---

The Dynamics of Momentum

------------------------

 Let's turn our attention back to momentum. Recall that the momentum update is

dt-math[block] {

 display: block;

}

zk+1=βzk+∇f(wk)wk+1=wk−αzk+1.

 \begin{aligned}

 z^{k+1}&=\beta z^{k}+\nabla f(w^{k})\\[0.4em]

 w^{k+1}&=w^{k}-\alpha z^{k+1}.

 \end{aligned}

 zk+1wk+1=βzk+∇f(wk)=wk−αzk+1.

 Since

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

zk+1=βzk+(Awk−b)wk+1=wk−αzk+1.

 \begin{aligned}

 z^{k+1}&=\beta z^{k}+ (Aw^{k}-b)\\[0.4em]

 w^{k+1}&=w^{k}-\alpha z^{k+1}.

 \end{aligned}

 zk+1wk+1=βzk+(Awk−b)=wk−αzk+1.

 Following [8], we go through the same motions, with the change of basis

dt-math[block] {

 display: block;

}

xk=Q(wk−w⋆)

 x^{k} = Q(w^{k} - w^\star)xk=Q(wk−w⋆) and

dt-math[block] {

 display: block;

}

yk=Qzk y^{k} = Qz^{k}yk=Qzk, to yield the update rule

dt-math[block] {

 display: block;

}

yik+1=βyik+λixikxik+1=xik−αyik+1.

 \begin{aligned}

 y\_{i}^{k+1}&=\beta y\_{i}^{k}+\lambda\_{i}x\_{i}^{k}\\[0.4em]

 x\_{i}^{k+1}&=x\_{i}^{k}-\alpha y\_{i}^{k+1}.

 \end{aligned}

 yik+1xik+1=βyik+λixik=xik−αyik+1.

 in which each component acts independently of the other components (though

dt-math[block] {

 display: block;

}

xikx^k\_ixik and

dt-math[block] {

 display: block;

}

yiky^k\_iyik are coupled). This lets us rewrite our iterates as

 3

dt-math[block] {

 display: block;

}

(yikxik)=Rk(yi0xi0)R=(βλi−αβ1−αλi).

 \left(\!\!\begin{array}{c}

 y\_{i}^{k}\\

 x\_{i}^{k}

 \end{array}\!\!\right)=R^k\left(\!\!\begin{array}{c}

 y\_{i}^{0}\\

 x\_{i}^{0}

 \end{array}\!\!\right)

 \qquad

 R = \left(\!\!\begin{array}{cc}

 \beta & \lambda\_{i}\\

 -\alpha\beta & 1-\alpha\lambda\_{i}

 \end{array}\!\!\right).

 (yikxik)=Rk(yi0xi0)R=(β−αβλi1−αλi).

 There are many ways of taking a matrix to the

dt-math[block] {

 display: block;

}

kthk^{th}kth power. But for the

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

RRR,

dt-math[block] {

 display: block;

}

σ1\sigma\_1σ1 and

dt-math[block] {

 display: block;

}

σ2\sigma\_2σ2.

dt-math[block] {

 display: block;

}

Rk={σ1kR1−σ2kR2σ1≠σ2σ1k(kR/σ1−(k−1)I)σ1=σ2,Rj=R−σjIσ1−σ2

\color{#AAA}{\color{black}{R^{k}}=\begin{cases}

\sigma\_{1}^{k}(kR/\sigma\_1-(k-1)I) & \sigma\_{1}=\sigma\_{2}

\end{cases},\qquad R\_{j}=\frac{R-\sigma\_{j}I}{\sigma\_{1}-\sigma\_{2}}}

 This formula is rather complicated, but the takeaway here is that it

plays the exact same role the individual convergence rates,

dt-math[block] {

 display: block;

}

1−αλi1-\alpha\lambda\_i1−αλi

 do in gradient descent. But instead of one geometric series, we have

two coupled series, which may have real or complex values. The

convergence rate is therefore the slowest of the two rates,

dt-math[block] {

 display: block;

}

max{∣σ1∣,∣σ2∣}\max\{|\sigma\_{1}|,|\sigma\_{2}|\}max{∣σ1∣,∣σ2∣}

4.

 By plotting this out, we see there are distinct regions of the

parameter space which reveal a rich taxonomy of convergence behavior [10]:

s1 =

 s2 =

 complex02410Momentum

dt-math[block] {

 display: block;

}

β=\beta=β=Step-size

dt-math[block] {

 display: block;

}

α=\alpha=α=

0.00.20.40.60.81.01.2

 Convergence Rate

 A plot of

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

2β2\sqrt{\beta}2√β is independent of

dt-math[block] {

 display: block;

}

α\alphaα and

dt-math[block] {

 display: block;

}

λi\lambda\_iλi.

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xik−xi∗x\_i^k - x\_i^\*xik−xi∗**1-Step Convergence**When

dt-math[block] {

 display: block;

}

α=1/λi\alpha = 1/\lambda\_iα=1/λi, and

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xik−xi∗x\_i^k - x\_i^\*xik−xi∗**Monotonic Oscillations**When

dt-math[block] {

 display: block;

}

α>1/λi\alpha > 1/\lambda\_iα>1/λi, the iterates flip between

dt-math[block] {

 display: block;

}

+++ and

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xik−xi∗x\_i^k - x\_i^\*xik−xi∗**Divergence**When

dt-math[block] {

 display: block;

}

 deleteQueue.push(renderLoading(d3.select("#momentum2D")))

 renderQueue.push(function(callback) {

 var defaults = [[0.0015, 0.9],

 [0.0015, 0.125],

 [0.01, 0.00001],

 [0.02, 0.05 ],

 [0.025, 0.235 ]]

 coor = render2DSliderGen(

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[0](xy[0], xy[1],bold)

 updateStemGraphs[0](a,b)

 },

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[1](xy[0], xy[1],bold)

 updateStemGraphs[1](a,b)

 },

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[2](xy[0], xy[1],bold)

 updateStemGraphs[2](a,b)

 },

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[3](xy[0], xy[1],bold)

 updateStemGraphs[3](a,b)

 },

 function(a,b,bold) {

 var xy = coor(a,b)

 updatePaths[4](xy[0], xy[1],bold)

 updateStemGraphs[4](a,b)

 }, defaults)(d3.select("#momentumCanvas"))

 var tax = renderTaxonomy(d3.select("#momentum2D"))

 var updatePaths = renderOverlay(d3.select("#momentumOverlay"), tax.div)

 var updateStemGraphs = tax.update

 colorMap(

 d3.select("#momentumAnnotation"),

 180,

 d3.scaleLinear().domain([0,1.2001]).range([0, 180])

 )

 var up = function (i, alpha, beta) {

 var xy = coor(alpha, beta)

 updatePaths[i](xy[0], xy[1], true)

 updateStemGraphs[i](alpha,beta)

 }

 for (var i = 0; i<5; i++) {

 up(i,defaults[i][0], defaults[i][1])

 }

 renderMath(document.getElementById("momentum2D"))

 callback(null);

 });

 For what values of

dt-math[block] {

 display: block;

}

α\alphaα and

dt-math[block] {

 display: block;

}

β\betaβ does momentum converge? Since we need both

dt-math[block] {

 display: block;

}

σ1\sigma\_1σ1 and

dt-math[block] {

 display: block;

}

σ2\sigma\_2σ2 to converge, our convergence criterion is now

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

 We recover the previous result for gradient descent when

dt-math[block] {

 display: block;

}

---

### The Critical Damping Coefficient

 The true magic happens, however, when we find the sweet spot of

dt-math[block] {

 display: block;

}

α\alphaα and

dt-math[block] {

 display: block;

}

β\betaβ. Let us try to first optimize over

dt-math[block] {

 display: block;

}

β\betaβ.

 Momentum admits an interesting physical interpretation when

dt-math[block] {

 display: block;

}

α\alphaα is [11]

 small: it is a discretization of a damped harmonic oscillator. Consider

 a physical simulation operating in discrete time (like a video game).

dt-math[block] {

 display: block;

}

yik+1y\_{i}^{k+1}yik+1

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

λixik\lambda\_{i}x\_{i}^{k}λixik

 and perturbed by an external force field

 We can think of

dt-math[block] {

 display: block;

}

−yik-y\_i^k−yik as **velocity**

dt-math[block] {

 display: block;

}

βyik\beta y\_{i}^{k}βyik

 which is dampened at each step

dt-math[block] {

 display: block;

}

xik+1x\_i^{k+1}xik+1

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

xik−αyik+1x\_i^k - \alpha y\_i^{k+1}xik−αyik+1

 And

dt-math[block] {

 display: block;

}

xxx is our particle's **position**

 which is moved at each step by a small amount in the direction of the velocity

dt-math[block] {

 display: block;

}

yik+1y^{k+1}\_iyik+1.

 renderMath(document.getElementById("momentum\_annotations"))

dt-math[block] {

 display: block;

}

β = 10.9850.950.90.850.8

λ = 0 0.02 0.1 0.25VelocityDampingExternal ForcePosition**dt-math[block] {

 display: block;

}

β\betaβ: Horizontal Axis**

When

dt-math[block] {

 display: block;

}

λi=0\lambda\_i = 0λi=0 and

dt-math[block] {

 display: block;

}

β=1\beta=1β=1, the object moves at constant speed. As

dt-math[block] {

 display: block;

}

 display: block;

}

λ\lambdaλ: Vertical Axis**

 The external force causes the particle to return to the origin.

Combining damping and the force field, the particle behaves like a

damped harmonic oscillator, returning lazily to equlibrium.

Initial point x = 1, y = 1

 deleteQueue.push(renderLoading(d3.select("#phasediagram0")))

 renderQueue.push(function(callback) {

 phaseDiagram\_dec(d3.select("#phasediagram0div"))

 renderMath(d3.select("#phasediagram0div").node())

 var figure = d3.select("#phasediagram0")

 var figwidth = figure.style("width")

 var figheight = figure.style("height")

 var svg = figure.append("svg")

 .style("width", figwidth)

 .style("height", figheight)

 .style("position", "absolute")

 .style("top","0px")

 .style("left","0px")

 .style("pointer-events", "none")

 callback(null);

 renderMath(document.getElementById("phasediagram0"))

 });

 This system is best imagined as a weight suspended on a spring. We

pull the weight down by one unit, and we study the path it follows as it

 returns to equilibrium. In the analogy, the spring is the source of our

 external force

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

xikx^k\_ixik and the speed

dt-math[block] {

 display: block;

}

yiky^k\_iyik are 0. The choice of

dt-math[block] {

 display: block;

}

β\betaβ crucially affects the rate of return to equilibrium.

reachesoptimum**Overdamping** When

dt-math[block] {

 display: block;

}

β\betaβ is too small (e.g. in Gradient Descent,

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

RRR are repeated, when

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

β\betaβ

 is too large we're under-damping. Here the resistance is too small, and

 spring oscillates up and down forever, missing the optimal value over

and over. 1.00.80.60.40.20.0Momentum β

 deleteQueue.push(renderLoading(d3.select("#phasediagram1")))

 renderQueue.push(function(callback) {

 phaseDiagram(d3.select("#phasediagram1"));

 renderMath(document.getElementById("phasediagram1"))

 callback(null);

 });

 The critical value of

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

iii) of

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

1−αλi1-\alpha\lambda\_i1−αλi! Alas, this only applies to the error in the

dt-math[block] {

 display: block;

}

ithi^{th}ith eigenspace, with

dt-math[block] {

 display: block;

}

α\alphaα fixed.

### Optimal parameters

 To get a global convergence rate, we must optimize over both

dt-math[block] {

 display: block;

}

α\alphaα and

dt-math[block] {

 display: block;

}

β\betaβ. This is a more complicated affair,6 but they work out to be

dt-math[block] {

 display: block;

}

α=(2λ1+λn)2β=(λn−λ1λn+λ1)2

 \alpha =

\left(\frac{2}{\sqrt{\lambda\_{1}}+\sqrt{\lambda\_{n}}}\right)^{2} \quad

\beta =

 Plug this into the convergence rate, and you get

dt-math[block] {

 display: block;

}

κ−1κ+1\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}√κ+1√κ−1

 Convergence rate, **Momentum**

dt-math[block] {

 display: block;

}

κ−1κ+1 \frac{\kappa-1}{\kappa+1}κ+1κ−1

 Convergence rate, **Gradient Descent**

 renderMath(document.getElementById("conv\_rate\_comparisons"))

 With barely a modicum of extra effort, we have essentially square

rooted the condition number! These gains, in principle, require explicit

 knowledge of

dt-math[block] {

 display: block;

}

λ1\lambda\_1λ1 and

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

111. So set

dt-math[block] {

 display: block;

}

β\betaβ as close to

dt-math[block] {

 display: block;

}

111 as you can, and then find the highest

dt-math[block] {

 display: block;

}

We can do the same decomposition here with momentum, with eigenvalues

dt-math[block] {

 display: block;

}

λ1=0.01\lambda\_1=0.01λ1=0.01,

dt-math[block] {

 display: block;

}

λ2=0.1\lambda\_2=0.1λ2=0.1, and

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

f(wk)−f(w⋆)f(w^k) - f(w^\star)f(wk)−f(w⋆)

 Note that the optimal parameters do not necessarily imply the fastest

convergence, though, only the fastest asymptotic convergence rate.

Step-size α =

Momentum β =

02401

020406080100120140Eigenvalue 123

Optimal parameters

 deleteQueue.push(renderLoading(d3.select("#milestonesMomentumFig")))

 renderQueue.push(function(callback) {

 var graphDiv = d3.select("#milestonesMomentum")

 .style("width", 920 + "px")

 .style("height", 300 + "px")

 .style("top", "170px")

 .style("position", "relative")

 .style("margin-left", "auto")

 .style("margin-right", "auto")

 .attr("width", 920)

 .attr("height", 500)

 var svg = graphDiv.append("svg").attr("width", 940).attr("height", 500)

 var update = renderMilestones(svg, function() {})

 var reset = slider2D(d3.select("#sliderStep2D"),

 function(x,y) { update(x,y) },

 1,

 100)

 // // Swoopy Annotator

 annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 798,98 A 27.97 27.97 0 0 0 760.9999389648438,70.9999771118164",

 "text": "Optimal parameters",

 "textOffset": [

 740,

 109

 ]

 }

 ]

 var sel = drawAnnotations(d3.select("#milestonesMomentumFig"), annotations)

 callback(null);

 });

 While the loss function of gradient descent had a graceful, monotonic

curve, optimization with momentum displays clear oscillations. These

ripples are not restricted to quadratics, and occur in all kinds of

functions in practice. They are not cause for alarm, but are an

indication that extra tuning of the hyperparameters is required.

---

Example: The Colorization Problem

---------------------------------

dt-math[block] {

 display: block;

}

GGG be the graph with vertices as pixels,

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

minimize\text{minimize} minimize

dt-math[block] {

 display: block;

}

 The **colorizer** pulls distinguished pixels towards 1

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

 The **smoother** spreads out the color

 renderMath(document.getElementById("colorizer\_equation"))

 The optimal solution to this problem is a vector of all

dt-math[block] {

 display: block;

}

111's 7.

 An inspection of the gradient iteration reveals why we take a long time

 to get there. The gradient step, for each component, is some form of

weighted average of the current value and its neighbors:

dt-math[block] {

 display: block;

}

wik+1=wik−α∑j∈N(wik−wjk)−{α(wik−1)i∈D0i∉D

\alpha(w\_{i}^{k}-1) & i\in D\\

0 & i\notin D

\end{cases}

 wik+1=wik−αj∈N∑(wik−wjk)−{α(wik−1)0i∈Di∉D

 This kind of local averaging is effective at smoothing out local

variations in the pixels, but poor at taking advantage of global

structure. The updates are akin to a drop of ink, diffusing through

water. Movement towards equilibrium is made only through local

corrections and so, left undisturbed, its march towards the solution is

slow and laborious. Fortunately, momentum speeds things up

significantly.

dt-math[block] {

 display: block;

}

RnR^nRn.

 The smallest eigenvalues have low frequencies, hence gradient descent

corrects high frequency errors well but not low frequency ones.

 deleteQueue.push(renderLoading(d3.select("#flow")))

 renderQueue.push(function(callback) {

 d3.queue()

 .defer(d3.json, "assets/data/Sigma.json")

 .defer(d3.json, "assets/data/matrix.json")

 .defer(d3.json, "assets/data/Uval.json")

 .await(function(error, FlowSigma, M, FlowU) {

 if (error) {

 console.error("Error loading data files");

 }

 else {

 var reset = renderFlowWidget(d3.select("#flow"), FlowSigma, M, FlowU)

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 389,55 A 28.57 28.57 0 0 0 352,30",

 "text": "Optimal parameters",

 "textOffset": [

 341,

 65

 ]

 }

 ]

 var sel = drawAnnotations(d3.select("#flow"), annotations)

 }

 });

 callback(null);

 });

 In vectorized form, the colorization problem is

dt-math[block] {

 display: block;

}

minimize\text{minimize}minimize

 The **smoother**'s quadratic form is the **Graph Laplacian**

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

12xTLGx\frac{1}{2}x^{T}L\_{G}x21xTLGx

 And the colorizer is a small low rank correction with a linear term.

dt-math[block] {

 display: block;

}

eie\_iei is the

dt-math[block] {

 display: block;

}

ithi^{th}ith unit vector.

 The Laplacian matrix,

dt-math[block] {

 display: block;

}

LGL\_GLG

8,

 which dominates the behavior of the optimization problem, is a valuable

 bridge between linear algebra and graph theory. This is a rich field of

 study, but one fact is pertinent to our discussion here. The

conditioning of

dt-math[block] {

 display: block;

}

LGL\_GLG,

 here defined as the ratio of the second eigenvector to the last (the

first eigenvalue is always 0, with eigenvector equal to the matrix of

all 1′s), is directly connected to the connectivity of the graph.

Small world graphs, like expanders and dense graphs, have excellent conditioning

The conditioning of grids improves with its dimensionality.

And long, wiry graphs, like paths, condition poorly.

 deleteQueue.push(renderLoading(d3.select("#laplacianConditioning")))

 renderQueue.push(function(callback) {

 var g1 = d3.select("#graph").append("g")

 genExpander(g1)

 var g2 = d3.select("#graph").append("g").attr("transform", "translate(300,0)")

 genGrid(g2)

 var g3 = d3.select("#graph").append("g").attr("transform", "translate(600,0)")

 genPath(g3);

 callback(null);

 });

 These observations carry through to the colorization problem, and the

intuition behind it should be clear. Well connected graphs allow rapid

diffusion of information through the edges, while graphs with poor

connectivity do not. And this principle, taken to the extreme, furnishes

 a class of functions so hard to optimize they reveal the limits of

first order optimization.

---

 The Limits of Descent

-----------------------

 Let's take a step back. We have, with a clever trick, improved the

convergence of gradient descent by a quadratic factor with the

introduction of a single auxiliary sequence. But is this the best we can

 do? Could we improve convergence even more with two sequences? Could

one perhaps choose the

dt-math[block] {

 display: block;

}

α\alphaα's and

dt-math[block] {

 display: block;

}

 Unfortunately, while improvements to the momentum algorithm do exist,

they all run into a certain, critical, almost inescapable lower bound.

### Adventures in Algorithmic Space

 To understand the limits of what we can do, we must first formally

define the algorithmic space in which we are searching. Here's one

possible definition. The observation we will make is that both gradient

descent and momentum can be "unrolled". Indeed, since

dt-math[block] {

 display: block;

}

 \begin{array}{lll}

 w^{1} & \!= & \!w^{0} ~-~ \alpha\nabla f(w^{0})\\[0.35em]

 w^{2} & \!= & \!w^{1} ~-~ \alpha\nabla f(w^{1})\\[0.35em]

 & \!= & \!w^{0} ~-~ \alpha\nabla f(w^{0}) ~-~ \alpha\nabla f(w^{1})\\[0.35em]

 & ~ \!\vdots \\

 \end{array}

 we can write gradient descent as

dt-math[block] {

 display: block;

}

wk+1  =  w0 − α∑ik∇f(wi).

 w^{k+1} ~~=~~ w^{0} ~-~ \alpha\sum\_i^k\nabla f(w^{i}).

 wk+1  =  w0 − αi∑k∇f(wi).

 A similar trick can be done with momentum:

dt-math[block] {

 display: block;

}

wk+1  =  w0 + α∑ik(1−βk+1−i)1−β∇f(wi).

 wk+1  =  w0 + αi∑k1−β(1−βk+1−i)∇f(wi).

 In fact, all manner of first order algorithms, including the Conjugate

 Gradient algorithm, AdaMax, Averaged Gradient and more, can be written

(though not quite so neatly) in this unrolled form. Therefore the class

of algorithms for which

dt-math[block] {

 display: block;

}

wk+1  =  w0 + ∑ikγik∇f(wi) for some γik

 wk+1  =  w0 + i∑kγik∇f(wi) for some γik

 contains momentum, gradient descent and a whole bunch of other

algorithms you might dream up. This is what is assumed in Assumption

dt-math[block] {

 display: block;

}

wk+1  =  w0 + ∑ikΓik∇f(wi) for some diagonal matrix Γik.

 wk+1  =  w0 + i∑kΓik∇f(wi) for some diagonal matrix Γik.

 This class of methods covers most of the popular algorithms for

training neural networks, including ADAM and AdaGrad. We shall refer to

this class of methods as "Linear First Order Methods", and we will show a

 single function all these methods ultimately fail on.

### The Resisting Oracle

 Earlier, when we talked about the colorizer problem, we observed that

wiry graphs cause bad conditioning in our optimization problem. Taking

this to its extreme, we can look at a graph consisting of a single

path — a function so badly conditioned that Nesterov called a variant of

 it "the worst function in the world". The function follows the same

structure as the colorizer problem, and we shall call this the Convex

Rosenbrock,

dt-math[block] {

 display: block;

}

fn(w)f^n(w)fn(w)

dt-math[block] {

 display: block;

}

===

 with a colorizer of one node

dt-math[block] {

 display: block;

}

12(w1−1)2\frac{1}{2}\left(w\_{1}-1\right)^{2}21(w1−1)2

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

 strong couplings of adjacent nodes in the path,

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

2κ−1∥w∥2.\frac{2}{\kappa-1}\|w\|^{2}.κ−12∥w∥2.

 and a small regularization term.

 The optimal solution of this problem is

dt-math[block] {

 display: block;

}

wi⋆=(κ−1κ+1)i

 w\_{i}^{\star}=\left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^{i}

 wi⋆=(√κ+1√κ−1)i

 and the condition number of the problem

dt-math[block] {

 display: block;

}

fnf^nfn approaches

dt-math[block] {

 display: block;

}

κ\kappaκ as

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

w0=0w^0 = 0w0=0.

02401

Step-size α =

Momentum β =

Here we see the first 50 iterates of momentum on the Convex Rosenbrock for

dt-math[block] {

 display: block;

}

The

 remaining expanding space is the "light cone" of our iterate's

influence. Momentum does very well here with the optimal parameters.

Iteration <PHONE>

Error

0.00.20.40.60.81.0

Weights

0.00.20.40.60.81.0

Optimal parameters

 deleteQueue.push(renderLoading(d3.select("#rosenViz")))

 renderQueue.push(function(callback) {

 var alpha = 0.003

 var beta = 0.8

 var b = zeros(25); b[0] = 249.75

 var iter = geniterMomentum(RQ, RLambda, b, alpha, beta).iter

 var res = function(i) { return iter(i)[0] }

 var sampleSVG = d3.select("#iterates")

 .style("display", "block")

 .append("svg")

 .attr("width", 770)

 .attr("height", 700)

 var xstar = iter(<PHONE>)

 var numIters = 52

 var pathstr = "M 8,0"

 for (var j = 0; j < 27; j++) {

 if (j != 26 && j > 0) {

 pathstr = pathstr + "L " + j\*8 + "," + (j-1)\*8 + "L " + j\*8 + "," + j\*8

 }

 if (j == 0) {

 //pathstr = pathstr + "L " + j\*8 + "," + j\*8

 }

 if (j == 26) {

 pathstr = pathstr + "L " + j\*8 + "," + (j-1)\*8

 }

 }

 pathstr = pathstr + "L "+ 26\*8 + ",0 L 8,0 "

 var Disps3 = []

 for (var j = 0; j < numIters; j++) {

 var disp = errorPlot.append("g")

 var denter = disp.selectAll("rect")

 .data(iter(j)[1])

 .enter()

 denter.append("rect")

 .style("fill", function(d,i) { return rosen(d) })

 .attr("height", 7.7)

 .attr("width", 7.7)

 .attr("x", function(d, i){return i\*8+ 10})

 .attr("y", function(d, i){return j\*8 })

 Disps3.push(disp)

 }

 var Disps4 = []

 for (var j = 0; j < numIters; j++) {

 var disp = errorPlot.append("g")

 var denter = disp.selectAll("rect")

 .data(iter(j)[1])

 .enter()

 denter.append("rect")

 .style("fill", function(d,i) { return jet(d) })

 .attr("height", 7.7)

 .attr("width", 7.7)

 .attr("x", function(d, i){return i\*8+ 10})

 .attr("y", function(d, i){return j\*8 })

 if ((j % 10 == 0) || j == 0) {

 disp.append("text")

 .attr("class", "figtext")

 .attr("text-anchor", "end")

 .attr("x", 0)

 .attr("y", function(d, i){return (j\*8 + 10) })

 .html( (j == 0) ? "Iteration 0" : "" + j)

 }

 Disps4.push(disp)

 }

 function update(alpha, beta) {

 for (var j = 0; j < numIters; j++) {

 var iterj = iter(j)

 }

 }

 update(0.0035749243028120182, 0.8095238095238095)

 colorMap( d3.select("#rosen\_colorbar1"),

 194,

 rosen,

 d3.scaleLinear().domain([0,1]).range([0, 194]) )

 colorMap( d3.select("#rosen\_colorbar2"),

 194,

 jet,

 d3.scaleLinear().domain([0,1]).range([0, 194]) );

 // Swoopy Annotator

 var annotations = [

 {

 "x": 0,

 "y": 0,

 "path": "M 817,95 A 23.869 23.869 0 0 0 792,72",

 "text": "Optimal parameters",

 "textOffset": [

 767,

 109

 ]

 }

 ]

 var sel = drawAnnotations(d3.select("#rosenViz"), annotations)

 callback(null);

 });

 The observations made in the above diagram are true for any Linear

First Order algorithm. Let us prove this. First observe that each

component of the gradient depends only on the values directly before and

 after it:

dt-math[block] {

 display: block;

}

∇f(x)i=2wi−wi−1−wi+1+4κ−1wi,i≠1.

 ∇f(x)i=2wi−wi−1−wi+1+κ−14wi,i≠1.

 Therefore the fact we start at 0 guarantees that that component must

remain stoically there till an element either before or after it turns

nonzero. And therefore, by induction, for any linear first order

algorithm,

dt-math[block] {

 display: block;

}

 \begin{array}{lllllllll}

 w^{0} & = & [~~0, & 0, & 0, & \ldots & 0,

& 0, & \ldots & 0~]\\[0.35em]

 w^{1} & = & [~w\_{1}^{1}, & 0, & 0, & \ldots

& 0, & 0, & \ldots & 0~]\\[0.35em]

 w^{2} & = & [~w\_{1}^{2}, & w\_{2}^{2}, & 0, &

\ldots & 0, & 0, & \ldots & 0~]\\[0.35em]

 & ~ \vdots \\

 w^{k} & = & [~w\_{1}^{k}, & w\_{2}^{k}, & w\_{3}^{k},

& \ldots & w\_{k}^{k}, & 0, & \ldots & 0~].\\

 \end{array}

dt-math[block] {

 display: block;

}

kkk steps to move from

dt-math[block] {

 display: block;

}

w0w\_0w0 to

dt-math[block] {

 display: block;

}

wkw\_kwk. We can therefore sum up the errors which cannot have changed yet9:

dt-math[block] {

 display: block;

}

∥wk−w⋆∥∞≥maxi≥k+1{∣wi⋆∣}=(κ−1κ+1)k+1=(κ−1κ+1)k∥w0−w⋆∥∞.

 \begin{aligned}

\|w^{k}-w^{\star}\|\_{\infty}&\geq\max\_{i\geq

 \end{aligned}

 As

dt-math[block] {

 display: block;

}

nnn gets large, the condition number of

dt-math[block] {

 display: block;

}

fnf^nfn approaches

dt-math[block] {

 display: block;

}

κ\kappaκ.

 And the gap therefore closes; the convergence rate that momentum

promises matches the best any linear first order algorithm can do. And

we arrive at the disappointing conclusion that on this problem, we

cannot do better.

 Like many such lower bounds, this result must not be taken literally,

but spiritually. It, perhaps, gives a sense of closure and finality to

our investigation. But this is not the final word on first order

optimization. This lower bound does not preclude the possibility, for

example, of reformulating the problem to change the condition number

itself! There is still much room for speedups, if you understand the

right places to look.

Momentum with Stochastic Gradients

----------------------------------

 There is a final point worth addressing. All the discussion above

assumes access to the true gradient — a luxury seldom afforded in modern

 machine learning. Computing the exact gradient requires a full pass

over all the data, the cost of which can be prohibitively expensive.

Instead, randomized approximations of the gradient, like minibatch

sampling, are often used as a plug-in replacement of

dt-math[block] {

 display: block;

}

∇f(w)\nabla f(w)∇f(w). We can write the approximation in two parts,

dt-math[block] {

 display: block;

}

∇f(w)\nabla f(w)∇f(w)

 the true gradient

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

error(w).\text{error}(w).error(w).

 and an approximation error.

If the estimator is unbiased e.g.

dt-math[block] {

 display: block;

}

E[error(w)]=0\mathbf{E}[\text{error}(w)] = 0E[error(w)]=0

 renderMath(document.getElementById("truegradientpluserror"))

 It is helpful to think of our approximate gradient as the injection of

 a special kind of noise into our iteration. And using the machinery

developed in the previous sections, we can deal with this extra term

directly. On a quadratic, the error term cleaves cleanly into a separate

 term, where 10

dt-math[block] {

 display: block;

}

(yikxik) \left(\begin{array}{c}

 y\_{i}^{k}\\

 x\_{i}^{k}

 \end{array}\right)(yikxik)

 the noisy iterates are a sum of

dt-math[block] {

 display: block;

}

===

dt-math[block] {

 display: block;

}

Rk(yi0xi0)R^{k}\left(\begin{array}{c}

 y\_{i}^{0}\\

 x\_{i}^{0}

 \end{array}\right)Rk(yi0xi0)

 the noiseless, deterministic iterates and

dt-math[block] {

 display: block;

}

+++

dt-math[block] {

 display: block;

}

ϵik∑j=1kRk−j(1−α)\epsilon^k\_i \sum\_{j=1}^{k}R^{k-j}\left(\begin{array}{c}

1\\

-\alpha

\end{array}\right)ϵikj=1∑kRk−j(1−α)

 a decaying sum of the errors, where

dt-math[block] {

 display: block;

}

ϵk=Q⋅error(wk)\epsilon^k = Q \cdot \text{error}(w^k)ϵk=Q⋅error(wk).

 renderMath(document.getElementById("iteratespluserror"))

 The error term,

dt-math[block] {

 display: block;

}

ϵk\epsilon^kϵk, with its dependence on the

dt-math[block] {

 display: block;

}

wkw^kwk, is a fairly hairy object. Following [10],

 we model this as independent 0-mean Gaussian noise. In this simplified

model, the objective also breaks into two separable components, a sum of

 a deterministic error and a stochastic error

 11, visualized here.

We decompose the expected value of the objective value

dt-math[block] {

 display: block;

}

dt-math[block] {

 display: block;

}

Ef(w)−f(w⋆)\mathbf{E} f(w) - f(w^\star) Ef(w)−f(w⋆)

 The small black dots are a single run of stochastic gradient

Step-size α =

Momentum β =

As [1]

 observes, the optimization has two phases. In the initial transient

phase the magnitude of the noise is smaller than the magnitude of the

gradient, and Momentum still makes good progress. In the second,

stochastic phase, the noise overwhelms the gradient, and momentum is

less effective.

02401

020406080100120140Fine-tuning phaseTransient phase

 deleteQueue.push(renderLoading(d3.select("#stochastic")))

 renderQueue.push(function(callback) {

 var graphDiv = d3.select("#stochasticgd")

 .style("width", 920 + "px")

 .style("height", 300 + "px")

 .style("top", "170px")

 .style("position", "relative")

 .style("margin-left", "auto")

 .style("margin-right", "auto")

 .attr("width", 920)

 .attr("height", 500)

 var svg = graphDiv.append("svg").attr("width", 940).attr("height", 500)

 var update = renderStochasticMilestones(svg, function() {})

 slidera(0.5,0.5)

 });

 Note that there are a set of unfortunate tradeoffs which seem to pit

the two components of error against each other. Lowering the step-size,

for example, decreases the stochastic error, but also slows down the

rate of convergence. And increasing momentum, contrary to popular

belief, causes the errors to compound. Despite these undesirable

properties, stochastic gradient descent with momentum has still been

shown to have competitive performance on neural networks. As [1]

 has observed, the transient phase seems to matter more than the

fine-tuning phase in machine learning. And in fact, it has been recently

 suggested [12]

 that this noise is a good thing — it acts as a implicit regularizer,

which, like early stopping, prevents overfitting in the fine-tuning

phase of optimization.

---

Onwards and Downwards

---------------------

 The study of acceleration is seeing a small revival within the

optimization community. If the ideas in this article excite you, you may

 wish to read [13],

 which fully explores the idea of momentum as the discretization of a

certain differential equation. But other, less physical, interpretations

 exist. There is an algebraic interpretation of momentum in terms of

 connecting momentum to older methods, like the Ellipsoid method. And

 But like the proverbial blind men feeling an elephant, momentum seems

like something bigger than the sum of its parts. One day, hopefully

soon, the many perspectives will converge into a satisfying whole.


Shan Carter
David Ha
Ian Johnson
Chris Olah

Abstract: Neural networks are an extremely successful approach to machine  learning, but it’s tricky to understand why they behave the way they do.  This has sparked a lot of interest and effort around trying to  understand and visualize them, which we think is so far just scratching  the surface of what is possible.

Four Experiments in Handwriting with a Neural Network

=====================================================

Let's start with generating new strokes based on your handwriting input

-----------------------------------------------------------------------

Play/Pause

Clear

Length of prediction

20

Variation1

0.1

 dt-byline {

 font-size: 12px;

 line-height: 18px;

 display: block;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 color: rgba(0, 0, 0, 0.5);

 padding-top: 12px;

 padding-bottom: 12px;

 }

 dt-article.centered dt-byline {

 text-align: center;

 }

 dt-byline a,

 dt-article dt-byline a {

 text-decoration: none;

 border-bottom: none;

 }

 dt-article dt-byline a:hover {

 text-decoration: underline;

 border-bottom: none;

 }

 dt-byline .authors {

 text-align: left;

 }

 dt-byline .name {

 display: inline;

 text-transform: uppercase;

 }

 dt-byline .affiliation {

 display: inline;

 }

 dt-byline .date {

 display: block;

 text-align: left;

 }

 dt-byline .year, dt-byline .month {

 display: inline;

 }

 dt-byline .citation {

 display: block;

 text-align: left;

 }

 dt-byline .citation div {

 display: inline;

 }

 @media(min-width: 768px) {

 dt-byline {

 }

 }

 @media(min-width: 1080px) {

 dt-byline {

 border-bottom: none;

 margin-bottom: 70px;

 }

 dt-byline a:hover {

 color: rgba(0, 0, 0, 0.9);

 }

 dt-byline .authors {

 display: inline-block;

 }

 dt-byline .author {

 display: inline-block;

 margin-right: 12px;

 /\*padding-left: 20px;\*/

 /\*border-left: 1px solid #ddd;\*/

 }

 dt-byline .affiliation {

 display: block;

 }

 dt-byline .author:last-child {

 margin-right: 0;

 }

 dt-byline .name {

 display: block;

 }

 dt-byline .date {

 border-left: 1px solid rgba(0, 0, 0, 0.1);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .year, dt-byline .month {

 display: block;

 }

 dt-byline .citation {

 border-left: 1px solid rgba(0, 0, 0, 0.15);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .citation div {

 display: block;

 }

 }

[Shan Carter](http://shancarter.com/)

[Google Brain](http://g.co/brain)

[David Ha](https://github.com/hardmaru)

[Google Brain](http://g.co/brain)

[Ian Johnson](https://github.com/enjalot)

[Google Cloud](http://cloud.google.com/)

[Chris Olah](http://colah.github.io/)

[Google Brain](http://g.co/brain)

Dec. 6

2016

[Citation:

Carter, et al., 2016](#citation)

 Neural networks are an extremely successful approach to machine

learning, but it's tricky to understand why they behave the way they do.

 This has sparked a lot of interest and effort around trying to

understand and visualize them, which we think is so far just scratching

the surface of what is possible.

 and visualizing it in a number of ways. The model is quite simple (so

as to run well in the browser) so the generated output mostly produces

gibberish letters and words (albeit, gibberish that look like real

handwriting), but it is still useful for our purposes of exploring

visualization techniques.

In the end we don't have some ultimate answer or visualization, but

 we do have some interesting ideas to share. Ultimately we hope they

make it easier to divine some meaning from the internals of these model.

Looking at the Output of the Model

----------------------------------

Our first experiment is the most obvious: when we want to see how

well someone has learned a task we usually ask them to demonstrate it.

So, let's ask our model to write something for us and see how well it

does.

Play/Pause

Variation[1](#footnote1)

0.6

Most of the marks are gibberish, but many of them are surprisingly

convincing. Some real (or real-ish) words even start to appear. One

surprising thing you'll notice is that the general style of handwriting

is more or less consistent within a sample. This is because the type of

architecture used for this model (LSTM) has a mechanism for remembering

previous strokes. It is therefore able to remember things like how loopy

 or jerky the handwriting is, or which letter preceeded the current one.

Even with this memory, the technique for generating samples from

neural networks is probabilistic, meaning each of these samples is one

of a much, much larger possibility space. Viewing them one at a time

feels unsatisfying — how can we infer much from one or two samples when

there are nearly infinite possibilities? If something looks wrong, how

do we know if it was because there is something fundamentally wrong with

 our model or if it was just dumb luck?

At each iteration our model could have produced many paths. Instead

 of picking one and throwing the rest away, let's draw, say, 50 of them.

 Green strokes below are places where the model would have veered

rightward from the chosen stroke, orange is where it would have veered

leftward.

Generated sample 1

Generated sample 2

Generated sample 3

Generated sample 4

Generated sample 5

Straight line

Steps

8

Variation[1](#footnote1)

0.5

With this technique we are casting a light into a wider possibility

 space. You can see some areas where there was general consensus as to

what to do next. Other areas had more of a "anything can happen next"

feeling. Others seem to show a possible fork in the road. We can be

drawing an "a" or "g". A few steps later it's clear the model has

converged on a cursive "g".

In addition to visualizing samples generated by the model, this

technique can also be applied to human-generated samples. Below we can

see what the model would have done at each point if it had taken over

for the person.

Validation sample 0

Validation sample 1

Validation sample 2

Validation sample 3

Validation sample 4

Validation sample 5

Validation sample 6

Steps

8

Variation[1](#footnote1)

0.5

It is obvious from these experiments that this model has learned

quite a lot about human handwriting. Which sort of raises the question,

can we extract that knowledge in any meaningful way, rather than just

blindly using it to mimic handwriting?

Examining the Internals of the Model

------------------------------------

Our model has 500 cells which act as a sort of memory which it will

 use as part of its input when deciding what to generate. If we can see

what those cells are doing as the model progresses we may be able to

gain some intuitive understanding about what the model is doing.

We begin by showing the activation of the cells over time. Each

column in the heatmap below represents one line segment of the

handwriting. Each row represents one cell of the model and is colored by

 its activations on that part of the stroke. By inspecting the diagram

you may be able to see some patterns in the way certain cells activate

for certain types of strokes. You can change which cell is used to color

 the strokes by clicking on the diagram.

Activation value

-4-2024

Step 0

Step 0

Cell 93

Activations at **step 0**

You may have been able to pick out one or two cells with

interesting patterns, but we have reason to believe that cells work in

tandem, such that an individual cell's activity may not be as

interesting as a group of cells.

 to the activations of the cells over time can organize them, bringing

cells with similar behavior close together. This makes the diagram

easier to read. We use a few small tricks to achieve the best results.3

Activation value

-4-2024

Step 0

Step 0

Cell 55

Activations at **step 0**

 seems to predict whether the pen is about to be lifted. In the middle,

we see a variety of cells, including some tracking absolute position.

Another way to explore the activations is to give it a sample and

interactively see the activations. Below you can write and see the

activations of all the cells in real time.

Clear

Step 0

Cell 55

Activations at **step 0**

---

Conclusion

----------

The black box reputation of machine learning models is well

deserved, but we believe part of that reputation has been born from the

programming context into which they have been locked. The experience of

having an easily inspectable model available in the same programming

context as the interactive visualization environment (here, javascript)

proved to be very productive for prototyping and exploring new ideas for

 this post.

As we are able to move them more and more into the same programming

 context that user interface work is done, we believe we will see richer

 modes of human-ai interactions flourish. This could have a marked

impact on debugging and building models, for sure, but also in how the

models are used. Machine learning research typically seeks to mimic and

substitute humans, and increasingly it's able to. What seems less

explored is using machine learning to augment humans. This sort of

complicated human-machine interaction is best explored when the full

capabilities of the model are available in the user interface context.