Attention and Augmented Recurrent Neural Networks

=================================================

 dt-byline {

 font-size: 12px;

 line-height: 18px;

 display: block;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 color: rgba(0, 0, 0, 0.5);

 padding-top: 12px;

 padding-bottom: 12px;

 }

 dt-article.centered dt-byline {

 text-align: center;

 }

 dt-byline a,

 dt-article dt-byline a {

 text-decoration: none;

 border-bottom: none;

 }

 dt-article dt-byline a:hover {

 text-decoration: underline;

 border-bottom: none;

 }

 dt-byline .authors {

 text-align: left;

 }

 dt-byline .name {

 display: inline;

 text-transform: uppercase;

 }

 dt-byline .affiliation {

 display: inline;

 }

 dt-byline .date {

 display: block;

 text-align: left;

 }

 dt-byline .year, dt-byline .month {

 display: inline;

 }

 dt-byline .citation {

 display: block;

 text-align: left;

 }

 dt-byline .citation div {

 display: inline;

 }

 @media(min-width: 768px) {

 dt-byline {

 }

 }

 @media(min-width: 1080px) {

 dt-byline {

 border-bottom: none;

 margin-bottom: 70px;

 }

 dt-byline a:hover {

 color: rgba(0, 0, 0, 0.9);

 }

 dt-byline .authors {

 display: inline-block;

 }

 dt-byline .author {

 display: inline-block;

 margin-right: 12px;

 /\*padding-left: 20px;\*/

 /\*border-left: 1px solid #ddd;\*/

 }

 dt-byline .affiliation {

 display: block;

 }

 dt-byline .author:last-child {

 margin-right: 0;

 }

 dt-byline .name {

 display: block;

 }

 dt-byline .date {

 border-left: 1px solid rgba(0, 0, 0, 0.1);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .year, dt-byline .month {

 display: block;

 }

 dt-byline .citation {

 border-left: 1px solid rgba(0, 0, 0, 0.15);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .citation div {

 display: block;

 }

 }

[Chris Olah](http://colah.github.io/)

[Google Brain](http://g.co/brain)

[Shan Carter](http://shancarter.com/)

[Google Brain](http://g.co/brain)

Sept. 8

2016

[Citation:

Olah & Carter, 2016](#citation)

Recurrent neural networks are one of the staples of deep learning,

allowing neural networks to work with sequences of data like text, audio

 and video. They can be used to boil a sequence down into a high-level

understanding, to annotate sequences, and even to generate new sequences

 from scratch!

rnn

x0

y0

x1

y1

x2

y2

x3

y3

One cell...

can be used over...

and over...

and over...

x4

y4

again.

 even work with these. Such models have been found to be very powerful,

achieving remarkable results in many tasks including translation, voice

recognition, and image captioning. As a result, recurrent neural

networks have become very widespread in the last few years.

As this has happened, we’ve seen a growing number of attempts to

augment RNNs with new properties. Four directions stand out as

particularly exciting:

 #previews a {

 text-decoration: none;

 overflow: hidden;

 margin-bottom: 12px;

 display: block;

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 margin-bottom: 12px;

 padding-bottom: 12px;

 }

 #previews figcaption {

 margin-left: 100px;

 }

 #previews svg, #previews img {

 width: 80px;

 float: left;

 }

 @media(min-width: 768px) {

 #previews {

 overflow: hidden;

 margin-top: 48px;

 margin-bottom: 48px;

 }

 #previews figcaption {

 margin-left: 0;

 }

 #previews figcaption b {

 display: block;

 }

 #previews figcaption b span {

 display: block;

 }

 #previews a {

 position: relative;

 float: left;

 width: 19%;

 margin-right: 3.6%;

 padding-right: 3.6%;

 border-right: 1px solid rgba(0, 0, 0, 0.05);

 border-bottom: none;

 margin-bottom: 0;

 padding-bottom: 0;

 }

 #previews a:last-child {

 margin-right: 0;

 padding-right: 0;

 border-right: 0;

 }

 #previews svg, #previews img {

 margin-bottom: 18px;

 display: block;

 width: 100%;

 float: none;

 }

 }

 @media(min-width: 1024px) {

 #previews {

 margin-top: 90px;

 margin-bottom: 90px;

 }

 }

Individually, these techniques are all potent extensions of RNNs,

but the really striking thing is that they can be combined, and seem to

just be points in a broader space. Further, they all rely on the same

underlying trick—something called attention—to work.

Our guess is that these “augmented RNNs” will have an important

role to play in extending deep learning’s capabilities over the coming

years.

---

Neural Turing Machines

----------------------

Neural Turing Machines [2]

 combine a RNN with an external memory bank. Since vectors are the

natural language of neural networks, the memory is an array of vectors:

rnn

Memory is an array of vectors.

Network A writes and reads from this memory each step.

x0

y0

x1

y1

x2

y2

x3

y3

But how does reading and writing work? The challenge is that we

want to make them differentiable. In particular, we want to make them

differentiable with respect to the location we read from or write to, so

 that we can learn where to read and write. This is tricky because

memory addresses seem to be fundamentally discrete. NTMs take a very

clever solution to this: every step, they read and write everywhere,

just to different extents.

As an example, let’s focus on reading. Instead of specifying a

single location, the RNN outputs an “attention distribution” that

describes how we spread out the amount we care about different memory

positions. As such, the result of the read operation is a weighted sum.

rnn

attention

memory

The read result is a weighted sum.

Similarly, we write everywhere at once to different extents. Again,

 an attention distribution describes how much we write at every

location. We do this by having the new value of a position in memory be a

 convex combination of the old memory content and the write value, with

the position between the two decided by the attention weight.

rnn

attention

old memory

new memory

write value

"use strict";

(function() {

 function appendArrowhead(selection) {

 selection.append("marker")

 .attr("id", "arrowhead")

 .attr("viewBox", "0 -5 10 10")

 .attr("refX", 5)

 .attr("refY", 0)

 .attr("markerWidth", 10)

 .attr("markerHeight", 10)

 .attr("orient", "auto")

 .append("path")

 .attr("d", "M0,-5L5,0L0,5")

 .style("fill", "none")

 .style("stroke", "grey");

 }

 var vectorScale = d3.scaleLinear()

 .domain([0, 1])

 .range([-14, 14]);

 var memoryData = [

 {x: 0.9, y: 0.9},

 {x: 0.8, y: 0.8},

 {x: 0.1, y: 0.9},

 {x: 0.4, y: 0.2},

 {x: 0.2, y: 0.2},

 {x: 0.4, y: 0.3},

 {x: 0.1, y: 0.5}

 ];

 var writeData = {x: 0.9, y: 0};

 //

 // Read

 //

 var nodeWidth = 36;

 var readHtml = d3.selectAll("#rnn-read");

 var svg = readHtml.select("svg");

 var result = svg.select("#read-result").style("opacity", 0);

 var resultBBox = result.node().getBBox();

 var memory = svg.selectAll("#memory use");

 var memoryBBox = svg.select("#memory").node().getBBox();

 var attention = svg.selectAll("#attention use");

 var attentionBBox = svg.select("#attention").node().getBBox();

 var path = svg.selectAll("#paths path");

 appendArrowhead(svg);

 var resultArrow = svg.append("line")

 .attr("class", "result-line")

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey");

 memory.data(memoryData)

 .each(function(d, i) {

 this.style.opacity = 0;

 d.top = memoryBBox.y;

 d.left = memoryBBox.x + nodeWidth \* i;

 d.width = nodeWidth;

 });

 var memoryVectors = svg.selectAll(".vector")

 .data(memoryData)

 .enter().append("line")

 .attr("class", "vector")

 .attr("transform", function(d) {

 })

 .attr("x2", function(d) { return vectorScale(d.x); })

 .attr("y2", function(d) { return vectorScale(d.y); })

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey");

 function updateReadAttention(data) {

 attention.data(data)

 .style("opacity", function(d) { return d + 0.05; });

 path.data(data)

 .style("stroke-opacity", function(d) { return d + 0.1; });

 resultArrow

 .attr("x2", function(d) { return vectorScale(resultX); })

 .attr("y2", function(d) { return vectorScale(resultY); });

 }

 var length = memory.size();

 var attentionData = d3.range(length).map(function() { return Math.random(); });

 updateReadAttention(attentionData);

 readHtml.on("mousemove", function() {

 var data = [];

 attention.each(function() {

 var bbox = this.getBoundingClientRect()

 var distance = Math.abs(bbox.left - d3.event.clientX + bbox.width / 2);

 data.push(Math.exp(-distance / 25));

 });

 data = data.map(function(d) { return d / d3.sum(data); });

 updateReadAttention(data);

 });

 updateReadAttention(initReadData);

 //

 // Write

 //

 var writeHtml = d3.selectAll("#rnn-write");

 var writeSvg = writeHtml.select("svg");

 var writeMemory = writeSvg.selectAll("#memory use");

 var writeMemoryBBox = writeSvg.select("#memory").node().getBBox();

 var writeNewMemory = writeSvg.selectAll("#new-memory use");

 var writeNewMemoryBBox = writeSvg.select("#new-memory").node().getBBox();

 var writeAttention = writeSvg.selectAll("#attention use");

 var writeAttentionBBox = writeSvg.select("#attention").node().getBBox();

 var writePath = writeSvg.selectAll("#paths path");

 var writeResult = writeSvg.select("#write-result").style("opacity", 0);

 var writeResultBBox = writeResult.node().getBBox();

 var newMemoryData = memoryData.map(function(){ return {}; });

 appendArrowhead(writeSvg);

 var writeArrow = writeSvg.append("line")

 .attr("class", "write-line")

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey")

 .attr("x2", vectorScale(writeData.x))

 .attr("y2", vectorScale(writeData.y));

 writeMemory.data(memoryData)

 .each(function(d, i) {

 this.style.opacity = 0;

 d.top = writeMemoryBBox.y;

 d.left = writeMemoryBBox.x + i \* nodeWidth;

 d.width = nodeWidth;

 });

 var writeMemoryVectors = writeSvg.selectAll(".vector")

 .data(memoryData)

 .enter().append("line")

 .attr("class", "vector")

 .attr("transform", function(d) {

 })

 .attr("x2", function(d) { return vectorScale(d.x); })

 .attr("y2", function(d) { return vectorScale(d.y); })

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey");

 writeNewMemory.data(newMemoryData)

 .each(function(d, i) {

 this.style.opacity = 0;

 d.top = writeNewMemoryBBox.y;

 d.left = writeNewMemoryBBox.x + i \* nodeWidth;

 d.width = nodeWidth;

 });

 var writeNewMemoryVectors = writeSvg.selectAll(".new-vector")

 .data(newMemoryData)

 .enter().append("line")

 .attr("class", "new-vector")

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey");

 function updateWriteAttention(data) {

 newMemoryData.forEach(function(d, i) {

 d.x = (memoryData[i].x + writeData.x \* data[i]) / (1 + data[i]);

 d.y = (memoryData[i].y + writeData.y \* data[i]) / (1 + data[i]);

 });

 writeNewMemoryVectors

 .attr("transform", function(d) {

 })

 .attr("x2", function(d) { return vectorScale(d.x); })

 .attr("y2", function(d) { return vectorScale(d.y); });

 writeAttention.data(data)

 .style("opacity", function(d) { return d + 0.05; });

 writePath.data(data)

 .style("stroke-opacity", function(d) { return d + 0.1; });

 }

 writeHtml.on("mousemove", function() {

 var data = [];

 writeAttention.each(function() {

 var bbox = this.getBoundingClientRect()

 var distance = Math.abs(bbox.left - d3.event.clientX + bbox.width / 2);

 data.push(Math.exp(-distance / 25));

 });

 data = data.map(function(d) { return d / d3.sum(data); });

 updateWriteAttention(data);

 });

 updateWriteAttention(initWriteData);

})();

But how do NTMs decide which positions in memory to focus their

attention on? They actually use a combination of two different methods:

content-based attention and location-based attention. Content-based

attention allows NTMs to search through their memory and focus on places

 that match what they’re looking for, while location-based attention

allows relative movement in memory, enabling the NTM to loop.

rnn

The scores are then converted into a distribution using softmax.

Next, we interpolate the attention from the previous time step.

memory

Blue shows high similarity, pink high dissimilarity.

interpolation amount

shift filter

RNN controller

attention mechanism

query vector

attention from previous step

new attention distribution

"use strict";

//

// Data

//

var vectorScale = d3.scaleLinear().domain([0, 1]).range([-14, 14]);

var interpolationData = [0.3];

var shiftData = [0, 0.2, 0.8];

var previousAttentionData = [0, 0.2, 0, 0.2, 0.6];

//

// DOM

//

var nodeWidth = 36;

var html = d3.selectAll("#rnn-write-detail");

var svg = html.select("svg");

var query = svg.selectAll("#query").style("opacity", 0);

var memory = svg.selectAll("#memory use").style("opacity", 0);

var memoryBBox = svg.select("#memory").node().getBBox();

var shift = svg.selectAll("#shift use");

//

// Render

//

// Store positions of vectors

function cachePositions(selection) {

 selection.each(function (d) {

 var bBox = this.getBBox();

 d.top = bBox.y;

 d.left = bBox.x;

 d.width = bBox.width;

 });

}

// Render new vector lines

function renderVector(selection) {

 selection.attr("transform", function (d) {

 }).attr("x2", function (d) {

 return vectorScale(d.x);

 }).attr("y2", function (d) {

 return vectorScale(d.y);

 }).attr("marker-end", "url(#arrowhead)").style("stroke", "grey");

}

// Memory

memory.data(memoryData);

memory.each(function(d, i) {

 d.top = memoryBBox.y;

 d.left = memoryBBox.x + nodeWidth \* i;

 d.width = nodeWidth;

})

// cachePositions(memory);

renderVector(memoryVector);

var queryData = [{ x: 0.7, y: 0.8 }];

query.data(queryData);

cachePositions(query);

var clamp = d3.scaleLinear().clamp(true);

function redraw() {

 renderVector(queryVector);

 // Dot

 var dotData = memoryData.map(function (v) {

 });

 svg.selectAll("#dot use").data(dotData).attr("xlink:href", function (d) {

 return d < 0 ? "#array-negative-value" : "#array-positive-value";

 }).style("opacity", function (d) {

 return Math.abs(d);

 });

 // Softmax

 var expDotData = dotData.map(Math.exp);

 var softmaxData = expDotData.map(function (x) {

 return x / d3.sum(expDotData);

 });

 svg.selectAll("#softmax use").data(softmaxData).style("opacity", function (d) {

 return d;

 });

 // Interpolate

 return d;

 });

 return d;

 });

 var interpolateData = previousAttentionData.map(function (d, i) {

 return interpolationData \* d + (1 - interpolationData) \* softmaxData[i];

 });

 return d;

 });

 // Convolve

 shift.data(shiftData).style("opacity", function (d) {

 return d;

 });

 var convolveData = dotData.map(function (x) {

 return 0;

 });

 for (var n in shiftData) {

 for (var i in convolveData) {

 if (i - (1 - n) >= 0) {

 convolveData[i - (1 - n)] += shiftData[n] \* interpolateData[i];

 }

 }

 }

 // = dotData;

 return d;

 });

 // Sharpen

 var sharpenData = convolveData.map(function (x) {

 return x \* x;

 });

 sharpenData = sharpenData.map(function (x) {

 return x / d3.sum(sharpenData);

 });

 return d;

 });

}

svg.select("#query-hover").on("mousemove", function () {

 var bbox = svg.select("#query").node().getBoundingClientRect();

 var x = d3.event.clientX - bbox.left - bbox.width / 2;

 var y = d3.event.clientY - bbox.top - bbox.height / 2;

 //TODO: this clamping is awkward

 queryData[0].x = clamp((x + 25) / 50);

 queryData[0].y = clamp((y + 25) / 50);

 redraw();

});

svg.select("#interpolate-hover").on("mousemove", function () {

 var bbox = svg.select("#interpolate-amount").node().getBoundingClientRect();

 redraw();

});

// .on("mouseout", function() {

// interpolationData = [0.3];

// redraw();

// });

svg.select("#shift-hover").on("mousemove", function () {

 // shiftData = [0, 0.2, 0.8];

 shiftData = [];

 shift.each(function () {

 var bbox = this.getBoundingClientRect();

 var distance = Math.abs(bbox.left - d3.event.clientX + bbox.width / 2);

 shiftData.push(Math.exp(-distance / 25));

 });

 shiftData = shiftData.map(function (d) {

 return d / d3.sum(shiftData);

 });

 redraw();

});

redraw();

This capability to read and write allows NTMs to perform many

simple algorithms, previously beyond neural networks. For example, they

can learn to store a long sequence in memory, and then loop over it,

repeating it back repeatedly. As they do this, we can watch where they

read and write, to better understand what they’re doing:

See more experiments in [3]. This figure is based on the Repeat Copy experiment.

They can also learn to mimic a lookup table, or even learn to sort

numbers (although they kind of cheat)! On the other hand, they still

can’t do many basic things, like add or multiply numbers.

 train NTMs using reinforcement learning instead of the differentiable

In some objective sense, many of the tasks these models can

perform—such as learning how to add numbers—aren’t that objectively

hard. The traditional program synthesis community would eat them for

lunch. But neural networks are capable of many other things, and models

like the Neural Turing Machine seem to have knocked away a very profound

 limit on their abilities.

### Code

There are a number of open source implementations of these models.

---

Attentional Interfaces

----------------------

When I’m translating a sentence, I pay special attention to the

word I’m presently translating. When I’m transcribing an audio

recording, I listen carefully to the segment I’m actively writing down.

And if you ask me to describe the room I’m sitting in, I’ll glance

around at the objects I’m describing as I do so.

Neural networks can achieve this same behavior using *attention*,

 focusing on part of a subset of the information they’re given. For

example, an RNN can attend over the output of another RNN. At every time

 step, it focuses on different positions in the other RNN.

We’d like attention to be differentiable, so that we can learn

where to focus. To do this, we use the same trick Neural Turing Machines

 use: we focus everywhere, just to different extents.

rnn

Network B focuses on different information from network A at every step.

The attention distribution is usually generated with content-based

attention. The attending RNN generates a query describing what it wants

to focus on. Each item is dot-producted with the query to produce a

score, describing how well it matches the query. The scores are fed into

 a softmax to create the attention distribution.

rnn

The attending RNN generates a query describing what it wants to focus on.

One use of attention between RNNs is translation [11].

 A traditional sequence-to-sequence model has to boil the entire input

down into a single vector and then expands it back out. Attention avoids

 this by allowing the RNN processing the input to pass along information

 about each word it sees, and then for the RNN generating the output to

focus on words as they become relevant.

```

```

output text

network B

network A

input text

 #attentional-ex2 svg {

 width: 100%;

 margin-bottom: 24px;

}

#attentional-ex2 .pointer {

 position: absolute;

 width: 26px;

 height: 26px;

 top: 36px;

 left: -48px;

}

#attentional-ex2 figcaption {

 text-align: center;

}

#attentional-ex2 svg .labels {

 font-size: 12px;

 fill-opacity: 0.5;

}

#attentional-ex2 .background {

 fill: rgb(240, 240, 245);

 fill-opacity: 0.8;

}

#attentional-ex2 .cell .node {

 fill: white;

 fill-opacity: 0.9;

 stroke: black;

 stroke-width: 1px;

 stroke-opacity: 0.25;

 /\*transition: stroke-opacity, 0.2s, linear;\*/

}

#attentional-ex2 .cell:hover .node {

 stroke-opacity: 1;

}

#attentional-ex2 .cell .shadow {

 fill: black;

 fill-opacity: 0.07;

}

#attentional-ex2 .cell text {

 font-size: 13px;

 fill-opacity: 0.6;

 text-anchor: middle;

 alignment-baseline: middle;

}

#attentional-ex2 .cell text.label {

 font-size: 13px;

 /\*fill-opacity: 0.9;\*/

 text-anchor: middle;

 alignment-baseline: middle;

}

#attentional-ex2 .link-group {

 transition: opacity, 0.3s, linear;

}

#attentional-ex2 .link-group .node-highlight {

 transition: stroke-opacity, 0.3s, linear;

 stroke: black;

 stroke-opacity: 0;

 fill: none;

}

#attentional-ex2 .link-group:hover .node-highlight {

 stroke-opacity: 1;

}

#attentional-ex2 .link {

 fill: none;

 stroke: rgb(131, 89, 178);

 stroke-width: 2px;

 pointer-events: none;

}

#attentional-ex2 .arrow {

 stroke: black;

 stroke-opacity: 0.6;

}

#attentional-ex2 .cell:last-child .arrow:first-of-type {

 display: none;

}

(function() {

var labelsA = [

 "the",

 "agreement",

 "on",

 "the",

 "European",

 "Economic",

 "Area",

 "was",

 "signed",

 "in",

 "August",

 "1992",

 ".",

 "<end>"

];

var labelsB = [

 "l’",

 "accord",

 "sur",

 "la",

 "zone",

 "économique",

 "européenne",

 "a",

 "été",

 "signé",

 "en",

 "août",

 "1992",

 ".",

 "<end>"

];

var data = d3.csvParseRows(d3.select("#attentional-ex2-data").text());

// console.log(data)

var linkData = [];

data.forEach(function(a, ai) {

 a.forEach(function(b, bi) {

 linkData.push({

 ai: +ai,

 bi: +bi,

 v: +data[ai][bi]

 });

 });

});

var nodeMargin = 10;

var networkMargin = 120;

var html = d3.select("#attentional-ex2");

var outerWidth = 984;//html.node().getBoundingClientRect().width;

var outerHeight = 240;//html.node().getBoundingClientRect().height;

var margin = {top: 36, right: nodeMargin, bottom: 24, left: nodeMargin};

var width = outerWidth - margin.left - margin.right;

var height = outerHeight - margin.top - margin.bottom;

var xa = d3.scaleBand()

 .domain(d3.range(labelsA.length))

 .paddingInner(0.5)

 .range([nodeMargin, width - nodeMargin]);

var xb = d3.scaleBand()

 .domain(d3.range(labelsB.length))

 .paddingInner(0.5)

 .range([nodeMargin, width - nodeMargin]);

var nodeSizeA = xa.bandwidth();

var nodeSizeB = xb.bandwidth();

var networkHeightA = nodeSizeA + nodeMargin \* 2;

var networkHeightB = nodeSizeB + nodeMargin \* 2;

var link = function(d) {

 // console.log(nodeSizeB, nodeSizeA, nodeMargin, d.bi, xb(d.bi))

 var p = "M" + (xb(d.bi) + nodeSizeB / 2) + "," + (nodeSizeB + nodeMargin)

 + " " + (xa(d.ai) + nodeSizeA / 2) + "," + (networkMargin + nodeMargin);

 // console.log(p);

 return p;

};

html.select("svg .labels")

 .attr("transform", "translate(" + (width + nodeMargin \* 3) + ", 0)");

var svg = html.select("svg")

 .attr("viewBox", "0 0 " + outerWidth + " " + outerHeight)

 .append("g")

 .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

var networkA = svg.append("g")

 .attr("transform", "translate(0, " + networkMargin + ")");

networkA.append("rect")

 .attr("class", "background")

 .attr("width", width)

 .attr("height", nodeSizeA + 2 \* nodeMargin)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

var cellA = networkA.selectAll(".cell")

 .data(labelsA)

 .enter().append("g")

 .attr("class", "cell")

 .on("mouseover", function(d, i) {

 link.style("display", function(ld) {

 return ld.ai === i ? "" : "none";

 });

 })

 .on("mouseout", function() { link.style("display", ""); });

cellA.append("rect")

 .attr("class", "shadow")

 .attr("transform", "translate(1, 3)")

 .attr("width", nodeSizeA)

 .attr("height", nodeSizeA)

 .attr("rx", 4)

 .attr("ry", 4);

cellA.append("rect")

 .attr("class", "node")

 .attr("width", nodeSizeA)

 .attr("height", nodeSizeA)

 .attr("rx", 4)

 .attr("ry", 4);

cellA.append("line")

 .attr("class", "arrow")

 .attr("marker-start", "url(#edgeArrowheadLeft)")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("x1", nodeSizeA + 8)

 .attr("x2", nodeSizeA + nodeMargin + 20)

 .attr("y1", nodeSizeA / 2)

 .attr("y2", nodeSizeA / 2);

cellA.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("transform", "translate(" + nodeSizeA / 2 + ", 0)")

 .attr("y1", nodeSizeA + nodeMargin \* 2 - 2)

 .attr("y2", nodeSizeA + 4);

cellA.append("text")

 .style("opacity", 0.8)

 .text("A")

 .attr("dx", nodeSizeA / 2)

 .attr("dy", nodeSizeA / 2 + 1);

cellA.append("text")

 .attr("class", "label")

 .attr("dx", nodeSizeA / 2)

 .attr("dy", nodeSizeA + 24 + 8)

 .text(function(d) { return d; });

cellA.append("rect")

 .style("opacity", 0)

 .attr("width", xa.step())

 .attr("height", networkHeightA + (networkMargin - networkHeightA) / 2);

//

// Network B

//

var networkB = svg.append("g");

networkB.append("rect")

 .attr("class", "background")

 .attr("width", width)

 .attr("height", nodeSizeB + 2 \* nodeMargin)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

var cellB = networkB.selectAll(".cell")

 .data(labelsB)

 .enter().append("g")

 .attr("class", "cell")

 .on("mouseover", function(d, i) {

 link.style("display", function(ld) {

 return ld.bi === i ? "" : "none";

 });

 })

 .on("mouseout", function() { link.style("display", ""); });

cellB.append("rect")

 .attr("class", "shadow")

 .attr("transform", "translate(1, 3)")

 .attr("width", nodeSizeB)

 .attr("height", nodeSizeB)

 .attr("rx", 4)

 .attr("ry", 4);

cellB.append("rect")

 .attr("class", "node")

 .attr("width", nodeSizeB)

 .attr("height", nodeSizeB)

 .attr("rx", 4)

 .attr("ry", 4);

cellB.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("x1", nodeSizeB)

 .attr("x2", nodeSizeB + nodeMargin + 14)

 .attr("y1", nodeSizeB / 2)

 .attr("y2", nodeSizeB / 2);

cellB.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("transform", "translate(" + nodeSizeB / 2 + ", 0)")

 .attr("y1", 0)

 .attr("y2", -nodeMargin \* 2 + 5);

cellB.append("text")

 .style("opacity", 0.8)

 .text("B")

 .attr("dx", nodeSizeB / 2)

 .attr("dy", nodeSizeB / 2 + 1);

cellB.append("text")

 .attr("class", "label")

 .attr("dy", - 24 - 5)

 .text(function(d) { return d; });

cellB.append("rect")

 .style("opacity", 0)

 .attr("width", xb.step())

 .attr("height", networkHeightB + (networkMargin - networkHeightB) / 2);

//

// Links

//

var linkGroup = svg.append("g").attr("class", ".link-group")

var link = linkGroup.selectAll(".link")

 .data(linkData)

 .enter().append("path")

 .attr("class", "link")

 .attr("d", link)

 .style("opacity", function(d) { return d.v; });

//

//

//

// var hoverB = svg.selectAll(".hover-b")

// .data(labelsB)

// .enter().append("rect")

// .attr("class", "hover-b")

// .style("opacity", 0)

// .attr("width", xb.step())

// .attr("height", networkMargin / 2 + nodeSizeB)

})();

 allowing one RNN to process the audio and then have another RNN skim

over it, focusing on relevant parts as it generates a transcript.

output text

network B

network A

input audio

how\_much\_would\_a\_woodchuck\_chuck<end>

```

```

Figure derived from [Chan, *et al.* 2015](https://arxiv.org/pdf/1508.01211.pdf)

#attentional-ex3 .stage {

 position: relative;

 margin-bottom: 24px;

}

#attentional-ex3 .waveform {

 position: absolute;

 opacity: 0.7;

}

#attentional-ex3 .pointer {

 position: absolute;

 width: 26px;

 height: 26px;

 top: 36px;

 left: -48px;

}

#attentional-ex3 svg {

 z-index: 1;

 position: relative;

 width: 100%;

}

#attentional-ex3 svg .labels {

 font-size: 12px;

 fill-opacity: 0.5;

}

#attentional-ex3 figcaption {

 text-align: center;

}

#attentional-ex3 .background {

 fill: rgb(240, 240, 245);

 fill-opacity: 0.8;

}

#attentional-ex3 .audio-background {

 fill: none;

 stroke: black;

 stroke-opacity: 0.3;

}

#attentional-ex3 .cell .node {

 fill: white;

 fill-opacity: 0.9;

 stroke: black;

 stroke-width: 1px;

 stroke-opacity: 0.15;

}

#attentional-ex3 .cell:hover .node {

 stroke-opacity: 1;

}

#attentional-ex3 .cell .shadow {

 fill: black;

 fill-opacity: 0.07;

}

#attentional-ex3 .cell text {

 font-size: 13px;

 fill-opacity: 0.5;

 text-anchor: middle;

 alignment-baseline: middle;

}

#attentional-ex3 .cell text.label {

 font-size: 15px;

 /\*fill-opacity: 0.8;\*/

 text-anchor: middle;

 alignment-baseline: middle;

}

#attentional-ex3 .cell:last-child text.label {

 font-size: 10px;

}

#attentional-ex3 .link-group {

 transition: opacity, 0.3s, linear;

}

#attentional-ex3 .link-group .node-highlight {

 transition: stroke-opacity, 0.3s, linear;

 stroke: black;

 stroke-opacity: 0;

 fill: none;

}

#attentional-ex3 .link-group:hover .node-highlight {

 stroke-opacity: 1;

}

#attentional-ex3 .link {

 fill: none;

 stroke: rgb(131, 89, 178);

 stroke-width: 2px;

 pointer-events: none;

}

#attentional-ex3 .arrow {

 stroke: black;

 stroke-opacity: 0.3;

}

#attentional-ex3 .cell:last-child .arrow:first-of-type {

 display: none;

}

#attentional-ex3 .tick {

 stroke: black;

 stroke-opacity: 0.2;

}

(function() {

var data = d3.csvParseRows(d3.select("#attentional-ex3-data").text());

var labelsA = d3.range(data.length);

var linkData = [];

data.forEach(function(a, ai) {

 a.forEach(function(b, bi) {

 linkData.push({

 ai: +ai,

 bi: +bi,

 v: +data[ai][bi]

 });

 });

});

var nodeMargin = 8;

var networkMargin = 140;

var audioHeight = 60;

var html = d3.select("#attentional-ex3");

var outerHeight = 280;

var margin = {top: 36, right: nodeMargin + 80, bottom: 24, left: nodeMargin};

var width = outerWidth - margin.left - margin.right;

var height = outerHeight - margin.top - margin.bottom;

var xa = d3.scaleBand()

 .domain(d3.range(labelsA.length))

 .paddingInner(0.5)

 .range([nodeMargin, width - nodeMargin]);

var xb = d3.scaleBand()

 .domain(d3.range(labelsB.length))

 .paddingInner(0.5)

 .range([nodeMargin, width - nodeMargin]);

var nodeSizeA = xa.bandwidth();

var nodeSizeB = xb.bandwidth();

var networkHeightA = nodeSizeA + nodeMargin \* 2;

var networkHeightB = nodeSizeB + nodeMargin \* 2;

var link = function(d) {

 return "M" + (xb(d.bi) + nodeSizeB / 2) + "," + (nodeSizeB + nodeMargin)

 + " " + (xa(d.ai) + nodeSizeA / 2) + "," + (networkMargin + nodeMargin);

};

var svg = html.select("svg")

 .attr("viewBox", "0 0 " + outerWidth + " " + outerHeight)

 .append("g")

 .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

html.select("svg .labels")

 .attr("transform", "translate(" + (width + nodeMargin \* 3) + ", 0)");

var audio = svg.append("g")

 .attr("class", "audio")

//

// Network A

//

var networkA = svg.append("g")

 .attr("transform", "translate(0, " + networkMargin + ")");

networkA.append("rect")

 .attr("class", "background")

 .attr("width", width)

 .attr("height", nodeSizeA + 2 \* nodeMargin)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

var cellA = networkA.selectAll(".cell")

 .data(labelsA)

 .enter().append("g")

 .attr("class", "cell")

 .on("mouseover", function(d, i) {

 link.style("display", function(ld) {

 return ld.ai === i ? "" : "none";

 });

 })

 .on("mouseout", function() { link.style("display", ""); });

cellA.append("rect")

 .attr("class", "shadow")

 .attr("transform", "translate(1, 3)")

 .attr("width", nodeSizeA)

 .attr("height", nodeSizeA)

 .attr("rx", 4)

 .attr("ry", 4);

cellA.append("rect")

 .attr("class", "node")

 .attr("width", nodeSizeA)

 .attr("height", nodeSizeA)

 .attr("rx", 2)

 .attr("ry", 2);

cellA.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("x1", nodeSizeA)

 .attr("x2", nodeSizeA + nodeMargin)

 .attr("y1", nodeSizeA / 2)

 .attr("y2", nodeSizeA / 2);

cellA.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("transform", "translate(" + nodeSizeA / 2 + ", " + nodeSizeA + ")")

 .attr("y1", nodeMargin \* 3)

 .attr("y2", 4);

cellA.append("rect")

 .style("opacity", 0)

 .attr("width", xa.step())

//

// Network B

//

var networkB = svg.append("g");

networkB.append("rect")

 .attr("class", "background")

 .attr("width", width)

 .attr("height", nodeSizeB + 2 \* nodeMargin)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

var cellB = networkB.selectAll(".cell")

 .data(labelsB)

 .enter().append("g")

 .attr("class", "cell")

 .on("mouseover", function(d, i) {

 link.style("display", function(ld) {

 return ld.bi === i ? "" : "none";

 });

 })

 .on("mouseout", function() { link.style("display", ""); });

cellB.append("rect")

 .attr("class", "shadow")

 .attr("transform", "translate(1, 3)")

 .attr("width", nodeSizeB)

 .attr("height", nodeSizeB)

 .attr("rx", 4)

 .attr("ry", 4);

cellB.append("rect")

 .attr("class", "node")

 .attr("width", nodeSizeB)

 .attr("height", nodeSizeB)

 .attr("rx", 2)

 .attr("ry", 2);

cellB.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("x1", nodeSizeB)

 .attr("x2", nodeSizeB + nodeMargin)

 .attr("y1", nodeSizeB / 2)

 .attr("y2", nodeSizeB / 2);

cellB.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("transform", "translate(" + nodeSizeB / 2 + ", 0)")

 .attr("y1", 0)

 .attr("y2", -nodeMargin \* 2);

cellB.append("text")

 .attr("class", "label")

 .attr("dx", nodeSizeB / 2)

 .attr("dy", - nodeMargin \* 2 - 15)

 .text(function(d) { return d; });

cellB.append("rect")

 .style("opacity", 0)

 .attr("width", xb.step())

 .attr("height", networkHeightB + (networkMargin - networkHeightB) / 2);

//

// Links

//

var linkGroup = svg.append("g").attr("class", ".link-group")

var link = linkGroup.selectAll(".link")

 .data(linkData)

 .enter().append("path")

 .attr("class", "link")

 .attr("d", link)

 .style("opacity", function(d) { return d.v; });

//

// Audio

//

html.select(".waveform")

 .style("left", "1%")

 .style("top", "76%")

 .style("width", "90%")

 .style("height", "20%")

audio.append("rect")

 .attr("class", "audio-background")

 .attr("width", width)

 .attr("height", audioHeight)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

audio.selectAll(".tick")

 .data(labelsA)

 .enter().append("line")

 .style("display", function(d) { return d == 0 ? "none" : ""; })

 .attr("class", "tick")

 .attr("x1", function(d) { return xa(d); })

 .attr("x2", function(d) { return xa(d); })

 .attr("y2", audioHeight)

audio.selectAll(".segment")

 .data(labelsA)

 .enter().append("rect")

 .style("opacity", 0)

 .attr("class", "segment")

 .attr("transform", function(d) {

 })

 .attr("width", xa.step())

 .attr("height", audioHeight)

})();

Attention can also be used on the interface between a convolutional

 neural network and an RNN. This allows the RNN to look at different

position of an image every step. One popular use of this kind of

attention is for image captioning. First, a conv net processes the

image, extracting high-level features. Then an RNN runs, generating a

description of the image. As it generates each word in the description,

the RNN focuses on the conv net’s interpretation of the relevant parts

of the image. We can explicitly visualize this:

Figure from [3]

More broadly, attentional interfaces can be used whenever one wants

 to interface with a neural network that has a repeating structure in

its output.

Attentional interfaces have been found to be an extremely general

and powerful technique, and are becoming increasingly widespread.

---

Adaptive Computation Time

-------------------------

Standard RNNs do the same amount of

computation for each time step. This seems unintuitive. Surely, one

Adaptive Computation Time [15]

 is a way for RNNs to do different amounts of computation each step. The

 big picture idea is simple: allow the RNN to do multiple steps of

computation for each time step.

In order for the network to learn how many steps to do, we want the

 number of steps to be differentiable. We achieve this with the same

trick we used before: instead of deciding to run for a discrete number

of steps, we have an attention distribution over the number of steps to

run. The output is a weighted combination of the outputs of each step.

rnn

For every time step the RNN can do multiple computation steps.

The output is a weighted combination of the computation step outputs.

The process is repeated for each time step.

A special bit is set to denote the first computation step.

There are a few more details, which were left out in the previous

diagram. Here’s a complete diagram of a time step with three computation

 steps.

rnn

That’s a bit complicated, so let’s work through it step by step. At

 a high-level, we’re still running the RNN and outputting a weighted

combination of the states:

rnn

The weight for each step is determined by a “halting neuron.” It’s a

 sigmoid neuron that looks at the RNN state and gives a halting weight,

which we can think of as the probability that we should stop at that

step.

rnn

halting neuron

We have a total budget for the halting weights of 1, so we track

that budget along the top. When it gets to less than epsilon, we stop.

rnn

When we stop, might have some left over halting budget because we

stop when it gets to less than epsilon. What should we do with it?

Technically, it’s being given to future steps but we don’t want to

compute those, so we attribute it to the last step.

rnn

When training Adaptive Computation Time models, one adds a “ponder

cost” term to the cost function. This penalizes the model for the amount

 of computation it uses. The bigger you make this term, the more it will

 trade-off performance for lowering compute time.

### Code

---

Neural Programmer

-----------------

Neural nets are excellent at many tasks, but they also struggle to

do some basic things like arithmetic, which are trivial in normal

approaches to computing. It would be really nice to have a way to fuse

neural nets with normal programming, and get the best of both worlds.

The actual model in the paper answers questions about tables by

generating SQL-like programs to query the table. However, there are a

number of details here that make it a bit complicated, so let’s start by

 imagining a slightly simpler model, which is given an arithmetic

expression and generates a program to evaluate it.

The generated program is a sequence of operations. Each operation

is defined to operate on the output of past operations. So an operation

might be something like “add the output of the operation 2 steps ago and

 the output of the operation 1 step ago.” It’s more like a Unix pipe

than a program with variables being assigned to and read from.

rnn

The program is generated one operation at a time by a controller

RNN. At each step, the controller RNN outputs a probability distribution

 for what the next operation should be. For example, we might be pretty

sure we want to perform addition at the first time step, then have a

hard time deciding whether we should multiply or divide at the second

step, and so on...

rnn

At each step the controller RNN outputs a probability distribution.

The resulting distribution over operations can now be evaluated.

Instead of running a single operation at each step, we do the usual

attention trick of running all of them and then average the outputs

together, weighted by the probability we ran that operation.

rnn

At each step the controller RNN outputs a probability distribution.

We run all of the operations and average the outputs together.

As long as we can define derivatives through the operations, the

program’s output is differentiable with respect to the probabilities. We

 can then define a loss, and train the neural net to produce programs

that give the correct answer. In this way, the Neural Programmer learns

to produce programs without examples of good programs. The only

supervision is the answer the program should produce.

That’s the core idea of Neural Programmer, but the version in the

paper answers questions about tables, rather than arithmetic

expressions. There are a few additional neat tricks:

* **Multiple Types:** Many of the operations in the

Neural Programmer deal with types other than scalar numbers. Some

* **Referencing Inputs:** The neural programmer

needs to answer questions like “How many cities have a population

greater than 1,000,000?” given a table of cities with a population

column. To facilitate this, some operations allow the network to

reference constants in the question they’re answering, or the names of

columns. This referencing happens by attention, in the style of pointer

networks [17].

The Neural Programmer isn’t the only approach to having neural

networks generate programs. Another lovely approach is the Neural

We think that this general space, of bridging the gap between more

traditional programming and neural networks is extremely important.

While the Neural Programmer is clearly not the final solution, we think

there are a lot of important lessons to be learned from it.

### Code

---

The Big Picture

---------------

A human with a piece of paper is, in some sense, much smarter than a

 human without. A human with mathematical notation can solve problems

they otherwise couldn’t. Access to computers makes us capable of

incredible feats that would otherwise be far beyond us.

In general, it seems like a lot of interesting forms of

intelligence are an interaction between the creative heuristic intuition

 of humans and some more crisp and careful media, like language or

equations. Sometimes, the medium is something that physically exists,

and stores information for us, prevents us from making mistakes, or does

 computational heavy lifting. In other cases, the medium is a model in

our head that we manipulate. Either way, it seems deeply fundamental to

intelligence.

Recent results in machine learning have started to have this

flavor, combining the intuition of neural networks with something else.

One approach is what one might call “heuristic search.” For example,

 uses neural networks as intuition for manipulating mathematical

expressions. The “augmented RNNs” we’ve talked about in this article are

 another approach, where we connect RNNs to engineered media, in order

to extend their general capabilities.

Interacting with media naturally involves making a sequence of

taking an action, observing, and taking more actions. This creates a

major challenge: how do we learn which actions to take? That sounds like

 a reinforcement learning problem and we could certainly take that

approach. But the reinforcement learning literature is really attacking

the hardest version of this problem, and its solutions are hard to use.

The wonderful thing about attention is that it gives us an easier way

out of this problem by partially taking all actions to varying extents.

This works because we can design media—like the NTM memory—to allow

fractional actions and to be differentiable. Reinforcement learning has

us take a single path, and try to learn from that. Attention takes every

 direction at a fork, and then merges the paths back together.

A major weaknesses of attention is that we have to take every

“action” every step. This causes the computational cost to grow linearly

 as you do things like increase the amount of memory in a Neural Turing

Machine. One thing you could imagine doing is having your attention be

sparse, so that you only have to touch some memories. However, it’s

still challenging because you may want to do things like have your

attention depend on the content of the memory, and doing that naively

forces you to look at each memory. We’ve seen some initial attempts to

attack this problem, such as [21],

 but it seems like there’s a lot more to be done. If we could really

make such sub-linear time attention work, that would be very powerful!

Augmented recurrent neural networks, and the underlying technique

of attention, are incredibly exciting. We look forward to seeing what

happens next!

Attention and Augmented Recurrent Neural Networks.html


Chris Olah
Nick Cammarata
Chelsea Voss
Ludwig Schubert
Gabriel Goh

Abstract: This article is part of the Circuits thread,  an experimental format collecting invited short articles and critical  commentary delving into the inner workings of neural networks.

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

[Curve Detectors](https://distill.pub/2020/circuits/curve-detectors/)

### Contents

[Equivariant Features](#features)

* [Rotation](#rotation-features),

 [Scale](#scale-features),

 [Hue](#hue-features),

 [Hue-Rotation](#hue-rotation-features),

 [Reflection](#reflection-features),

 [Miscellaneous](#miscellaneous-features)

Equivariant Circuits

* [High-Low](#circuit-hilo),

 [Contrast→Center](#circuit-greenpurple),

 [BW-Color](#circuit-BW),

 [Line→Circle/Divergence](#circuit-circlestar),

 [Curve→Circle/Evolute](#circuit-circle-evolute),

 [Human-Animal](#circuit-person),

 [Invariant Dog Head](#circuit-dog),

 [Hue→Hue](#circuit-hue),

 [Curve→Curve](#circuit-curve),

 [Contrast→Line](#circuit-coloredge)

[Equivariant Architectures](#architectures)

[Conclusion](#conclusion)

Convolutional neural networks contain a hidden world of

symmetries within themselves. This symmetry is a powerful tool in

understanding the features and circuits inside neural networks. It also

suggests that efforts to design neural networks with additional

symmetries baked in (eg. ) may be on a promising track.

 that connect them.

 It turns out that many neurons are slightly transformed versions

of the same basic feature.

 This includes rotated copies of the same feature, scaled copies,

flipped copies, features detecting different colors, and much more.

 Before we talk about the examples introduced in this article, let’s

talk about how this definition maps to the classic example of

equivariance in neural networks: translation and convolutional neural

network nets. In a conv net, translating the input image is equivalent

to translating the neurons in the hidden layers (ignoring pooling,

 Now let’s consider the case of curve detectors (the first example in

the Equivariant Features section), which have ten rotated copies. In

 acts on the hidden layers by reorganizing the neurons so that their

orientations correspond to the appropriate rotations. This satisfies, at

 least approximately, the original definition of equivariance.

This transformed neuron form of equivariance is a special case of

equivariance. There are many ways a neural network could be equivariant

without having transformed versions of neurons. Conversely, we’ll also

see a number of examples of equivariance that don’t map exactly to the

group theory definition of equivariance: some have “holes” where a

transformed neuron is missing, while others consist of a set of

transformations that have a weaker structure than a group or don’t

correspond to a simple action on the image. But this general structure

remains.

Equivariance can be seen as a kind of ”[circuit

 on ImageNet, but we’ve observed at least some equivariance in every model

 trained on natural images we’ve studied.

---

Equivariant Features

--------------------

Rotational Equivariance

Curve Detectors

Rotational Equivariance (mod 180)

High-Low Frequency Detectors

Edge Detectors

Line Detectors

One can test that these are genuinely rotated versions of the

same feature by taking examples that cause one to fire, rotating them,

 on curve detectors tests their equivariance through several

experiments, including rotating stimuli that activate one neuron and

seeing how the others respond.

One way to verify that units like curve detectors

are truly rotated versions of the same feature is to take stimuli that

**Scale Equivariance:** Rotated versions

 aren’t the only kind of variation we see. It’s also quite common to see

 the same feature at different scales, although usually the scaled

Scale Equivariance

Circle Detectors

 units will detect one hue in the center, and the opposing hue on around

 it. Units can be found doing this up until the seventh or even eighth

layer of InceptionV1.

Hue Equivariance

Color Center

-Surround

 These units detect one hue on one side, and the opposite hue on the

other. As a result, they have variation in both hue and rotation. These

 variations are particularly interesting, because there’s an interaction

 between hue and rotation. But cycling hue by 180 degrees flips which

hue is on which side, and is so is equivalent to rotating by 180

degrees.

In the following diagram, we show orientation rotating the

whole 360 degrees, but hue only rotating 180. At the bottom of the

chart, it wraps around to the top but shifts by 180 degrees.

Rotational Equivariance

Hue Equivariance

Color Contrast Detectors

(hue+180, orientation) = (hue, orientation+180)

**Reflection Equivariance:** As we

move into the mid layers of the network, rotated variations become less

prominent, but horizontally flipped pairs become quite prevalent.

Horizontal Flip

Dog snout detectors

S-curve detectors

Human beside animal

Horizontal Flip

Horizontal Flip

**Miscellaneous Equivariance:**

Finally, we see variations of features transformed in other

miscellaneous ways. For example, short vs long-snouted versions of the

same dog head features, or human vs dog versions of the same feature. We

 even see units which are equivariant to camera perspective (found in a

Places365 model). These aren’t

necessarily something that we would classically think of as forms of

equivariance, but do seem to essentially be the same thing.

Snout Length

Human vs Dog

Perspective

---

Equivariant Circuits

--------------------

The equivariant behavior we observe in neurons is really a

reflection of a deeper symmetry that exists in the weights of neural

We’ll start by focusing on rotationally equivariant

features that are formed from rotationally invariant features. This

“invariant→equivariant” case is probably the simplest form of

equivariant circuit. Next, we’ll look at “equivariant→invariant”

circuits, and then finally the more complex “equivariant→equivariant”

circuits.

 get built from a high-frequency factor and a low-frequency factor (both

 factors correspond to a combination of neurons in the previous layer).

Each high-low frequency detector responds to a transition in frequency

in a given direction, detecting high-frequency patterns on one side, and

 low frequency patterns on the other. Notice how the same weight pattern

 rotates, making rotated versions of the feature.

positive (excitation)

negative (inhibition)

High-low frequency detectors

This makes them rotationally equivariant.

**Contrast→Center Circuit:** This

same pattern can be used in reverse to turn rotationally equivariant

features back into rotationally invariant features (an

“equivariant→invariant” circuit). In the following example, we see

Rotational equivariance can be turned into invariance with the transpose of an

invariant -> equivariant circuit.

color contrast unitscolor center surround units

positive (excitation)

negative (inhibition)

Sometimes we see one of these immediately follow the other:

equivariance be created, and then immediately partially used to create

invariant units.

**BW-Color Circuit:** In the following

example, a generic color factor and a black and white factor are used to

positive (excitation)

negative (inhibition)

positive (excitation)

negative (inhibition)

A  is created by detecting early edges  to a normal line.

circle detectorperpendicular

A  is created by detecting early edges  to a normal line.

diverging line detectorparallel

 This circuit is also an example of scale equivariance. The same general

 pattern which turns small curve detectors into a small circle detector

turns large curve detectors into a large circle detector. The same

pattern which turns medium curve detectors into a medium evolute

detector turns large curves into a large evolute detector.

conv2d -> mixed3a

Small circle from curves

mixed3a -> mixed3b

Medium evolute from curves

mixed3b -> mixed4a

Large circle and evolute from curves

**Human-Animal Circuit:** So far, all of

 the examples we’ve seen of circuits have involved rotation. These

human-animal and animal-human detectors are an example of horizontal

flip equivariance instead:

positive (excitation)

negative (inhibition)

Human detectors excite the human side of each unit and inhibit the other.

### “Equivariant→Equivariant” Circuits

The circuits we’ve looked at so far were either

“invariant→equivariant” or “equivariant→invariant.” Either they had

invariant input units, or invariant output units. Circuits of this form

are quite simple: the weights rotate, or flip, or otherwise transform,

but only in response to the transformation of a single feature. When we

look at “equivariant→equivariant” circuits, things become a bit more

complex. Both the input and output features transform, and we need to

consider the relative relationship between the two units.

**Hue→Hue Circuit:** Let’s start with a

 circuit connecting two sets of hue-equivariant center-surround

detectors. Each unit in the second layer is excited by the unit

selecting for a similar hue in the previous layer.

Each unit is excited by the unit with the same hue in the previous layer.

To understand the above, we need to focus on the relative

relationships between each input and output neuron — in this case, how

far the hues are apart on the color wheel. When they have the same hue,

the relationship is excitatory. When they have close but different hues,

 it’s inhibitory. And when they are very different, the weight is close

to zero. The units used to illustrate hue

equivariance here were selected to have a straightforward circuit. Other

 units may have more complex relationships. For example, some units

respond to a range of hues like yellow-red and have correspondingly more

 complex weights.

**Curve→Curve Circuit:** Let’s now

consider a slightly more complex example, how early curve detectors

 the curve detectors presented were selected to make the circuit as

simple and pedagogical as possible. They have clean weights and even

spacing between them, which will make the pattern easier to see. A

forthcoming article will discuss curve circuits in detail.

If we just look at the matrix of weights, it’s a bit hard

to understand. But if we focus on how each curve detector connects to

the earlier curve in the same and opposite orientations, it becomes

easier to see the structure. Rather than each curve being built from the

 same neurons in the previous layer, they shift. Each curve is excited

by curves in the same orientation and inhibited by those in the

opposite. At the same time, the spatial structure of the weights also

rotate.

 The general idea is line detectors should fire more strongly if there

are different colors on each side of the line. Conversely, they should

be inhibited by a change in color if it is perpendicular to the line.

Note that this is an “equivariant→equivariant” circuit with

 respect to rotation, but “equivariant→invariant” with respect to hue.

 roughly arranged by orientation and hue.

Color contrast detectors

horizontal line detector

vertical line detector

---

Equivariant Architectures

-------------------------

 Equivariance has a rich history in deep learning.

 Many important neural network architectures have equivariance at

 their core, and there is a very active thread of research around more

aggressively incorporating equivariance.

 But the focus is normally on designing equivariant

architectures, rather than “natural equivariance” we’ve discussed so

far.

 How should we think about the relationship between “natural” and

 “designed” equivariance?

 As we’ll see, there appears to be quite a deep connection.

 Historically, there has been some interesting back and forth

between the two.

 Researchers have often observed that many features in the first

layer of neural networks are transformed versions of one basic template.Features

 in the first layer of neural networks are much more often studied than

in other layers. This is because they are easy to study: you can just

visualize the weights to pixel values, or more generally to input

features.

 This naturally occurring equivariance in the first layer has

then sometimes been — and in other cases, easily could have

been — inspiration for the design of new architectures.

For example, if you train a fully-connected neural network

on a visual task, the first layer will learn variants of the same

features over and over: Gabor filters at different positions,

orientations, and scales. Convolutional neural networks changed this.

By baking the existence of translated copies of each feature directly

into the network architecture, they generally remove the need for the

network to learn translated copies of each feature. This resulted in a

massive increase in statistical efficiency, and became a cornerstone of

modern deep learning approaches to computer vision. But if we look at

the first layer of a well-trained convolutional neural network, we see

that other transformed versions of the same feature remain:

The weights for the units in the first layer of the TF-Slim

 version of InceptionV1 .We

 show the first layer conv weights of the tf-slim version of InceptionV1

 rather than the canonical one because its weights are cleaner. This is

likely due to the inclusion of batch-norm in the slim variant, causing

cleaner gradients. Units are sorted by the first principal

component of the adjacency matrix between the first and second layers.

Note how many features are similar except for rotation, scale, and hue.

Inspired by this, a 2011 paper subtitled “One Gabor to Rule Them All”

 created a sparse coding model which had a single Gabor filter

translated, rotated, and scaled. In more recent years, a number of

papers have extended this equivariance to the hidden layers of neural

networks, and to broader kinds of transformations .

 Just as convolutional neural networks enforce that the weights between

two features be the same if they have the same relative position:

… these more sophisticated equivariant networks make the

weights between two neurons equal if they have the same relative

relationship under more general transformations:

 For our purposes, it suffices to know that these equivariant

neural networks have the same weights when there is the same relative

relationship between neurons. This footnote is for the benefit of

readers who may wish to engage more deeply in the enforced equivariance

literature, and can be safely skipped.

 Group theory is an area of mathematics that gives us tools for

describing symmetries and sets of interacting transformations. To build

equivariant neural networks, we often borrow an idea from group theory

called a group convolution. Just as a regular convolution can describe

weights that correctly respect translational equivariance, a group

convolution can describe weights that respect a complex set of

interacting transformations (the group it operates over). Although you

could try to work out how to tie the weights to achieve this from first

principles, it’s easy to make mistakes. (One of the authors participated

 in many conversations with researchers in 2012 where people made errors

 on whiteboards about how sets of rotated and translated features should

 interact, without using convolutions.) Group convolutions can take any

group you describe and give you the correct weight tying.

 If you dig further, you may begin to see papers discussing

something called a group representation instead of group convolutions.

This is a more advanced topic in group theory. The core idea is

analogous to the Fourier transform. Recall that the Fourier transform

turns convolution into pointwise multiplication (this is sometimes used

to accelerate convolution). Well, the Fourier transform has a version

that can operate over functions on groups, and also maps convolution to

pointwise multiplication. And when you apply the Fourier transform to a

group, the resulting coefficients correspond to something called a group

 representation, which you can think of as being analogous to a

frequency in the regular Fourier transform.

This is, at least approximately, what we saw conv nets

naturally doing when we look at equivariant circuits! The weights had

symmetries that caused neurons with similar relationships to have

similar weights, much like an equivariant architecture would force them

to.

Given that we have neural network architectures which mimic

 the natural structures we observe, it seems natural to wonder what

features and circuits such models learn. Do they learn the same

equivariant features we see naturally form? Or do they do something

entirely different?

 To answer these questions, we trained an equivariant model roughly

 inspired by InceptionV1 on ImageNet. We made half the neurons

rotationally equivariant (with 16

rotations), and made the others rotationally invariant. Since we put no

effort into tuning it, the model achieved abysmal test accuracy but

still learns interesting features.

 Here are the full set of features learned by the equivariant

model. Half are forced to be rotationally equivariant, while half are

forced to be rotationally invariant.

 Looking at mixed3b, we found that the equivariant model learned

analogues of many large rotationally equivariant families from

[respond to curves in various orientations.

Curve detectors](https://distill.pub/2020/circuits/curve-detectors/)

Naturally ocurring rotationally equivariant features

[Oriented fur detectors

[Boundary detectors

[Divot detectors

 The existence of analogous features in equivariant models can be

 seen as a successful prediction of interpretability.

 As researchers engaged in more qualitative research, we should

always be worried that we may be fooling ourselves.

 Successfully predicting which features will form in an

equivariant neural network architecture is actually a pretty non-trivial

 prediction to make, and a nice confirmation that we’re correctly

understanding things.

 Another exciting possibility is that this kind of feature and

circuit analysis may be able to help inform equivariance research.

 For example, the kinds of equivariance that naturally form might

 be helpful in informing what types of equivariance we should design

into different layers of a neural network.

---

Conclusion

----------

Equivariance has a remarkable ability to simplify our

understanding of neural networks. When we see neural networks as

families of features, interacting in structured ways, understanding

small templates can actually turn into understanding how large numbers

of neurons interact. Equivariance is a big help whenever we discover it.

We sometimes think of understanding neural networks as

being like reverse engineering a regular computer program. In this

analogy, equivariance is like finding the same inlined function repeated

 throughout the code. Once you realize that you’re seeing many copies of

 the same function, you only need to understand it once.

But natural equivariance does have some limitations. For

starters, we have to find the equivariant families. This can actually

take us quite a bit of work, poring through neurons. Further, they may

not be exactly equivariant: one unit may be wired up slightly

differently, or have a small exception, and so understanding it as

equivariant could leave gaps in our understanding.

We’re excited about the potential of equivariant

architectures to make the features and circuits of neural networks

easier to understand. This seems especially promising in the context of

early vision, where the vast majority of features seem to be equivariant

 to rotation, hue, scale, or a combination of those.

One of the biggest — and least discussed — advantages we

have over neuroscientists in studying vision in artificial neural

networks instead of biological neural networks is translational

equivariance. By only having one neuron for each feature instead of tens

 of thousands of translated copies, convolutional neural networks

massively reduce the complexity of studying artificial vision systems

relative to biological ones. This has been a key ingredient in making it

 at all plausible that we can systematically understand InceptionV1.

Perhaps in the future, the right equivariant architecture

will be able to shave another order of magnitude of complexity off of

understanding early vision in neural networks. If so, understanding

early vision might move from “possible with effort” to “easily

achievable.”

 This article is part of the Circuits thread, a collection

of short articles and commentary by an open scientific collaboration

delving into the inner workings of neural networks.

[Curve Detectors](https://distill.pub/2020/circuits/curve-detectors/)

The statistical inefficiency of sparse coding for images (or, one Gabor to rule them all)

Group equivariant convolutional networks

Exploiting cyclic symmetry in convolutional neural networks

Steerable CNNs

Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds

3D G-CNNs for pulmonary nodule detection

An introduction to systems biology: design principles of biological circuits

Going deeper with convolutions

Imagenet: A large-scale hierarchical image database

Places: An image database for deep scene understanding

TF-Slim: A high level library to define complex models in TensorFlow

Naturally Occurring Equivariance in Neural Networks.html


High-Low Frequency Detectors

Ludwig Schubert
Chelsea Voss
Nick Cammarata
Gabriel Goh
Chris Olah

![](High-Low%20Frequency%20Detectors_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

[Curve Circuits](https://distill.pub/2020/circuits/curve-circuits/)

Introduction

------------

 for example, are a pretty natural feature for a vision system to have.

In fact, they had already been discovered in the animal visual cortex.

 It’s easy to imagine how curve detectors are built up from earlier edge

 detectors, and it’s easy to guess why curve detection might be useful

to the rest of the neural network.

 Fourier transform of an image.

[Function

----------](#function)

 How can we be sure that “high-low frequency detectors” are

actually detecting directional transitions from low to high spatial

frequency?

 We will rely on three methods:

 We will be able to understand the algorithm that implements them,

confirming that they detect high to low frequency transitions.

[### Feature Visualization](#feature-visualization)

 optimized to elicit maximal activation of a single, specific neuron.

 establishes a causal link! The behavior shown in the

 feature visualization is behavior that causes the neuron to fire:

 .gallery {

 display: grid;

 grid-template-columns: repeat(auto-fit, minmax(28px, 64px));

 grid-gap: 0.5rem;

 justify-content: start;

 }

 ul.gallery {

 padding-left: 0;

 }

 .gallery img,

 .gallery-img {

 max-width: 100%;

 width: unset;

 object-fit: none;

 object-position: center;

 border-radius: 8px;

 }

 @media screen and (min-width: 768px) {

 .gallery {

 grid-template-columns: repeat(7, minmax(28px, 96px));

 justify-content: left;

 }

 }

 @media screen and (min-width: 1180px) {

 .gallery {

 grid-gap: 1rem;

 }

 }

[![](High-Low%20Frequency%20Detectors_files/neuron-136.png "Unit 136")

[![](High-Low%20Frequency%20Detectors_files/neuron-108.png "Unit 108")

[![](High-Low%20Frequency%20Detectors_files/neuron-132.png "Unit 132")

[![](High-Low%20Frequency%20Detectors_files/neuron-88.png "Unit 88")

[![](High-Low%20Frequency%20Detectors_files/neuron-110.png "Unit 110")

[![](High-Low%20Frequency%20Detectors_files/neuron-180.png "Unit 180")

[![](High-Low%20Frequency%20Detectors_files/neuron-153.png "Unit 153")

[![](High-Low%20Frequency%20Detectors_files/neuron-186.png "Unit 186")

[![](High-Low%20Frequency%20Detectors_files/neuron-86.png "Unit 86")

[![](High-Low%20Frequency%20Detectors_files/neuron-117.png "Unit 117")

[![](High-Low%20Frequency%20Detectors_files/neuron-112.png "Unit 112")

[![](High-Low%20Frequency%20Detectors_files/neuron-70.png "Unit 70")

[![](High-Low%20Frequency%20Detectors_files/neuron-106.png "Unit 106")

[![](High-Low%20Frequency%20Detectors_files/neuron-113.png "Unit 113")

[1](#figure-1):

 characteristics:

* **Rotational equivariance.**

 each unit detects a high-low frequency change along a particular angle,

 with different units spanning the full 360º of possible orientations.

 in our feature visualizations to jointly optimize for the activation of

 a neuron while encouraging different activation patterns in a batch of

visualizations.

 We are thus reasonably confident that if high-low frequency

detectors were also sensitive to other patterns, we would see signs of

them in these feature visualizations. Instead, the frequency contrast

remains an invariant aspect of all these visualizations. (Although other

 patterns form along the boundary, these are likely outside the neuron’s

 effective receptive field.)

[1-2](#figure-1-2):

[### Dataset Examples](#dataset-examples)

 We generate dataset examples by sampling from a natural data

distribution (in this case, the training set) and selecting the images

that cause the neurons to maximally activate.

 Checking against these examples helps ensure we’re not misreading

the feature visualizations.

![](High-Low%20Frequency%20Detectors_files/placeholder.png)

[2](#1.2.0-dataset-examples):

 argmaxed over spatial locations.

A wide range of real-world situations can cause high-low

frequency detectors to fire. Oftentimes it’s a highly-textured, in-focus

 foreground object against a blurry background — for example, the

foreground might be the microphone’s latticework, the hummingbird’s tiny

 but not always: we also observe that it fires for the MP3 player’s

brushed metal finish against its shiny screen, or the text of a

watermark.

In all cases, we see one area with high frequency and another

area with low frequency. Although they often fire at an object boundary,

 they can also fire in cases where there is a frequency change

without an object boundary.

[### Synthetic Tuning Curves](#tuning-curves)

 To construct such a curve, we’ll need a set of *synthetic stimuli*

 which cause high-low frequency detectors to fire.

 We generate images with a high-frequency pattern on one side and a

 low-frequency pattern on the other. Since we’re interested in

orientation, we’ll rotate this pattern to create a 1D family of stimuli:

![](High-Low%20Frequency%20Detectors_files/orientation.png)

 The first axis of variation of our synthetic stimuli is *orientation*.

![](High-Low%20Frequency%20Detectors_files/ratio.png)

 The second axis of variation of our synthetic stimuli is the *frequency ratio*.

 : Responses of six high-low frequency detectors to artificial stimuli. The

 stimuli vary over two axes: orientation, ranging from 0 to 2π, and frequency

 ratio ƒ, ranging from 1:1 to 1:10. Activation values are normalized across all

 units shown, ranging between 0

 and 1.

[Implementation

--------------](#implementation)

How are high-low frequency detectors built up from lower-level

neurons?

 One could imagine many different circuits which could implement

this behavior. To give just one example, it seems like there are at

least two different ways that the oriented nature of these units could

form.

* **Equivariant→Equivariant Hypothesis.** The first possibility

 is that the previous layer already has precursor features which detect

oriented transitions from high frequency to low frequency. The extreme

version of this hypothesis would be that the high-low frequency detector

 is just an identity passthrough of some lower layer neuron. A more

moderate version would be something like what we see with curve

* **Invariant→Equivariant Hypothesis.** Alternatively, previous

 layers might not have anything like high-low frequency detectors.

Instead, the orientation might come from spatial arrangements in the

neuron’s weights that govern where it is excited by low-frequency and

high-frequency features.

 most of them can be roughly divided into two categories: those that

activate on the left and inhibit on the right, and those that do the

opposite.

[4](#figure-4):

 Six neurons from conv2d2 contributing weights to mixed3a 110.

 two clusters! One cluster appears to detect textures with a generally

high frequency, and one cluster appears to detect textures with a

generally low frequency.

[5](#figure-5):

 This is exactly what we would expect to see if the

Invariant→Equivariant hypothesis is true: each high-low frequency

detector composes the same two components in different spatial

arrangements, which then in turn govern the detector’s orientation.

 These two different clusters are really striking.

 In the next section, we’ll investigate them in more detail.

### High and Low Frequency Factors

It would be nice if we could confirm that these two clusters of

neurons are real. It would also be nice if we could create a simpler way

 to represent them for circuit analysis later.

 the downstream layers. We treat positive and negative influences separately.

 between lower layers and the high-low frequency detectors is one way

that we can check whether these two clusters are meaningful, and

investigate their significance. Performing a one-sided non-negative

Each factor corresponds to a vector over neurons. Feature

visualization can also be used to visualize these linear combinations of

 neurons. Strikingly, one clearly displays a generic high-frequency

 between conv2d2 and mixed3a. Because of this, we’re not here looking at

 direct weights between conv2d2 and mixed3a, but rather the “expanded

weights,” which are a product of a 1x1 convolution (which reduces down

to a small number of neurons) combined with a 3x3 convolution. This

structure is very similar to the factorization we apply. However, as we

see later in [Universality](#universality), we recover

similar factors for other models where this bottleneck doesn’t exist.

NMF makes it easy to see this abstract circuit across many models which

 .upstream-neurons {

 display: grid;

 grid-gap: 1em;

 margin-bottom: 1em;

 }

 h5 {

 margin-bottom: 0px;

 }

 .upstream-neurons .row {

 display: grid;

 grid-column-gap: .25em;

 column-gap: .25em;

 align-items: center;

 }

 .units,

 .weights {

 display: grid;

 grid-template-columns: repeat(6, 1fr);

 grid-gap: 0.5rem;

 grid-column-start: 3;

 }

 img.fv {

 display: block;

 max-width: 100%;

 border-radius: 8px;

 }

 img.full {

 width: unset;

 object-fit: none;

 object-position: center;

 image-rendering: optimizeQuality;

 }

 img.weight {

 width: 100%;

 image-rendering: pixelated;

 align-self: center;

 border: 1px solid #ccc;

 }

 .layer-label {

 grid-row-start: span 2;

 }

 .layer-label label {

 display: inline-block;

 /\* transform: rotate(-90deg); \*/

 }

 .annotation {

 font-size: 1.5em;

 font-weight: 200;

 color: #666;

 margin-bottom: 0.2em;

 }

 .equal-sign {

 padding: 0 0.25em;

 }

 .ellipsis {

 padding: 0 0.25em;

 /\* vertically align the ellipsis \*/

 position: relative;

 bottom: 0.5ex;

 }

 .unit {

 display: block;

 min-width: 50px;

 }

 .factor {

 box-shadow: 0 0 8px #888;

 }

 .unit .bar {

 display: block;

 margin-top: 0.5em;

 background-color: #CCC;

 height: 4px;

 }

 .row h4 {

 border-bottom: 1px solid #ccc;

 }

#### mixed3a → conv2d2

![](High-Low%20Frequency%20Detectors_files/conv2d2-hi.png)

=

+

+

+

+

+

…

HF-factor

 × 0.93

 × 0.73

 × 0.66

 × 0.59

 × 0.55

![](High-Low%20Frequency%20Detectors_files/conv2d2-lo.png)

=

+

+

+

+

+

…

LF-factor

 × 0.44

 × 0.41

 × 0.38

 × 0.36

 × 0.34

#### mixed3a → conv2d1

![](High-Low%20Frequency%20Detectors_files/conv2d1-hi.png)

=

+

+

+

+

+

…

HF-factor

 × 0.86

 × 0.81

 × 0.64

 × 0.53

 × 0.52

![](High-Low%20Frequency%20Detectors_files/conv2d1-lo.png)

=

+

+

+

+

+

…

LF-factor

 × 0.49

 × 0.48

 × 0.45

 × 0.43

 × 0.42

[6](#figure-6):

 left side of the equal sign shows feature visualizations of the NMF factors.

The feature visualizations are suggestive, but how can we be sure

 that these factors really correspond to high and low frequency in

general, rather than specific high or low frequency patterns? One thing

we can do is to create synthetic stimuli again, but now plotting the

responses of those two NMF factors.

 Since our factors don’t correspond to an edge, our synthetic stimuli

 will only have one frequency region for each stimulus. To add a second

dimension and again demonstrate robustness, we also vary the rotation of

 that region. (The frequency texture is not exactly rotationally

invariant because we construct the stimulus out of orthogonal cosine

waves.)

 : Responses to a grid of synthetic stimuli varying frequency and angle of

 orientation, against HF-factor and LF-factor from

 `conv2d2`

 . Orientation ranges from 0 to π/4. Wavelength as a proportion of the full

 input image ranges from 1:1 to 1:10. Activation values are normalized across

 all units shown, ranging from 0 to 1.

Unlike last time, these activations now mostly ignore the image’s

 orientation, but are sensitive to its frequency. We can average these

results over all orientations in order to produce a simple tuning curve

of how each factor responds to frequency. As predicted, the HF-factor

responds to high frequency and the LF-factor responds to low frequency.

[8](#figure-8):

 Tuning curve for HF-factor and LF-factor from `conv2d2`

 against images with synthetic frequency, averaged across orientation.

Wavelength as a proportion of the full input image ranges from 1:1 to

1:10.

### Construction of High-Low Frequency Detectors

 NMF factors the weights into both a channel factor and a spatial

factor. So far, we’ve looked at the two parts of the channel factor. The

 spatial factor shows the spatial weighting that combines the HF and LF

factors into high-low frequency detectors.

 also appears to be lightly activated by bright greens and magentas.

This might be responsible for the feature visualizations of these

high-low frequency detectors showing only greens and magentas on the

high-frequency side.

 .upstream-nmf {

 display: grid;

 grid-row-gap: 2rem;

 margin-bottom: 2rem;

 }

 .upstream-nmf .row {

 display: grid;

 grid-template-columns: min-content 1fr 6fr;

 grid-column-gap: 1rem;

 grid-row-gap: .5rem;

 }

 .units,

 .weights {

 display: grid;

 grid-template-columns: repeat(6, 1fr);

 grid-gap: 0.5rem;

 grid-column-start: 3;

 }

 img.fv {

 max-width: 100%;

 border-radius: 8px;

 }

 div.units img.full {

 margin-left: 1px;

 }

 img.full {

 width: unset;

 object-fit: none;

 object-position: center;

 image-rendering: optimizeQuality;

 }

 img.weight {

 width: 100%;

 image-rendering: pixelated;

 align-self: center;

 border: 1px solid #ccc;

 }

 .annotated-image {

 display: grid;

 grid-auto-flow: column;

 align-items: center;

 }

 .annotated-image span {

 writing-mode: vertical-lr;

 }

 .layer-label {

 grid-row-start: span 2;

 border-right: 1px solid #aaa;

 text-align: end;

 }

 .layer-label label {

 display: inline-block;

 margin-right: .5em;

 writing-mode: vertical-lr;

 }

 .layer-label.hidden {

 border-color: transparent;

 }

 .layer-label.hidden label {

 visibility: hidden;

 }

mixed3a

![](High-Low%20Frequency%20Detectors_files/neuron-136.png "Unit 136")

![](High-Low%20Frequency%20Detectors_files/neuron-108.png "Unit 108")

![](High-Low%20Frequency%20Detectors_files/neuron-132.png "Unit 132")

![](High-Low%20Frequency%20Detectors_files/neuron-88.png "Unit 88")

![](High-Low%20Frequency%20Detectors_files/neuron-110.png "Unit 110")

![](High-Low%20Frequency%20Detectors_files/neuron-180.png "Unit 180")

mixed3a → conv2d2

HF-factor

![](High-Low%20Frequency%20Detectors_files/conv2d2-hi.png)

![](High-Low%20Frequency%20Detectors_files/neuron136-layermaxpool1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron108-layermaxpool1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron132-layermaxpool1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron88-layermaxpool1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron110-layermaxpool1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron180-layermaxpool1-factor1.png)

LF-factor

![](High-Low%20Frequency%20Detectors_files/conv2d2-lo.png)

![](High-Low%20Frequency%20Detectors_files/neuron136-layermaxpool1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron108-layermaxpool1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron132-layermaxpool1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron88-layermaxpool1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron110-layermaxpool1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron180-layermaxpool1-factor0.png)

mixed3a → conv2d1

HF-factor

![](High-Low%20Frequency%20Detectors_files/conv2d1-hi.png)

![](High-Low%20Frequency%20Detectors_files/neuron136-layerconv2d1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron108-layerconv2d1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron132-layerconv2d1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron88-layerconv2d1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron110-layerconv2d1-factor1.png)

![](High-Low%20Frequency%20Detectors_files/neuron180-layerconv2d1-factor1.png)

LF-factor

![](High-Low%20Frequency%20Detectors_files/conv2d1-lo.png)

![](High-Low%20Frequency%20Detectors_files/neuron136-layerconv2d1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron108-layerconv2d1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron132-layerconv2d1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron88-layerconv2d1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron110-layerconv2d1-factor0.png)

![](High-Low%20Frequency%20Detectors_files/neuron180-layerconv2d1-factor0.png)

[9](#figure-9):

 two directly

 preceding convolutional layers, `conv2d2` and `conv2d1`.

Their spatial arrangement is very clear, with LF factors activating

 factors

 are very close to symmetric. Weight magnitudes normalized between -1  and 1.

High-low frequency detectors are therefore built up by circuits

that arrange high frequency detection on one side and low frequency

detection on the other.

 to be a solid positive stripe down the middle.

 What we instead observe is that it contributes as a component of

high frequency detection, though perhaps with a slight positive overall

bias.

![](High-Low%20Frequency%20Detectors_files/2d2-181-3a-70-weight.png)

[Usage

-----](#usage)

[`mixed3b`](https://distill.pub/2020/circuits/early-vision/#mixed3b)

 is the next layer immediately after the high-low frequency detectors.

Here, high-low frequency detectors contribute to a variety of features.

![](High-Low%20Frequency%20Detectors_files/usage-1.png)

[10](#figure-10):

 Oftentimes, downstream features appear to ignore the “polarity” of

 a high-low frequency detector, responding roughly the same way

regardless of which side is high frequency. For example, the vertical

 (see above) is strongly excited by high-low frequency detectors that

detect frequency change across a vertical line in either direction.

 Whereas activation from a high-low frequency detector can help

detect boundaries between different objects, inhibition from a high-low

frequency detector can also add structure to an object detector by

detecting regions that must be contiguous along some

direction — essentially, indicating the absence of a boundary.

![](High-Low%20Frequency%20Detectors_files/usage-2.png)

[11](#figure-11):

 with the highest L2-norm of weights across all high-low frequency

detectors, eight of those 20 neurons participate in boundary detection

#### Role in object boundary detection

 are neurons which detect boundaries between objects, whether that means

 the boundary between one object and another or the transition from

foreground to background. They are different from edge detectors or

curve detectors: although they are sensitive to edges (indeed, some of

their strongest weights are contributed by lower-level edge detectors!),

 object boundary detectors are also sensitive to other indicators such

as color contrast and high-low frequency detection.

![](High-Low%20Frequency%20Detectors_files/usage-boundary.png)

 orientation, and along a vertical line of positive weights.

 High-low frequency detectors contribute to these object boundary

 detectors by providing one piece of evidence that an object has ended

and something else has begun. Some examples of object boundary detectors

 are shown below, along with their weights to a selection of high-low

frequency detectors, grouped by orientation (ignoring polarity).

In particular, note how similar the weights are within each

grouping! This shows us again that the later layers ignore the high-low

frequency detectors’ polarity. Furthermore, the arrangement of

excitatory and inhibitory weights contributes to each boundary

detector’s overall shape, following the principles outlined above.

 /\* TODO: Optimize smaller breakpoints by hand \*/

![](High-Low%20Frequency%20Detectors_files/usage-boundaries.png)

 So far, the scope of our investigation has been limited to InceptionV1.

Universality

------------

### High-Low Frequency Detectors in Other Networks

 strong d-cite {

 font-weight: normal;

 }

 figure.columns {

 display: grid;

 grid-gap: 0.5em;

 grid-template-columns: repeat(4, 1fr);

 overflow: hidden;

 }

 .columns .model {

 display: grid;

 /\* position: relative;

 grid-row-gap: 0.5em;

 grid-auto-flow: row;

 align-content: start;

 grid-template-rows: min-content; \*/

 }

 .model picture {

 position: relative;

 }

 .model img {

 box-sizing: border-box;

 position: relative;

 width: unset;

 object-fit: none;

 z-index: 1;

 outline: 6px solid white;

 border-radius: 8px;

 }

 .model figcaption>\* {

 display: block;

 }

 /\*

 HAAAACKY

 \*/

 .model:first-of-type picture::before {

 content: "";

 display: block;

 position: absolute;

 height: 1px;

 width: 350%;

 left: 0;

 top: 48px;

 background-color: #aaa;

 z-index: 0;

 }

 .columns>figcaption {

 grid-column: 1 / span 4;

 }

**InceptionV1**

Layer `mixed3a`

At ~33% CNN depth

**AlexNet**

Layer `Conv2D_2`

At ~29% CNN depth

**InceptionV4**

Layer `Mixed_5a`

At ~33% CNN depth

**ResNetV2-50**

Layer `B2_U1_conv2`

At ~29% CNN depth

[14](#figure-14). High-low

frequency detectors that we’ve found in AlexNet, InceptionV4, and

ResnetV2-50 (right), compared to their most similar counterpart from

InceptionV1 (left). These are individual neurons, not linear

combinations approximating the detectors in InceptionV1.

 While the particular orientations each network’s high-low frequency

detectors respond to may vary slightly, each network has its own family

of detectors that together cover the full 360º and comprise a

 In the extreme case, one could imagine a synthetic dataset where

high-low frequency detectors don’t arise. For most practical datasets,

however, we expect to find them. For example, we even find some

Even though these families are from three completely different

networks, we also discover that their high-low frequency detectors are

built up from high and low frequency components.

### HF-factor and LF-factor in Other Networks

 As we did with InceptionV1, we can again perform NMF on the

weights of the high-low frequency detectors in each network in order to

extract the strongest two factors.

 .upstream-nmf {

 display: grid;

 grid-row-gap: 2rem;

 margin-bottom: 3rem;

 }

 .upstream-nmf .row {

 display: grid;

 grid-template-columns: min-content 1fr 6fr;

 grid-column-gap: 1rem;

 grid-row-gap: .5rem;

 }

 .units,

 .weights {

 display: grid;

 grid-template-columns: repeat(6, 1fr);

 grid-gap: 0.5rem;

 grid-column-start: 3;

 }

 img.fv {

 max-width: 100%;

 border-radius: 8px;

 }

 div.units img.full {

 margin-left: 1px;

 }

 img.full {

 width: unset;

 object-fit: none;

 object-position: center;

 image-rendering: optimizeQuality;

 }

 img.weight {

 width: 100%;

 image-rendering: pixelated;

 align-self: center;

 border: 1px solid #ccc;

 }

 img.factor {

 /\* padding-right: 0.75rem; \*/

 }

 .annotated-image {

 display: grid;

 grid-auto-flow: column;

 align-items: center;

 }

 .annotated-image span {

 writing-mode: vertical-lr;

 }

 .layer-label {

 grid-row-start: span 2;

 border-right: 1px solid #aaa;

 text-align: end;

 }

 .layer-label label {

 display: inline-block;

 margin-right: .5em;

 writing-mode: vertical-lr;

 }

 .layer-label.hidden {

 border-color: transparent;

 }

 .layer-label.hidden label {

 visibility: hidden;

 }

####  AlexNet

layer

Conv2D\_2 → conv1\_1

HF-factor

![](High-Low%20Frequency%20Detectors_files/AlexNet-hi.png)

![](High-Low%20Frequency%20Detectors_files/AlexNet-0-hi.png "Unit 55")

![](High-Low%20Frequency%20Detectors_files/AlexNet-1-hi.png "Unit 47")

![](High-Low%20Frequency%20Detectors_files/AlexNet-2-hi.png "Unit 87")

![](High-Low%20Frequency%20Detectors_files/AlexNet-3-hi.png "Unit 33")

![](High-Low%20Frequency%20Detectors_files/AlexNet-4-hi.png "Unit 77")

![](High-Low%20Frequency%20Detectors_files/AlexNet-5-hi.png "Unit 102")

LF-factor

![](High-Low%20Frequency%20Detectors_files/AlexNet-lo.png)

![](High-Low%20Frequency%20Detectors_files/AlexNet-0-lo.png "Unit 55")

![](High-Low%20Frequency%20Detectors_files/AlexNet-1-lo.png "Unit 47")

![](High-Low%20Frequency%20Detectors_files/AlexNet-2-lo.png "Unit 87")

![](High-Low%20Frequency%20Detectors_files/AlexNet-3-lo.png "Unit 33")

![](High-Low%20Frequency%20Detectors_files/AlexNet-4-lo.png "Unit 77")

![](High-Low%20Frequency%20Detectors_files/AlexNet-5-lo.png "Unit 102")

####  InceptionV3\_slim

layer

Conv2d\_4a → Conv2d\_3b

HF-factor

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-hi.png)

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-0-hi.png "Unit 82")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-1-hi.png "Unit 83")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-2-hi.png "Unit 137")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-3-hi.png "Unit 139")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-4-hi.png "Unit 155")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-5-hi.png "Unit 159")

LF-factor

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-lo.png)

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-0-lo.png "Unit 82")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-1-lo.png "Unit 83")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-2-lo.png "Unit 137")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-3-lo.png "Unit 139")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-4-lo.png "Unit 155")

![](High-Low%20Frequency%20Detectors_files/InceptionV3_slim-5-lo.png "Unit 159")

####  ResnetV2\_50\_slim

layer

B2\_U1\_conv2 → B2\_U1\_conv1

HF-factor

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-hi.png)

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-0-hi.png "Unit 118")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-1-hi.png "Unit 41")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-2-hi.png "Unit 53")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-3-hi.png "Unit 44")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-4-hi.png "Unit 25")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-5-hi.png "Unit 50")

LF-factor

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-lo.png)

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-0-lo.png "Unit 118")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-1-lo.png "Unit 41")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-2-lo.png "Unit 53")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-3-lo.png "Unit 44")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-4-lo.png "Unit 25")

![](High-Low%20Frequency%20Detectors_files/ResnetV2_50_slim-5-lo.png "Unit 50")

[15](#figure-15):

 NMF of high-low frequency detectors in

 AlexNet’s

 InceptionV3\_slim’s

 and

 ResnetV2\_50\_slim’s

 showing activations and inhibitions.

The feature visualizations of the two factors reveal one clear

HF-factor and one clear LF-factor, just like what we found in

InceptionV1. Furthermore, the weights on the two factors are again very

close to symmetric.

Our earlier conclusions therefore also hold across these

different networks: high-low frequency detectors are built up from the

specific spatial arrangement of a high frequency component and a low

frequency component.

Conclusion

----------

Although high-low frequency detectors represent a feature that we

 didn’t necessarily expect to find in a neural network, we find that we

can still explore and understand them using the interpretability tools

we’ve built up for exploring circuits: NMF, feature visualization,

synthetic stimuli, and more.

We’ve also learned that high-low frequency detectors are

built up from comprehensible lower-level parts, and we’ve shown how they

 contribute to later, higher-level features.

 Finally, we’ve seen that high-low frequency detectors are common

across multiple network architectures.

Given the universality observations, we might wonder whether the

existence of high-low frequency detectors isn’t so unnatural after all.

 high-low frequency detectors in AlexNet Places, with its substantially

different training data. Beyond neural networks, the aesthetic quality

imparted by the blurriness of an out-of-focus region of an image is

 And in VR, visual blur can either provide an effective depth-of-field

cue or, conversely, can induce nausea in the user when implemented in a

dissonant way. Perhaps frequency detection might well be commonplace in

both natural and artificial vision systems as yet another type of

informational cue.

Nevertheless, whether their existence is natural or not, we find

that high-low frequency detectors are possible to characterize and

understand.

![](High-Low%20Frequency%20Detectors_files/multiple-pages.svg)

 This article is part of the Circuits thread, a collection of

short articles and commentary by an open scientific collaboration

delving into the inner workings of neural networks.

[Curve Circuits](https://distill.pub/2020/circuits/curve-circuits/)

Complex pattern selectivity in macaque primary visual cortex revealed by large-scale two-photon imaging

Shape representation in area V4: position-specific tuning for boundary conformation

Discrete neural clusters encode orientation, curvature and corners in macaque V4

An Overview of Early Vision in InceptionV1

Going deeper with convolutions

Zoom In: An Introduction to Circuits

Visualizing higher-layer features of a deep network

Visualizing and understanding convolutional networks

Understanding neural networks through deep visualization

Visualizing and understanding recurrent networks

Feature Visualization

ImageNet: A large-scale hierarchical image database

Receptive fields, binocular interaction and functional architecture in the cat's visual cortex

Convergent learning: Do different neural networks learn the same representations?

ImageNet Classification with Deep Convolutional Neural Networks

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Deep Residual Learning for Image Recognition