Computing Receptive Fields of Convolutional Neural Networks

 While deep neural networks have overwhelmingly established state-of-the-art

 results in many artificial intelligence problems, they can still be

 difficult to develop and debug.

 Recent research on deep learning understanding has focused on

 feature visualization ,

 theoretical guarantees ,

 model interpretability ,

 and generalization .

 In this work, we analyze deep neural networks from a complementary

 perspective, focusing on convolutional models.

 We are interested in understanding the extent to

 which input signals may affect output features, and mapping

 features at any part of the network to the region in the input that

 produces them. The key parameter to associate an output feature to an input

 region is the *receptive field* of the convolutional network, which is

 defined as the size of the region in the input that produces the feature.

 As our first contribution, we

 present a mathematical derivation and an efficient algorithm to compute

 receptive fields of modern convolutional neural networks.

 Previous work

 discussed receptive field

 computation for simple convolutional

 networks where there is a single path from the input to the output,

 providing recurrence equations that apply to this case.

 In this work, we revisit these derivations to obtain a closed-form

 expression for receptive field computation in the single-path case.

 Furthermore, we extend receptive field computation to modern convolutional

 networks where there may be multiple paths from the input to the output.

 To the best of our knowledge, this is the first exposition of receptive

 field computation for such recent convolutional architectures.

 for the computer vision task of object detection, it is important

 understanding a convolutional feature's span is often required for that goal

 As our second contribution, we fill the void by introducing an

 [open-source library](https://github.com/google-research/receptive_field)

 can be easily employed to analyze a variety of models,

 as presented in this article.

 leading to more productive machine learning research.

Overview of the article

-----------------------

 We consider fully-convolutional neural networks, and derive their receptive

 input signal.

 appropriate.

 First, we derive closed-form expressions when the network has a

 single path from input to output (as in

 AlexNet

 or

 VGG ). Then, we discuss the

 more general case of arbitrary computation graphs with multiple paths from the

 input to the output (as in

 ResNet

 or

 Inception ). We consider

 potential alignment issues that arise in this context, and explain

 an algorithm to compute the receptive field size and locations.

 using our open-source library.

Problem setup

-------------

 output of the lll-th layer, with height hlhlh\_l, width wlwlw\_l and depth

 dldld\_l. We denote the input image by f0f0f\_0. The final output feature map

 corresponds to fLfLf\_{L}.

 111-dimensional input signals and feature maps. For higher-dimensional signals

 (e.g., 222D images), the

* klklk\_l: kernel size (positive integer)

* slsls\_l: stride (positive integer)

* plplp\_l: padding applied to the left side of the input feature map

 (non-negative integer)

 A more general definition of padding may also be considered: negative

 padding, interpreted as cropping, can be used in our derivations

 without any changes. In order to make the article more concise, our

 presentation focuses solely on non-negative padding.

* qlqlq\_l: padding applied to the right side of the input feature map

 (non-negative integer)

 Sorry, your browser does not support inline SVG.

Kernel Size (kl): 2

Left Padding (pl): 1

Right Padding (ql): 1

Stride (sl): 3

fl

fl-1

kl

kl

kl

sl

pl

ql

 We consider layers whose output features depend locally on input features:

 e.g., convolution, pooling, or elementwise operations such as non-linearities,

 addition and filter concatenation. These are commonly used in state-of-the-art

 networks. We define elementwise operations to

 have a "kernel size" of 111, since each output feature depends on a single

 location of the input feature maps.

 Our notation is further illustrated with the simple

 network below. In this case, L=4L=4L=4 and the model consists of a

 convolution, followed by ReLU, a second convolution and max-pooling.

 We adopt the convention where the first output feature for each layer is

 computed by placing the kernel on the left-most position of the input,

 including padding. This convention is adopted by all major deep learning

 libraries.

f0

f1

f2

f3

f4

**Convolution**

 Kernel Size (k1): 3

 Padding (p1, q1): 1

 Stride (s1): 2

Invalid setup! Decrease kernel

size or increase padding.

**ReLU**

 Kernel Size (k2): 1

 Padding (p2, q2): 0

 Stride (s2): 1

**Convolution**

 Kernel Size (k3): 1

 Padding (p3, q3): 0

 Stride (s3): 1

Invalid setup! Decrease kernel

size or increase padding.

**Max Pooling**

 Kernel Size (k4): 3

 Padding (p4, q4): 1

 Stride (s4): 2

Invalid setup! Decrease kernel

size or increase padding.

Single-path networks

--------------------

 In this section, we compute recurrence and closed-form expressions for

 fully-convolutional networks with a single path from input to output

 (e.g.,

 AlexNet

 or

 VGG ).

### Computing receptive field size

 Define rlrlr\_l as the receptive field size of

 other words, rlrlr\_l corresponds to the number of features in feature map

 flflf\_l which contribute to generate one feature in fLfLf\_{L}. Note

 that rL=1rL=1r\_{L}=1.

 input, and generates fLfLf\_{L} as output. Here is an illustration:

 Sorry, your browser does not support inline SVG.

Kernel Size (kL): 2

Padding (pL, qL): 0

Stride (sL): 3

kL

kL

fL-1

fL

 It is easy to see that kLkLk\_{L}

 feature from fLfLf\_{L} is directly connected to kLkLk\_{L} features from

 fL−1fL−1f\_{L-1}. So, rL−1=kLrL−1=kLr\_{L-1} = k\_{L}.

 fl−1fl−1f\_{l-1}.

 entire region where the

 which is why it needs to be deducted.

 As in the illustration below, note that, in some cases, the receptive

 field region may contain "holes", i.e., some of the input features may be

 unused for a given layer.

 Sorry, your browser does not support inline SVG.

Kernel Size (kl): 1

Padding (pl, ql): 0

Stride (sl): 3

sl⋅rl

sl-1

fl-1

fl

rl=2

 will cover those from the left and the right of the region. For example, if we

 on each side, adding 444 in total. If klklk\_l is even, this works as well,

 since the left and right padding will add to kl−1kl−1k\_l-1.

 Due to border effects, note that the size of the region in the original

 image which is used to compute each output feature may be different. This

 happens if padding is used, in which case the receptive field for

 border features includes the padded region. Later in the article, we

 discuss how to compute the receptive field region for each feature,

 which can be used to determine exactly which image pixels are used for

 each output feature.

 Sorry, your browser does not support inline SVG.

Kernel Size (kl): 5

Padding (pl, ql): 0

Stride (sl): 3

sl⋅rl - (sl-1)

(kl-1)/2

(kl-1)/2

fl-1

fl

rl=2

 So, we obtain the general recurrence equation (which is

 [first-order,

 non-homogeneous, with variable

 ):

rl−1=sl⋅rl+(kl−sl) (1)(1)rl−1=sl⋅rl+(kl−sl) egin{align}

 r\_{l-1} = s\_l \cdot r\_{l} + (k\_l - s\_l)

 \label{eq:rf\_recurrence}
 \end{align}

 This equation can be used in a recursive algorithm to compute the receptive

 slsls\_l's:

r0=L∑l=1((kl−1)l−1∏i=1si)+1(2)(2)r0=∑l=1L((kl−1)∏i=1l−1si)+1egin{equation}

 r\_0 = \sum\_{l=1}^{L} \left((k\_l-1)\prod\_{i=1}^{l-1}

 s\_i
ight) + 1 \label{eq:rf\_recurrence\_final} \end{equation}

 This expression makes intuitive sense, which can be seen by considering some

 special cases. For example, if all kernels are of size 1, naturally the

 receptive field is also of size 1. If all strides are 1, then the receptive

 increases proportionally for all layers below that one. Finally, note that

 padding does not need to be taken into account for this derivation.

### Computing receptive field region in input image

 in the output feature map, in many cases it is also critical to precisely

 localize the region which generated a feature. For example, given feature

 is addressed in this section.

 Let's denote ululu\_l and vlvlv\_l the left-most

 each map is at coordinate 000).

 Note that uL=vLuL=vLu\_{L} = v\_{L} corresponds to the

 location of the desired feature in fLfLf\_{L}. The figure below illustrates a

 simple 2-layer network, where we highlight the region in f0f0f\_0 which is used

 to compute the first feature from f2f2f\_2. Note that in this case the region

 u0=−1,v0=4u0=−1,v0=4u\_0=-1, v\_0=4.

 Sorry, your browser does not support inline SVG.

Kernel Size (k1): 3

Padding (p1, q1): 1

Stride (s1): 3

Kernel Size (k2): 2

Padding (p2, q2): 0

Stride (s2): 1

u0 = -1

v0 = 4

f0

f1

f2

 compute ul−1,vl−1ul−1,vl−1u\_{l-1},v\_{l-1}?

 clearly be located at −pl−pl-p\_l, since the first feature will be generated by

 u\_{l-1}&= -p\_l + u\_{l}\cdot s\_l \label{eq:rf\_loc\_recurrence\_u} \

 v\_{l-1}&= -p\_l + v\_{l}\cdot s\_l + k\_l -1

 \label{eq:rf\_loc\_recurrence\_v}

 \end{align}

 needed since in this case we want to find the right-most position.

 Note that these expressions are very similar to the recursion derived for the

 network parameters:

u0=uLL∏i=1si−L∑l=1pll−1∏i=1si(5)(5)u0=uL∏i=1Lsi−∑l=1Lpl∏i=1l−1siegin{align}

 u\_0&= u\_{L}\prod\_{i=1}^{L}s\_i - \sum\_{l=1}^{L}

 p\_l\prod\_{i=1}^{l-1} s\_i

 \label{eq:rf\_loc\_recurrence\_final\_left}

 \end{align}

 and of the feature location in the output feature map (uLuLu\_{L}).

 And for the right-most feature location v0v0v\_0:

 v\_0&= v\_{L}\prod\_{i=1}^{L}s\_i -\sum\_{l=1}^{L}(1 + p\_l -

 k\_l)\prod\_{i=1}^{l-1} s\_i

 \label{eq:rf\_loc\_recurrence\_final\_right}

 \end{align}

 expression also depends on the kernel sizes (klklk\_l) of each layer.

**Relation between receptive field size and region.**

 You may be wondering that

 the receptive field size r0r0r\_0 must be directly related to u0u0u\_0 and

 1, which we leave as a follow-up exercise for the curious reader. To

 emphasize, this means that we can rewrite

v0=u0+r0−1(7)(7)v0=u0+r0−1egin{align}

 v\_0&= u\_0 + r\_0 - 1

 \label{eq:rf\_loc\_recurrence\_final\_right\_rewrite}

 \end{align}

**Effective stride and effective padding.**

 To compute u0u0u\_0 and v0v0v\_0 in practice, it

 is convenient to define two other variables, which depend only on the paddings

 and strides of the different layers:

* *effective stride*

Sl=∏Li=l+1siSl=∏i=l+1LsiS\_l = \prod\_{i=l+1}^{L}s\_i: the stride between a

 given feature map flflf\_l and the output feature map fLfLf\_{L}

* *effective padding*

 the padding between a given feature map flflf\_l and the output feature map

 fLfLf\_{L}

 as:

u0=−P0+uL⋅S0(8)(8)u0=−P0+uL⋅S0egin{align}

 u\_0&= -P\_0 + u\_{L}\cdot S\_0

 \label{eq:rf\_loc\_recurrence\_final\_left\_effective}

 \end{align}

 the output feature map uLuLu\_{L}. When one is interested in computing feature

 locations for a given network, it is handy to pre-compute three variables:

 mapping from any output feature location to the input region which influences

 it.

 It is also possible to derive recurrence equations for the effective stride and

 effective padding. It is straightforward to show that:

Sl−1=sl⋅SlPl−1=sl⋅Pl+pl(9)(10)(9)Sl−1=sl⋅Sl(10)Pl−1=sl⋅Pl+plegin{align}

 S\_{l-1}&= s\_l \cdot S\_l \label{eq:effective\_stride\_recurrence} \

 P\_{l-1}&= s\_l \cdot P\_l + p\_l \label{eq:effective\_padding\_recurrence}

 \end{align}

 These expressions will be handy when deriving an algorithm to solve the case

 for arbitrary computation graphs, presented in the next section.

**Center of receptive field region.**

 It is also interesting to derive an

 expression for the center of the receptive field region which influences a

 particular output feature. This can be used as the location of the feature in

 the input image (as done for recent

 deep learning-based local features , for

 example).

 We define the center of the receptive field region for each layer lll as

 c\_0&= u\_{L}\prod\_{i=1}^{L}s\_i

 - \sum\_{l=1}^{L}

 - \sum\_{l=1}^{L}

 \left(p\_l - rac{k\_l - 1}{2}
ight)\prod\_{i=1}^{l-1} s\_i

onumber \&= -P\_0 + u\_{L}\cdot S\_0 + \left(rac{r\_0 - 1}{2}
ight)

 \label{eq:rf\_loc\_recurrence\_final\_center\_effective}

 \end{align}

 This expression can be compared to

 Note that the receptive field centers for the different output features are

 receptive field regions for the output features will be aligned to the first

 case all klklk\_l's must be odd).

**Other network operations.**

 The derivations provided in this section cover most basic operations at the

 core of convolutional neural networks. A curious reader may be wondering

 about other commonly-used operations, such as dilation, upsampling, etc. You

 can find a discussion on these [in the appendix](#other-network-operations).

Arbitrary computation graphs

----------------------------

 Most state-of-the-art convolutional neural networks today (e.g.,

 ResNet  or

 Inception ) rely on models

 where each layer may have more than one input, which

 means that there might be several different paths from the input image to the

 final output feature map. These architectures are usually represented using

 directed acyclic computation graphs, where the set of nodes LL\mathcal{L}

 represents the layers and the set of edges EE\mathcal{E} encodes the

 connections between them (the feature maps flow through the edges).

 The computation presented in the previous section can be used for each of the

 possible paths from input to output independently. The situation becomes

 trickier when one wants to take into account all different paths to find the

 receptive field size of the network and the receptive field regions which

 correspond to each of the output features.

**Alignment issues.**

 The first potential issue is that one output feature may

 be computed using misaligned regions of the input image, depending on the

 used for the computation of each output feature may vary. As a consequence,

 **the receptive field size may not be shift-invariant**

  . This is illustrated in the

 figure below with a toy example, in which case the centers of the regions used

 in the input image are different for the two paths from input to output.

 Sorry, your browser does not support inline SVG.

Kernel Size (k1): 5

Left Pad (p1): 2

Right Pad (q1): 1

Stride (s1): 2

Kernel Size (k2): 3

Left Pad (p2): 0

Right Pad (q2): 0

Stride (s2): 1

Kernel Size (k3): 3

Left Pad (p3): 0

Right Pad (q3): 0

Stride (s3): 1

Add

 are convolutional, while the last layer performs a simple addition.

 The relative position between the receptive field regions of the left and

 right paths is inconsistent for different output features, which leads to a

 Also, note that the receptive field size for each output

 feature may be different. For the second feature from the left, 666 input

 samples are used, while only 555 are used for the third feature. This means

 aligned.

 surface normal estimation, colorization, etc), local feature matching and

 retrieval, among others.

 When the network is aligned, all different paths lead to output features being

 centered consistently in the same locations. All different paths must have the

 same effective stride. It is easy to see that the receptive field size will be

 the largest receptive field among all possible paths. Also, the effective

 padding of the network corresponds to the effective padding for the path with

 largest receptive field size, such that one can apply

 generated an output feature.

 The figure below gives one simple example of an aligned network. In this case,

 the two different paths lead to the features being centered at the same

 locations. The receptive field size is 333, the effective stride is 444 and

 the effective padding is 111.

 Sorry, your browser does not support inline SVG.

Kernel Size (k1): 1

Left Pad (p1): 0

Right Pad (q1): 0

Stride (s1): 4

Kernel Size (k2): 3

Left Pad (p2): 1

Right Pad (q2): 0

Stride (s2): 2

Kernel Size (k3): 1

Left Pad (p3): 0

Right Pad (q3): 0

Stride (s3): 2

Add

**Alignment criteria**

  . More precisely, for a network to be aligned at every

 layer, we need every possible pair of paths iii and jjj to have

 this to happen, we can see from

 satisfied:

 S\_l^{(i)}&= S\_l^{(j)} \label{eq:align\_crit\_1} \

 \label{eq:align\_crit\_2}

 \end{align}

for all i,j,li,j,li,j,l.

**Algorithm for computing receptive field parameters: sketch.**

 field size and associated parameters for such computation graphs.

 Naturally, a brute-force approach is to use the expressions presented above to

 This method has a worst-case complexity of

 But we can do better. Start by topologically sorting the computation graph.

 The sorted representation arranges the layers in order of dependence: each

 layer's output only depends on layers that appear before it.

 By visiting layers in reverse topological order, we ensure that all paths

 from a given layer lll to the output layer LLL have been taken into account

 when lll is visited. Once the input layer l=0l=0l=0 is reached, all paths

 have been considered and the receptive field parameters of the entire model

 are obtained. The complexity of this algorithm is

 which is much better than the brute-force alternative.

 As each layer is visited, some bookkeeping must be done in order to keep

 track of the network's receptive field parameters. In particular, note that

 there might be several different paths from layer lll to the output layer

 LLL. In order to handle this situation, we keep track of the parameters

 for lll and update them if a new path with larger receptive field is found,

Discussion: receptive fields of modern networks

-----------------------------------------------

 In this section, we present the receptive field parameters of modern

 convolutional networks

 are drawn from the [TF-Slim

  , which were computed using the new open-source

 library (script

 The pre-computed parameters for

 AlexNet ,

 VGG ,

 ResNet ,

 Inception

 and

 MobileNet

 are presented in the table below.

 For a more

 comprehensive list, including intermediate network end-points, see

 [this

| --- | --- | --- | --- | --- |

| alexnet\_v2 | 195 | 32 | 64 | [2014](https://arxiv.org/abs/1404.5997v2) |

| vgg\_16 | 212 | 32 | 90 | [2014](https://arxiv.org/abs/<PHONE>) |

| mobilenet\_v1 | 315 | 32 | 126 | [2017](https://arxiv.org/abs/1704.04861) |

| resnet\_v1\_50 | 483 | 32 | 239 | [2015](https://arxiv.org/abs/1512.03385) |

| inception\_v2 | 699 | 32 | 318 | [2015](https://arxiv.org/abs/1502.03167) |

| resnet\_v1\_101 | 1027 | 32 | 511 | [2015](https://arxiv.org/abs/1512.03385) |

| inception\_v3 | 1311 | 32 | 618 | [2015](https://arxiv.org/abs/1512.00567) |

| resnet\_v1\_152 | 1507 | 32 | 751 | [2015](https://arxiv.org/abs/1512.03385) |

| resnet\_v1\_200 | 1763 | 32 | 879 | [2015](https://arxiv.org/abs/1512.03385) |

| inception\_v4 | 2071 | 32 | 998 | [2016](https://arxiv.org/abs/1602.07261) |

 As models evolved, from

 AlexNet, to VGG, to ResNet and Inception, the receptive fields increased

 (which is a natural consequence of the increased number of layers).

 includes all of the input pixels.

 We can also relate the growth in receptive fields to increased

 classification accuracy. The figure below plots ImageNet

 top-1 accuracy as a function of the network's receptive field size, for

 the same networks listed above. The circle size for each data point is

 proportional to the number of floating-point operations (FLOPs) for each

 architecture.

 google.charts.load('current', { 'packages': ['corechart'] });

 google.charts.setOnLoadCallback(drawSeriesChart);

 function drawSeriesChart() {

 var data = google.visualization.arrayToDataTable([

 ['alexnet\_v2', 195, 0.5720, 'alexnet', 1.38],

 ['vgg\_16', 212, 0.7150, 'vgg\_16', 30.71],

 ['inception\_v2', 699, 0.7390, 'inception', 3.88],

 ['inception\_v3', 1311, 0.7800, 'inception', 5.69],

 ['inception\_v4', 2071, 0.8020, 'inception', 12.27],

 ['inception\_resnet\_v2', 3039, 0.8040, 'inception\_resnet', 12.96],

 ['resnet\_v1\_50', 483, 0.7520, 'resnet', 6.97],

 ['resnet\_v1\_101', 1027, 0.7640, 'resnet', 14.40],

 ['resnet\_v1\_152', 1507, 0.7680, 'resnet', 21.82],

 ['mobilenet\_v1', 315, 0.7090, 'mobilenet', 1.14]

 ]);

 var options = {

 title: '',

 hAxis: { title: 'Receptive field size (pixels)', gridlines: { count: 10 } },

 vAxis: { title: 'ImageNet top-1 accuracy', format: 'percent' },

 sizeAxis: { maxSize: 10 },

 legend: { position: 'bottom', textStyle: { fontSize: 12 } },

 bubble: { textStyle: { fontSize: 11 } }

 };

 chart.draw(data, options);

 }

| --- | --- | --- | --- | --- |

| alexnet\_v2 | 195 | 0.572 | alexnet | 1.38 |

| vgg\_16 | 212 | 0.715 | vgg\_16 | 30.71 |

| inception\_v2 | 699 | 0.739 | inception | 3.88 |

| inception\_v3 | 1,311 | 0.78 | inception | 5.69 |

| inception\_v4 | 2,071 | 0.802 | inception | 12.27 |

| inception\_resnet\_v2 | 3,039 | 0.804 | inception\_resnet | 12.96 |

| resnet\_v1\_50 | 483 | 0.752 | resnet | 6.97 |

| resnet\_v1\_101 | 1,027 | 0.764 | resnet | 14.4 |

| resnet\_v1\_152 | 1,507 | 0.768 | resnet | 21.82 |

| mobilenet\_v1 | 315 | 0.709 | mobilenet | 1.14 |

0/0

 We observe a logarithmic relationship between

 classification accuracy and receptive field size, which suggests

 that large receptive fields are necessary for high-level

 recognition tasks, but with diminishing rewards.

 For example, note how MobileNets achieve high recognition performance even

 if using a very compact architecture: with depth-wise convolutions,

 the receptive field is increased with a small compute footprint.

 In comparison, VGG-16 requires 27X more FLOPs than MobileNets, but produces

 a smaller receptive field size; even if much more complex, VGG's accuracy

 is only slightly better than MobileNet's.

 This suggests that networks which can efficiently generate large receptive

 fields may enjoy enhanced recognition performance.

 residual connections, batch normalization, to name only a few.

 In other words, while we conjecture that a large receptive field is necessary,

 by no means it is sufficient.

 Additional experimentation is needed to confirm this hypothesis: for

 example, researchers may experimentally investigate how classification

 accuracy changes as kernel sizes and strides vary for different

 architectures. This may indicate if, at least for those architectures, a

 large receptive field is necessary.

 the feature, and consequently carry more weight.

 The relative importance of each input pixel defines the

 *effective receptive field* of the feature.

 Recent work

 provides a mathematical formulation and a procedure to measure effective

 receptive fields, experimentally observing a Gaussian shape,

 with the peak at the receptive field center. Better understanding the

 relative importance of input pixels in convolutional neural networks is

 an active research topic.

Visualizing and Understanding Convolutional Networks

Global Optimality in Neural Network Training

On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport

Object Detectors Emerge in Deep Scene CNNs

Interpreting Deep Visual Representations via Network Dissection

In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Understanding Deep Learning Requires Rethinking Generalization

A Guide to Receptive Field Arithmetic for Convolutional Neural Networks

What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?

ImageNet Classification with Deep Convolutional Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Deep Residual Learning for Image Recognition

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Large-Scale Image Retrieval with Attentive Deep Local Features

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks


Attention and Augmented Recurrent Neural Networks

=================================================

 dt-byline {

 font-size: 12px;

 line-height: 18px;

 display: block;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 color: rgba(0, 0, 0, 0.5);

 padding-top: 12px;

 padding-bottom: 12px;

 }

 dt-article.centered dt-byline {

 text-align: center;

 }

 dt-byline a,

 dt-article dt-byline a {

 text-decoration: none;

 border-bottom: none;

 }

 dt-article dt-byline a:hover {

 text-decoration: underline;

 border-bottom: none;

 }

 dt-byline .authors {

 text-align: left;

 }

 dt-byline .name {

 display: inline;

 text-transform: uppercase;

 }

 dt-byline .affiliation {

 display: inline;

 }

 dt-byline .date {

 display: block;

 text-align: left;

 }

 dt-byline .year, dt-byline .month {

 display: inline;

 }

 dt-byline .citation {

 display: block;

 text-align: left;

 }

 dt-byline .citation div {

 display: inline;

 }

 @media(min-width: 768px) {

 dt-byline {

 }

 }

 @media(min-width: 1080px) {

 dt-byline {

 border-bottom: none;

 margin-bottom: 70px;

 }

 dt-byline a:hover {

 color: rgba(0, 0, 0, 0.9);

 }

 dt-byline .authors {

 display: inline-block;

 }

 dt-byline .author {

 display: inline-block;

 margin-right: 12px;

 /\*padding-left: 20px;\*/

 /\*border-left: 1px solid #ddd;\*/

 }

 dt-byline .affiliation {

 display: block;

 }

 dt-byline .author:last-child {

 margin-right: 0;

 }

 dt-byline .name {

 display: block;

 }

 dt-byline .date {

 border-left: 1px solid rgba(0, 0, 0, 0.1);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .year, dt-byline .month {

 display: block;

 }

 dt-byline .citation {

 border-left: 1px solid rgba(0, 0, 0, 0.15);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .citation div {

 display: block;

 }

 }

[Chris Olah](http://colah.github.io/)

[Google Brain](http://g.co/brain)

[Shan Carter](http://shancarter.com/)

[Google Brain](http://g.co/brain)

Sept. 8

2016

[Citation:

Olah & Carter, 2016](#citation)

Recurrent neural networks are one of the staples of deep learning,

allowing neural networks to work with sequences of data like text, audio

 and video. They can be used to boil a sequence down into a high-level

understanding, to annotate sequences, and even to generate new sequences

 from scratch!

rnn

x0

y0

x1

y1

x2

y2

x3

y3

One cell...

can be used over...

and over...

and over...

x4

y4

again.

 even work with these. Such models have been found to be very powerful,

achieving remarkable results in many tasks including translation, voice

recognition, and image captioning. As a result, recurrent neural

networks have become very widespread in the last few years.

As this has happened, we've seen a growing number of attempts to

augment RNNs with new properties. Four directions stand out as

particularly exciting:

 #previews a {

 text-decoration: none;

 overflow: hidden;

 margin-bottom: 12px;

 display: block;

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 margin-bottom: 12px;

 padding-bottom: 12px;

 }

 #previews figcaption {

 margin-left: 100px;

 }

 #previews svg, #previews img {

 width: 80px;

 float: left;

 }

 @media(min-width: 768px) {

 #previews {

 overflow: hidden;

 margin-top: 48px;

 margin-bottom: 48px;

 }

 #previews figcaption {

 margin-left: 0;

 }

 #previews figcaption b {

 display: block;

 }

 #previews figcaption b span {

 display: block;

 }

 #previews a {

 position: relative;

 float: left;

 width: 19%;

 margin-right: 3.6%;

 padding-right: 3.6%;

 border-right: 1px solid rgba(0, 0, 0, 0.05);

 border-bottom: none;

 margin-bottom: 0;

 padding-bottom: 0;

 }

 #previews a:last-child {

 margin-right: 0;

 padding-right: 0;

 border-right: 0;

 }

 #previews svg, #previews img {

 margin-bottom: 18px;

 display: block;

 width: 100%;

 float: none;

 }

 }

 @media(min-width: 1024px) {

 #previews {

 margin-top: 90px;

 margin-bottom: 90px;

 }

 }

Individually, these techniques are all potent extensions of RNNs,

but the really striking thing is that they can be combined, and seem to

just be points in a broader space. Further, they all rely on the same

underlying trick—something called attention—to work.

Our guess is that these "augmented RNNs" will have an important

role to play in extending deep learning's capabilities over the coming

years.

---

Neural Turing Machines

----------------------

Neural Turing Machines [2]

 combine a RNN with an external memory bank. Since vectors are the

natural language of neural networks, the memory is an array of vectors:

rnn

Memory is an array of vectors.

Network A writes and reads from this memory each step.

x0

y0

x1

y1

x2

y2

x3

y3

But how does reading and writing work? The challenge is that we

want to make them differentiable. In particular, we want to make them

differentiable with respect to the location we read from or write to, so

 that we can learn where to read and write. This is tricky because

memory addresses seem to be fundamentally discrete. NTMs take a very

clever solution to this: every step, they read and write everywhere,

just to different extents.

As an example, let's focus on reading. Instead of specifying a

single location, the RNN outputs an "attention distribution" that

describes how we spread out the amount we care about different memory

positions. As such, the result of the read operation is a weighted sum.

rnn

attention

memory

The read result is a weighted sum.

Similarly, we write everywhere at once to different extents. Again,

 an attention distribution describes how much we write at every

location. We do this by having the new value of a position in memory be a

 convex combination of the old memory content and the write value, with

the position between the two decided by the attention weight.

rnn

attention

old memory

new memory

write value

"use strict";

(function() {

 function appendArrowhead(selection) {

 selection.append("marker")

 .attr("id", "arrowhead")

 .attr("viewBox", "0 -5 10 10")

 .attr("refX", 5)

 .attr("refY", 0)

 .attr("markerWidth", 10)

 .attr("markerHeight", 10)

 .attr("orient", "auto")

 .append("path")

 .attr("d", "M0,-5L5,0L0,5")

 .style("fill", "none")

 .style("stroke", "grey");

 }

 var vectorScale = d3.scaleLinear()

 .domain([0, 1])

 .range([-14, 14]);

 var memoryData = [

 {x: 0.9, y: 0.9},

 {x: 0.8, y: 0.8},

 {x: 0.1, y: 0.9},

 {x: 0.4, y: 0.2},

 {x: 0.2, y: 0.2},

 {x: 0.4, y: 0.3},

 {x: 0.1, y: 0.5}

 ];

 var writeData = {x: 0.9, y: 0};

 //

 // Read

 //

 var nodeWidth = 36;

 var readHtml = d3.selectAll("#rnn-read");

 var svg = readHtml.select("svg");

 var result = svg.select("#read-result").style("opacity", 0);

 var resultBBox = result.node().getBBox();

 var memory = svg.selectAll("#memory use");

 var memoryBBox = svg.select("#memory").node().getBBox();

 var attention = svg.selectAll("#attention use");

 var attentionBBox = svg.select("#attention").node().getBBox();

 var path = svg.selectAll("#paths path");

 appendArrowhead(svg);

 var resultArrow = svg.append("line")

 .attr("class", "result-line")

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey");

 memory.data(memoryData)

 .each(function(d, i) {

 this.style.opacity = 0;

 d.top = memoryBBox.y;

 d.left = memoryBBox.x + nodeWidth \* i;

 d.width = nodeWidth;

 });

 var memoryVectors = svg.selectAll(".vector")

 .data(memoryData)

 .enter().append("line")

 .attr("class", "vector")

 .attr("transform", function(d) {

 })

 .attr("x2", function(d) { return vectorScale(d.x); })

 .attr("y2", function(d) { return vectorScale(d.y); })

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey");

 function updateReadAttention(data) {

 attention.data(data)

 .style("opacity", function(d) { return d + 0.05; });

 path.data(data)

 .style("stroke-opacity", function(d) { return d + 0.1; });

 resultArrow

 .attr("x2", function(d) { return vectorScale(resultX); })

 .attr("y2", function(d) { return vectorScale(resultY); });

 }

 var length = memory.size();

 var attentionData = d3.range(length).map(function() { return Math.random(); });

 updateReadAttention(attentionData);

 readHtml.on("mousemove", function() {

 var data = [];

 attention.each(function() {

 var bbox = this.getBoundingClientRect()

 var distance = Math.abs(bbox.left - d3.event.clientX + bbox.width / 2);

 data.push(Math.exp(-distance / 25));

 });

 data = data.map(function(d) { return d / d3.sum(data); });

 updateReadAttention(data);

 });

 updateReadAttention(initReadData);

 //

 // Write

 //

 var writeHtml = d3.selectAll("#rnn-write");

 var writeSvg = writeHtml.select("svg");

 var writeMemory = writeSvg.selectAll("#memory use");

 var writeMemoryBBox = writeSvg.select("#memory").node().getBBox();

 var writeNewMemory = writeSvg.selectAll("#new-memory use");

 var writeNewMemoryBBox = writeSvg.select("#new-memory").node().getBBox();

 var writeAttention = writeSvg.selectAll("#attention use");

 var writeAttentionBBox = writeSvg.select("#attention").node().getBBox();

 var writePath = writeSvg.selectAll("#paths path");

 var writeResult = writeSvg.select("#write-result").style("opacity", 0);

 var writeResultBBox = writeResult.node().getBBox();

 var newMemoryData = memoryData.map(function(){ return {}; });

 appendArrowhead(writeSvg);

 var writeArrow = writeSvg.append("line")

 .attr("class", "write-line")

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey")

 .attr("x2", vectorScale(writeData.x))

 .attr("y2", vectorScale(writeData.y));

 writeMemory.data(memoryData)

 .each(function(d, i) {

 this.style.opacity = 0;

 d.top = writeMemoryBBox.y;

 d.left = writeMemoryBBox.x + i \* nodeWidth;

 d.width = nodeWidth;

 });

 var writeMemoryVectors = writeSvg.selectAll(".vector")

 .data(memoryData)

 .enter().append("line")

 .attr("class", "vector")

 .attr("transform", function(d) {

 })

 .attr("x2", function(d) { return vectorScale(d.x); })

 .attr("y2", function(d) { return vectorScale(d.y); })

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey");

 writeNewMemory.data(newMemoryData)

 .each(function(d, i) {

 this.style.opacity = 0;

 d.top = writeNewMemoryBBox.y;

 d.left = writeNewMemoryBBox.x + i \* nodeWidth;

 d.width = nodeWidth;

 });

 var writeNewMemoryVectors = writeSvg.selectAll(".new-vector")

 .data(newMemoryData)

 .enter().append("line")

 .attr("class", "new-vector")

 .attr("marker-end", "url(#arrowhead)")

 .style("stroke", "grey");

 function updateWriteAttention(data) {

 newMemoryData.forEach(function(d, i) {

 d.x = (memoryData[i].x + writeData.x \* data[i]) / (1 + data[i]);

 d.y = (memoryData[i].y + writeData.y \* data[i]) / (1 + data[i]);

 });

 writeNewMemoryVectors

 .attr("transform", function(d) {

 })

 .attr("x2", function(d) { return vectorScale(d.x); })

 .attr("y2", function(d) { return vectorScale(d.y); });

 writeAttention.data(data)

 .style("opacity", function(d) { return d + 0.05; });

 writePath.data(data)

 .style("stroke-opacity", function(d) { return d + 0.1; });

 }

 writeHtml.on("mousemove", function() {

 var data = [];

 writeAttention.each(function() {

 var bbox = this.getBoundingClientRect()

 var distance = Math.abs(bbox.left - d3.event.clientX + bbox.width / 2);

 data.push(Math.exp(-distance / 25));

 });

 data = data.map(function(d) { return d / d3.sum(data); });

 updateWriteAttention(data);

 });

 updateWriteAttention(initWriteData);

})();

But how do NTMs decide which positions in memory to focus their

attention on? They actually use a combination of two different methods:

content-based attention and location-based attention. Content-based

attention allows NTMs to search through their memory and focus on places

 that match what they're looking for, while location-based attention

allows relative movement in memory, enabling the NTM to loop.

rnn

The scores are then converted into a distribution using softmax.

Next, we interpolate the attention from the previous time step.

memory

Blue shows high similarity, pink high dissimilarity.

interpolation amount

shift filter

RNN controller

attention mechanism

query vector

attention from previous step

new attention distribution

"use strict";

//

// Data

//

var vectorScale = d3.scaleLinear().domain([0, 1]).range([-14, 14]);

var interpolationData = [0.3];

var shiftData = [0, 0.2, 0.8];

var previousAttentionData = [0, 0.2, 0, 0.2, 0.6];

//

// DOM

//

var nodeWidth = 36;

var html = d3.selectAll("#rnn-write-detail");

var svg = html.select("svg");

var query = svg.selectAll("#query").style("opacity", 0);

var memory = svg.selectAll("#memory use").style("opacity", 0);

var memoryBBox = svg.select("#memory").node().getBBox();

var shift = svg.selectAll("#shift use");

//

// Render

//

// Store positions of vectors

function cachePositions(selection) {

 selection.each(function (d) {

 var bBox = this.getBBox();

 d.top = bBox.y;

 d.left = bBox.x;

 d.width = bBox.width;

 });

}

// Render new vector lines

function renderVector(selection) {

 selection.attr("transform", function (d) {

 }).attr("x2", function (d) {

 return vectorScale(d.x);

 }).attr("y2", function (d) {

 return vectorScale(d.y);

 }).attr("marker-end", "url(#arrowhead)").style("stroke", "grey");

}

// Memory

memory.data(memoryData);

memory.each(function(d, i) {

 d.top = memoryBBox.y;

 d.left = memoryBBox.x + nodeWidth \* i;

 d.width = nodeWidth;

})

// cachePositions(memory);

renderVector(memoryVector);

var queryData = [{ x: 0.7, y: 0.8 }];

query.data(queryData);

cachePositions(query);

var clamp = d3.scaleLinear().clamp(true);

function redraw() {

 renderVector(queryVector);

 // Dot

 var dotData = memoryData.map(function (v) {

 });

 svg.selectAll("#dot use").data(dotData).attr("xlink:href", function (d) {

 return d < 0 ? "#array-negative-value" : "#array-positive-value";

 }).style("opacity", function (d) {

 return Math.abs(d);

 });

 // Softmax

 var expDotData = dotData.map(Math.exp);

 var softmaxData = expDotData.map(function (x) {

 return x / d3.sum(expDotData);

 });

 svg.selectAll("#softmax use").data(softmaxData).style("opacity", function (d) {

 return d;

 });

 // Interpolate

 return d;

 });

 return d;

 });

 var interpolateData = previousAttentionData.map(function (d, i) {

 return interpolationData \* d + (1 - interpolationData) \* softmaxData[i];

 });

 return d;

 });

 // Convolve

 shift.data(shiftData).style("opacity", function (d) {

 return d;

 });

 var convolveData = dotData.map(function (x) {

 return 0;

 });

 for (var n in shiftData) {

 for (var i in convolveData) {

 if (i - (1 - n) >= 0) {

 convolveData[i - (1 - n)] += shiftData[n] \* interpolateData[i];

 }

 }

 }

 // = dotData;

 return d;

 });

 // Sharpen

 var sharpenData = convolveData.map(function (x) {

 return x \* x;

 });

 sharpenData = sharpenData.map(function (x) {

 return x / d3.sum(sharpenData);

 });

 return d;

 });

}

svg.select("#query-hover").on("mousemove", function () {

 var bbox = svg.select("#query").node().getBoundingClientRect();

 var x = d3.event.clientX - bbox.left - bbox.width / 2;

 var y = d3.event.clientY - bbox.top - bbox.height / 2;

 //TODO: this clamping is awkward

 queryData[0].x = clamp((x + 25) / 50);

 queryData[0].y = clamp((y + 25) / 50);

 redraw();

});

svg.select("#interpolate-hover").on("mousemove", function () {

 var bbox = svg.select("#interpolate-amount").node().getBoundingClientRect();

 redraw();

});

// .on("mouseout", function() {

// interpolationData = [0.3];

// redraw();

// });

svg.select("#shift-hover").on("mousemove", function () {

 // shiftData = [0, 0.2, 0.8];

 shiftData = [];

 shift.each(function () {

 var bbox = this.getBoundingClientRect();

 var distance = Math.abs(bbox.left - d3.event.clientX + bbox.width / 2);

 shiftData.push(Math.exp(-distance / 25));

 });

 shiftData = shiftData.map(function (d) {

 return d / d3.sum(shiftData);

 });

 redraw();

});

redraw();

This capability to read and write allows NTMs to perform many

simple algorithms, previously beyond neural networks. For example, they

can learn to store a long sequence in memory, and then loop over it,

repeating it back repeatedly. As they do this, we can watch where they

read and write, to better understand what they're doing:

See more experiments in [3]. This figure is based on the Repeat Copy experiment.

They can also learn to mimic a lookup table, or even learn to sort

numbers (although they kind of cheat)! On the other hand, they still

can't do many basic things, like add or multiply numbers.

 train NTMs using reinforcement learning instead of the differentiable

In some objective sense, many of the tasks these models can

perform—such as learning how to add numbers—aren't that objectively

hard. The traditional program synthesis community would eat them for

lunch. But neural networks are capable of many other things, and models

like the Neural Turing Machine seem to have knocked away a very profound

 limit on their abilities.

### Code

There are a number of open source implementations of these models.

---

Attentional Interfaces

----------------------

When I'm translating a sentence, I pay special attention to the

word I'm presently translating. When I'm transcribing an audio

recording, I listen carefully to the segment I'm actively writing down.

And if you ask me to describe the room I'm sitting in, I'll glance

around at the objects I'm describing as I do so.

Neural networks can achieve this same behavior using *attention*,

 focusing on part of a subset of the information they're given. For

example, an RNN can attend over the output of another RNN. At every time

 step, it focuses on different positions in the other RNN.

We'd like attention to be differentiable, so that we can learn

where to focus. To do this, we use the same trick Neural Turing Machines

 use: we focus everywhere, just to different extents.

rnn

Network B focuses on different information from network A at every step.

The attention distribution is usually generated with content-based

attention. The attending RNN generates a query describing what it wants

to focus on. Each item is dot-producted with the query to produce a

score, describing how well it matches the query. The scores are fed into

 a softmax to create the attention distribution.

rnn

The attending RNN generates a query describing what it wants to focus on.

One use of attention between RNNs is translation [11].

 A traditional sequence-to-sequence model has to boil the entire input

down into a single vector and then expands it back out. Attention avoids

 this by allowing the RNN processing the input to pass along information

 about each word it sees, and then for the RNN generating the output to

focus on words as they become relevant.

```

```

output text

network B

network A

input text

 #attentional-ex2 svg {

 width: 100%;

 margin-bottom: 24px;

}

#attentional-ex2 .pointer {

 position: absolute;

 width: 26px;

 height: 26px;

 top: 36px;

 left: -48px;

}

#attentional-ex2 figcaption {

 text-align: center;

}

#attentional-ex2 svg .labels {

 font-size: 12px;

 fill-opacity: 0.5;

}

#attentional-ex2 .background {

 fill: rgb(240, 240, 245);

 fill-opacity: 0.8;

}

#attentional-ex2 .cell .node {

 fill: white;

 fill-opacity: 0.9;

 stroke: black;

 stroke-width: 1px;

 stroke-opacity: 0.25;

 /\*transition: stroke-opacity, 0.2s, linear;\*/

}

#attentional-ex2 .cell:hover .node {

 stroke-opacity: 1;

}

#attentional-ex2 .cell .shadow {

 fill: black;

 fill-opacity: 0.07;

}

#attentional-ex2 .cell text {

 font-size: 13px;

 fill-opacity: 0.6;

 text-anchor: middle;

 alignment-baseline: middle;

}

#attentional-ex2 .cell text.label {

 font-size: 13px;

 /\*fill-opacity: 0.9;\*/

 text-anchor: middle;

 alignment-baseline: middle;

}

#attentional-ex2 .link-group {

 transition: opacity, 0.3s, linear;

}

#attentional-ex2 .link-group .node-highlight {

 transition: stroke-opacity, 0.3s, linear;

 stroke: black;

 stroke-opacity: 0;

 fill: none;

}

#attentional-ex2 .link-group:hover .node-highlight {

 stroke-opacity: 1;

}

#attentional-ex2 .link {

 fill: none;

 stroke: rgb(131, 89, 178);

 stroke-width: 2px;

 pointer-events: none;

}

#attentional-ex2 .arrow {

 stroke: black;

 stroke-opacity: 0.6;

}

#attentional-ex2 .cell:last-child .arrow:first-of-type {

 display: none;

}

(function() {

var labelsA = [

 "the",

 "agreement",

 "on",

 "the",

 "European",

 "Economic",

 "Area",

 "was",

 "signed",

 "in",

 "August",

 "1992",

 ".",

 "<end>"

];

var labelsB = [

 "l'",

 "accord",

 "sur",

 "la",

 "zone",

 "économique",

 "européenne",

 "a",

 "été",

 "signé",

 "en",

 "août",

 "1992",

 ".",

 "<end>"

];

var data = d3.csvParseRows(d3.select("#attentional-ex2-data").text());

// console.log(data)

var linkData = [];

data.forEach(function(a, ai) {

 a.forEach(function(b, bi) {

 linkData.push({

 ai: +ai,

 bi: +bi,

 v: +data[ai][bi]

 });

 });

});

var nodeMargin = 10;

var networkMargin = 120;

var html = d3.select("#attentional-ex2");

var outerWidth = 984;//html.node().getBoundingClientRect().width;

var outerHeight = 240;//html.node().getBoundingClientRect().height;

var margin = {top: 36, right: nodeMargin, bottom: 24, left: nodeMargin};

var width = outerWidth - margin.left - margin.right;

var height = outerHeight - margin.top - margin.bottom;

var xa = d3.scaleBand()

 .domain(d3.range(labelsA.length))

 .paddingInner(0.5)

 .range([nodeMargin, width - nodeMargin]);

var xb = d3.scaleBand()

 .domain(d3.range(labelsB.length))

 .paddingInner(0.5)

 .range([nodeMargin, width - nodeMargin]);

var nodeSizeA = xa.bandwidth();

var nodeSizeB = xb.bandwidth();

var networkHeightA = nodeSizeA + nodeMargin \* 2;

var networkHeightB = nodeSizeB + nodeMargin \* 2;

var link = function(d) {

 // console.log(nodeSizeB, nodeSizeA, nodeMargin, d.bi, xb(d.bi))

 var p = "M" + (xb(d.bi) + nodeSizeB / 2) + "," + (nodeSizeB + nodeMargin)

 + " " + (xa(d.ai) + nodeSizeA / 2) + "," + (networkMargin + nodeMargin);

 // console.log(p);

 return p;

};

html.select("svg .labels")

 .attr("transform", "translate(" + (width + nodeMargin \* 3) + ", 0)");

var svg = html.select("svg")

 .attr("viewBox", "0 0 " + outerWidth + " " + outerHeight)

 .append("g")

 .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

var networkA = svg.append("g")

 .attr("transform", "translate(0, " + networkMargin + ")");

networkA.append("rect")

 .attr("class", "background")

 .attr("width", width)

 .attr("height", nodeSizeA + 2 \* nodeMargin)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

var cellA = networkA.selectAll(".cell")

 .data(labelsA)

 .enter().append("g")

 .attr("class", "cell")

 .on("mouseover", function(d, i) {

 link.style("display", function(ld) {

 return ld.ai === i ? "" : "none";

 });

 })

 .on("mouseout", function() { link.style("display", ""); });

cellA.append("rect")

 .attr("class", "shadow")

 .attr("transform", "translate(1, 3)")

 .attr("width", nodeSizeA)

 .attr("height", nodeSizeA)

 .attr("rx", 4)

 .attr("ry", 4);

cellA.append("rect")

 .attr("class", "node")

 .attr("width", nodeSizeA)

 .attr("height", nodeSizeA)

 .attr("rx", 4)

 .attr("ry", 4);

cellA.append("line")

 .attr("class", "arrow")

 .attr("marker-start", "url(#edgeArrowheadLeft)")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("x1", nodeSizeA + 8)

 .attr("x2", nodeSizeA + nodeMargin + 20)

 .attr("y1", nodeSizeA / 2)

 .attr("y2", nodeSizeA / 2);

cellA.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("transform", "translate(" + nodeSizeA / 2 + ", 0)")

 .attr("y1", nodeSizeA + nodeMargin \* 2 - 2)

 .attr("y2", nodeSizeA + 4);

cellA.append("text")

 .style("opacity", 0.8)

 .text("A")

 .attr("dx", nodeSizeA / 2)

 .attr("dy", nodeSizeA / 2 + 1);

cellA.append("text")

 .attr("class", "label")

 .attr("dx", nodeSizeA / 2)

 .attr("dy", nodeSizeA + 24 + 8)

 .text(function(d) { return d; });

cellA.append("rect")

 .style("opacity", 0)

 .attr("width", xa.step())

 .attr("height", networkHeightA + (networkMargin - networkHeightA) / 2);

//

// Network B

//

var networkB = svg.append("g");

networkB.append("rect")

 .attr("class", "background")

 .attr("width", width)

 .attr("height", nodeSizeB + 2 \* nodeMargin)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

var cellB = networkB.selectAll(".cell")

 .data(labelsB)

 .enter().append("g")

 .attr("class", "cell")

 .on("mouseover", function(d, i) {

 link.style("display", function(ld) {

 return ld.bi === i ? "" : "none";

 });

 })

 .on("mouseout", function() { link.style("display", ""); });

cellB.append("rect")

 .attr("class", "shadow")

 .attr("transform", "translate(1, 3)")

 .attr("width", nodeSizeB)

 .attr("height", nodeSizeB)

 .attr("rx", 4)

 .attr("ry", 4);

cellB.append("rect")

 .attr("class", "node")

 .attr("width", nodeSizeB)

 .attr("height", nodeSizeB)

 .attr("rx", 4)

 .attr("ry", 4);

cellB.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("x1", nodeSizeB)

 .attr("x2", nodeSizeB + nodeMargin + 14)

 .attr("y1", nodeSizeB / 2)

 .attr("y2", nodeSizeB / 2);

cellB.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("transform", "translate(" + nodeSizeB / 2 + ", 0)")

 .attr("y1", 0)

 .attr("y2", -nodeMargin \* 2 + 5);

cellB.append("text")

 .style("opacity", 0.8)

 .text("B")

 .attr("dx", nodeSizeB / 2)

 .attr("dy", nodeSizeB / 2 + 1);

cellB.append("text")

 .attr("class", "label")

 .attr("dy", - 24 - 5)

 .text(function(d) { return d; });

cellB.append("rect")

 .style("opacity", 0)

 .attr("width", xb.step())

 .attr("height", networkHeightB + (networkMargin - networkHeightB) / 2);

//

// Links

//

var linkGroup = svg.append("g").attr("class", ".link-group")

var link = linkGroup.selectAll(".link")

 .data(linkData)

 .enter().append("path")

 .attr("class", "link")

 .attr("d", link)

 .style("opacity", function(d) { return d.v; });

//

//

//

// var hoverB = svg.selectAll(".hover-b")

// .data(labelsB)

// .enter().append("rect")

// .attr("class", "hover-b")

// .style("opacity", 0)

// .attr("width", xb.step())

// .attr("height", networkMargin / 2 + nodeSizeB)

})();

 allowing one RNN to process the audio and then have another RNN skim

over it, focusing on relevant parts as it generates a transcript.

output text

network B

network A

input audio

how\_much\_would\_a\_woodchuck\_chuck<end>

```

```

Figure derived from [Chan, *et al.* 2015](https://arxiv.org/pdf/1508.01211.pdf)

#attentional-ex3 .stage {

 position: relative;

 margin-bottom: 24px;

}

#attentional-ex3 .waveform {

 position: absolute;

 opacity: 0.7;

}

#attentional-ex3 .pointer {

 position: absolute;

 width: 26px;

 height: 26px;

 top: 36px;

 left: -48px;

}

#attentional-ex3 svg {

 z-index: 1;

 position: relative;

 width: 100%;

}

#attentional-ex3 svg .labels {

 font-size: 12px;

 fill-opacity: 0.5;

}

#attentional-ex3 figcaption {

 text-align: center;

}

#attentional-ex3 .background {

 fill: rgb(240, 240, 245);

 fill-opacity: 0.8;

}

#attentional-ex3 .audio-background {

 fill: none;

 stroke: black;

 stroke-opacity: 0.3;

}

#attentional-ex3 .cell .node {

 fill: white;

 fill-opacity: 0.9;

 stroke: black;

 stroke-width: 1px;

 stroke-opacity: 0.15;

}

#attentional-ex3 .cell:hover .node {

 stroke-opacity: 1;

}

#attentional-ex3 .cell .shadow {

 fill: black;

 fill-opacity: 0.07;

}

#attentional-ex3 .cell text {

 font-size: 13px;

 fill-opacity: 0.5;

 text-anchor: middle;

 alignment-baseline: middle;

}

#attentional-ex3 .cell text.label {

 font-size: 15px;

 /\*fill-opacity: 0.8;\*/

 text-anchor: middle;

 alignment-baseline: middle;

}

#attentional-ex3 .cell:last-child text.label {

 font-size: 10px;

}

#attentional-ex3 .link-group {

 transition: opacity, 0.3s, linear;

}

#attentional-ex3 .link-group .node-highlight {

 transition: stroke-opacity, 0.3s, linear;

 stroke: black;

 stroke-opacity: 0;

 fill: none;

}

#attentional-ex3 .link-group:hover .node-highlight {

 stroke-opacity: 1;

}

#attentional-ex3 .link {

 fill: none;

 stroke: rgb(131, 89, 178);

 stroke-width: 2px;

 pointer-events: none;

}

#attentional-ex3 .arrow {

 stroke: black;

 stroke-opacity: 0.3;

}

#attentional-ex3 .cell:last-child .arrow:first-of-type {

 display: none;

}

#attentional-ex3 .tick {

 stroke: black;

 stroke-opacity: 0.2;

}

(function() {

var data = d3.csvParseRows(d3.select("#attentional-ex3-data").text());

var labelsA = d3.range(data.length);

var linkData = [];

data.forEach(function(a, ai) {

 a.forEach(function(b, bi) {

 linkData.push({

 ai: +ai,

 bi: +bi,

 v: +data[ai][bi]

 });

 });

});

var nodeMargin = 8;

var networkMargin = 140;

var audioHeight = 60;

var html = d3.select("#attentional-ex3");

var outerHeight = 280;

var margin = {top: 36, right: nodeMargin + 80, bottom: 24, left: nodeMargin};

var width = outerWidth - margin.left - margin.right;

var height = outerHeight - margin.top - margin.bottom;

var xa = d3.scaleBand()

 .domain(d3.range(labelsA.length))

 .paddingInner(0.5)

 .range([nodeMargin, width - nodeMargin]);

var xb = d3.scaleBand()

 .domain(d3.range(labelsB.length))

 .paddingInner(0.5)

 .range([nodeMargin, width - nodeMargin]);

var nodeSizeA = xa.bandwidth();

var nodeSizeB = xb.bandwidth();

var networkHeightA = nodeSizeA + nodeMargin \* 2;

var networkHeightB = nodeSizeB + nodeMargin \* 2;

var link = function(d) {

 return "M" + (xb(d.bi) + nodeSizeB / 2) + "," + (nodeSizeB + nodeMargin)

 + " " + (xa(d.ai) + nodeSizeA / 2) + "," + (networkMargin + nodeMargin);

};

var svg = html.select("svg")

 .attr("viewBox", "0 0 " + outerWidth + " " + outerHeight)

 .append("g")

 .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

html.select("svg .labels")

 .attr("transform", "translate(" + (width + nodeMargin \* 3) + ", 0)");

var audio = svg.append("g")

 .attr("class", "audio")

//

// Network A

//

var networkA = svg.append("g")

 .attr("transform", "translate(0, " + networkMargin + ")");

networkA.append("rect")

 .attr("class", "background")

 .attr("width", width)

 .attr("height", nodeSizeA + 2 \* nodeMargin)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

var cellA = networkA.selectAll(".cell")

 .data(labelsA)

 .enter().append("g")

 .attr("class", "cell")

 .on("mouseover", function(d, i) {

 link.style("display", function(ld) {

 return ld.ai === i ? "" : "none";

 });

 })

 .on("mouseout", function() { link.style("display", ""); });

cellA.append("rect")

 .attr("class", "shadow")

 .attr("transform", "translate(1, 3)")

 .attr("width", nodeSizeA)

 .attr("height", nodeSizeA)

 .attr("rx", 4)

 .attr("ry", 4);

cellA.append("rect")

 .attr("class", "node")

 .attr("width", nodeSizeA)

 .attr("height", nodeSizeA)

 .attr("rx", 2)

 .attr("ry", 2);

cellA.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("x1", nodeSizeA)

 .attr("x2", nodeSizeA + nodeMargin)

 .attr("y1", nodeSizeA / 2)

 .attr("y2", nodeSizeA / 2);

cellA.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("transform", "translate(" + nodeSizeA / 2 + ", " + nodeSizeA + ")")

 .attr("y1", nodeMargin \* 3)

 .attr("y2", 4);

cellA.append("rect")

 .style("opacity", 0)

 .attr("width", xa.step())

//

// Network B

//

var networkB = svg.append("g");

networkB.append("rect")

 .attr("class", "background")

 .attr("width", width)

 .attr("height", nodeSizeB + 2 \* nodeMargin)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

var cellB = networkB.selectAll(".cell")

 .data(labelsB)

 .enter().append("g")

 .attr("class", "cell")

 .on("mouseover", function(d, i) {

 link.style("display", function(ld) {

 return ld.bi === i ? "" : "none";

 });

 })

 .on("mouseout", function() { link.style("display", ""); });

cellB.append("rect")

 .attr("class", "shadow")

 .attr("transform", "translate(1, 3)")

 .attr("width", nodeSizeB)

 .attr("height", nodeSizeB)

 .attr("rx", 4)

 .attr("ry", 4);

cellB.append("rect")

 .attr("class", "node")

 .attr("width", nodeSizeB)

 .attr("height", nodeSizeB)

 .attr("rx", 2)

 .attr("ry", 2);

cellB.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("x1", nodeSizeB)

 .attr("x2", nodeSizeB + nodeMargin)

 .attr("y1", nodeSizeB / 2)

 .attr("y2", nodeSizeB / 2);

cellB.append("line")

 .attr("class", "arrow")

 .attr("marker-end", "url(#edgeArrowhead)")

 .attr("transform", "translate(" + nodeSizeB / 2 + ", 0)")

 .attr("y1", 0)

 .attr("y2", -nodeMargin \* 2);

cellB.append("text")

 .attr("class", "label")

 .attr("dx", nodeSizeB / 2)

 .attr("dy", - nodeMargin \* 2 - 15)

 .text(function(d) { return d; });

cellB.append("rect")

 .style("opacity", 0)

 .attr("width", xb.step())

 .attr("height", networkHeightB + (networkMargin - networkHeightB) / 2);

//

// Links

//

var linkGroup = svg.append("g").attr("class", ".link-group")

var link = linkGroup.selectAll(".link")

 .data(linkData)

 .enter().append("path")

 .attr("class", "link")

 .attr("d", link)

 .style("opacity", function(d) { return d.v; });

//

// Audio

//

html.select(".waveform")

 .style("left", "1%")

 .style("top", "76%")

 .style("width", "90%")

 .style("height", "20%")

audio.append("rect")

 .attr("class", "audio-background")

 .attr("width", width)

 .attr("height", audioHeight)

 .attr("rx", nodeMargin \* 0.7)

 .attr("ry", nodeMargin \* 0.7);

audio.selectAll(".tick")

 .data(labelsA)

 .enter().append("line")

 .style("display", function(d) { return d == 0 ? "none" : ""; })

 .attr("class", "tick")

 .attr("x1", function(d) { return xa(d); })

 .attr("x2", function(d) { return xa(d); })

 .attr("y2", audioHeight)

audio.selectAll(".segment")

 .data(labelsA)

 .enter().append("rect")

 .style("opacity", 0)

 .attr("class", "segment")

 .attr("transform", function(d) {

 })

 .attr("width", xa.step())

 .attr("height", audioHeight)

})();

Attention can also be used on the interface between a convolutional

 neural network and an RNN. This allows the RNN to look at different

position of an image every step. One popular use of this kind of

attention is for image captioning. First, a conv net processes the

image, extracting high-level features. Then an RNN runs, generating a

description of the image. As it generates each word in the description,

the RNN focuses on the conv net's interpretation of the relevant parts

of the image. We can explicitly visualize this:

Figure from [3]

More broadly, attentional interfaces can be used whenever one wants

 to interface with a neural network that has a repeating structure in

its output.

Attentional interfaces have been found to be an extremely general

and powerful technique, and are becoming increasingly widespread.

---

Adaptive Computation Time

-------------------------

Standard RNNs do the same amount of

computation for each time step. This seems unintuitive. Surely, one

Adaptive Computation Time [15]

 is a way for RNNs to do different amounts of computation each step. The

 big picture idea is simple: allow the RNN to do multiple steps of

computation for each time step.

In order for the network to learn how many steps to do, we want the

 number of steps to be differentiable. We achieve this with the same

trick we used before: instead of deciding to run for a discrete number

of steps, we have an attention distribution over the number of steps to

run. The output is a weighted combination of the outputs of each step.

rnn

For every time step the RNN can do multiple computation steps.

The output is a weighted combination of the computation step outputs.

The process is repeated for each time step.

A special bit is set to denote the first computation step.

There are a few more details, which were left out in the previous

diagram. Here's a complete diagram of a time step with three computation

 steps.

rnn

That's a bit complicated, so let's work through it step by step. At

 a high-level, we're still running the RNN and outputting a weighted

combination of the states:

rnn

The weight for each step is determined by a "halting neuron." It's a

 sigmoid neuron that looks at the RNN state and gives a halting weight,

which we can think of as the probability that we should stop at that

step.

rnn

halting neuron

We have a total budget for the halting weights of 1, so we track

that budget along the top. When it gets to less than epsilon, we stop.

rnn

When we stop, might have some left over halting budget because we

stop when it gets to less than epsilon. What should we do with it?

Technically, it's being given to future steps but we don't want to

compute those, so we attribute it to the last step.

rnn

When training Adaptive Computation Time models, one adds a "ponder

cost" term to the cost function. This penalizes the model for the amount

 of computation it uses. The bigger you make this term, the more it will

 trade-off performance for lowering compute time.

### Code

---

Neural Programmer

-----------------

Neural nets are excellent at many tasks, but they also struggle to

do some basic things like arithmetic, which are trivial in normal

approaches to computing. It would be really nice to have a way to fuse

neural nets with normal programming, and get the best of both worlds.

The actual model in the paper answers questions about tables by

generating SQL-like programs to query the table. However, there are a

number of details here that make it a bit complicated, so let's start by

 imagining a slightly simpler model, which is given an arithmetic

expression and generates a program to evaluate it.

The generated program is a sequence of operations. Each operation

is defined to operate on the output of past operations. So an operation

might be something like "add the output of the operation 2 steps ago and

 the output of the operation 1 step ago." It's more like a Unix pipe

than a program with variables being assigned to and read from.

rnn

The program is generated one operation at a time by a controller

RNN. At each step, the controller RNN outputs a probability distribution

 for what the next operation should be. For example, we might be pretty

sure we want to perform addition at the first time step, then have a

hard time deciding whether we should multiply or divide at the second

step, and so on...

rnn

At each step the controller RNN outputs a probability distribution.

The resulting distribution over operations can now be evaluated.

Instead of running a single operation at each step, we do the usual

attention trick of running all of them and then average the outputs

together, weighted by the probability we ran that operation.

rnn

At each step the controller RNN outputs a probability distribution.

We run all of the operations and average the outputs together.

As long as we can define derivatives through the operations, the

program's output is differentiable with respect to the probabilities. We

 can then define a loss, and train the neural net to produce programs

that give the correct answer. In this way, the Neural Programmer learns

to produce programs without examples of good programs. The only

supervision is the answer the program should produce.

That's the core idea of Neural Programmer, but the version in the

paper answers questions about tables, rather than arithmetic

expressions. There are a few additional neat tricks:

* **Multiple Types:** Many of the operations in the

Neural Programmer deal with types other than scalar numbers. Some

* **Referencing Inputs:** The neural programmer

needs to answer questions like "How many cities have a population

greater than 1,000,000?" given a table of cities with a population

column. To facilitate this, some operations allow the network to

reference constants in the question they're answering, or the names of

columns. This referencing happens by attention, in the style of pointer

networks [17].

The Neural Programmer isn't the only approach to having neural

networks generate programs. Another lovely approach is the Neural

We think that this general space, of bridging the gap between more

traditional programming and neural networks is extremely important.

While the Neural Programmer is clearly not the final solution, we think

there are a lot of important lessons to be learned from it.

### Code

---

The Big Picture

---------------

A human with a piece of paper is, in some sense, much smarter than a

 human without. A human with mathematical notation can solve problems

they otherwise couldn't. Access to computers makes us capable of

incredible feats that would otherwise be far beyond us.

In general, it seems like a lot of interesting forms of

intelligence are an interaction between the creative heuristic intuition

 of humans and some more crisp and careful media, like language or

equations. Sometimes, the medium is something that physically exists,

and stores information for us, prevents us from making mistakes, or does

 computational heavy lifting. In other cases, the medium is a model in

our head that we manipulate. Either way, it seems deeply fundamental to

intelligence.

Recent results in machine learning have started to have this

flavor, combining the intuition of neural networks with something else.

One approach is what one might call "heuristic search." For example,

 uses neural networks as intuition for manipulating mathematical

expressions. The "augmented RNNs" we've talked about in this article are

 another approach, where we connect RNNs to engineered media, in order

to extend their general capabilities.

Interacting with media naturally involves making a sequence of

taking an action, observing, and taking more actions. This creates a

major challenge: how do we learn which actions to take? That sounds like

 a reinforcement learning problem and we could certainly take that

approach. But the reinforcement learning literature is really attacking

the hardest version of this problem, and its solutions are hard to use.

The wonderful thing about attention is that it gives us an easier way

out of this problem by partially taking all actions to varying extents.

This works because we can design media—like the NTM memory—to allow

fractional actions and to be differentiable. Reinforcement learning has

us take a single path, and try to learn from that. Attention takes every

 direction at a fork, and then merges the paths back together.

A major weaknesses of attention is that we have to take every

"action" every step. This causes the computational cost to grow linearly

 as you do things like increase the amount of memory in a Neural Turing

Machine. One thing you could imagine doing is having your attention be

sparse, so that you only have to touch some memories. However, it's

still challenging because you may want to do things like have your

attention depend on the content of the memory, and doing that naively

forces you to look at each memory. We've seen some initial attempts to

attack this problem, such as [21],

 but it seems like there's a lot more to be done. If we could really

make such sub-linear time attention work, that would be very powerful!

Augmented recurrent neural networks, and the underlying technique

of attention, are incredibly exciting. We look forward to seeing what

happens next!


Thread: Circuits

Nick Cammarata
Shan Carter
Gabriel Goh
Chris Olah
Michael Petrov
Ludwig Schubert
Chelsea Voss
Ben Egan
Swee Kiat Lim

Abstract: In the original narrative of deep learning, each neuron builds       progressively more abstract, meaningful features by composing features in       the preceding layer. In recent years, there’s been some skepticism of this       view, but what happens if you take it really seriously?

 In the original narrative of deep learning, each neuron builds

 progressively more abstract, meaningful features by composing features in

 the preceding layer. In recent years, there's been some skepticism of this

 view, but what happens if you take it really seriously?

 InceptionV1 is a classic vision model with around 10,000 unique

neurons — a large number, but still on a scale that a group effort could

 attack.

 What if you simply go through the model, neuron by neuron, trying

to

 understand each one and the connections between them? The circuits

 collaboration aims to find out.

Articles & Comments

-------------------

 The natural unit of publication for investigating circuits seems to be

 short papers on individual circuits or small families of features.

 Compared to normal machine learning papers, this is a small and unusual

 topic for a paper.

 To facilitate exploration of this direction, Distill is inviting a

 "thread" of short articles on circuits, interspersed with critical

 commentary by experts in adjacent fields. The thread will be a living

 document, with new articles added over time, organized through an open

 slack channel (#circuits in the

 [Distill slack](http://slack.distill.pub/)). Content in this

 thread should be seen as early stage exploratory research.

Articles and comments are presented below in chronological order:

###

### Authors

### Affiliations

[Chris Olah](https://colah.github.io/),

 [Nick Cammarata](http://nickcammarata.com/),

 [Ludwig Schubert](https://schubert.io/),

 [Gabriel Goh](http://gabgoh.github.io/),

 [Michael Petrov](https://twitter.com/mpetrov),

 [Shan Carter](http://shancarter.com/)

[OpenAI](https://openai.com/)

 Does it make sense to treat individual neurons and the connections

 between them as a serious object of study? This essay proposes three

 claims which, if true, might justify serious inquiry into them: the

 existence of meaningful features, the existence of meaningful circuits

 between features, and the universality of those features and circuits.

 It also discuses historical successes of science "zooming in," whether

 we should be concerned about this research being qualitative, and

 approaches to rigorous investigation.

[Read Full Article](https://distill.pub/2020/circuits/zoom-in/)

###

### Authors

### Affiliations

[Chris Olah](https://colah.github.io/),

 [Nick Cammarata](http://nickcammarata.com/),

 [Ludwig Schubert](https://schubert.io/),

 [Gabriel Goh](http://gabgoh.github.io/),

 [Michael Petrov](https://twitter.com/mpetrov),

 [Shan Carter](http://shancarter.com/)

[OpenAI](https://openai.com/)

 An overview of all the neurons in the first five layers of

 InceptionV1, organized into a taxonomy of "neuron groups." This

 article sets the stage for future deep dives into particular aspects

 of early vision.

 [Read Full Article](https://distill.pub/2020/circuits/early-vision/)

###

[Curve Detectors](https://distill.pub/2020/circuits/curve-detectors/)

### Authors

### Affiliations

[Nick Cammarata](http://nickcammarata.com/),

 [Gabriel Goh](http://gabgoh.github.io/),

 [Shan Carter](http://shancarter.com/),

 [Ludwig Schubert](https://schubert.io/),

 [Michael Petrov](https://twitter.com/mpetrov),

 [Chris Olah](https://colah.github.io/)

[OpenAI](https://openai.com/)

 Every vision model we've explored in detail contains neurons which

 detect curves. Curve detectors is the first in a series of three

 articles exploring this neuron family in detail.

[Read Full Article](https://distill.pub/2020/circuits/curve-detectors/)

###

### Authors

### Affiliations

[Chris Olah](https://colah.github.io/),

 [Ludwig Schubert](https://schubert.io/),

 [Gabriel Goh](http://gabgoh.github.io/)

[OpenAI](https://openai.com/)

 Neural networks naturally learn many transformed copies of the same

 feature, connected by symmetric weights.

[Read Full Article](https://distill.pub/2020/circuits/equivariance/)

###

### Authors

### Affiliations

[Ludwig Schubert](https://schubert.io/),

 [Chelsea Voss](https://csvoss.com/),

 [Nick Cammarata](http://nickcammarata.com/),

 [Gabriel Goh](http://gabgoh.github.io/),

 [Chris Olah](https://colah.github.io/)

[OpenAI](https://openai.com/)

 A family of early-vision neurons reacting to directional transitions

 from high to low spatial frequency.

[Read Full Article](https://distill.pub/2020/circuits/frequency-edges/)

###

[Curve Circuits](https://distill.pub/2020/circuits/curve-circuits/)

### Authors

### Affiliations

[Nick Cammarata](http://nickcammarata.com/),

 [Gabriel Goh](http://gabgoh.github.io/),

 [Shan Carter](http://shancarter.com/),

 [Chelsea Voss](https://csvoss.com/),

 [Ludwig Schubert](https://schubert.io/),

 [Chris Olah](https://colah.github.io/)

[OpenAI](https://openai.com/)

 We reverse engineer a non-trivial learned algorithm from the weights

 of a neural network and use its core ideas to craft an artificial

 artificial neural network from scratch that reimplements it.

[Read Full Article](https://distill.pub/2020/circuits/curve-circuits/)

###

[Visualizing Weights](https://distill.pub/2020/circuits/visualizing-weights/)

### Authors

### Affiliations

[Chelsea Voss](https://csvoss.com/),

 [Nick Cammarata](http://nickcammarata.com/),

 [Gabriel Goh](https://gabgoh.github.io/),

 [Michael Petrov](https://twitter.com/mpetrov),

 [Ludwig Schubert](https://schubert.io/),

 Ben Egan,

 [Swee Kiat Lim](https://greentfrapp.github.io/),

 [Chris Olah](https://colah.github.io/)

[OpenAI](https://openai.com/),

 [Mount Royal University](https://mtroyal.ca/),

 [Stanford University](https://stanford.edu/)

 We present techniques for visualizing, contextualizing, and

 understanding neural network weights.

[Read Full Article](https://distill.pub/2020/circuits/visualizing-weights/)

###

### Authors

### Affiliations

[Chelsea Voss](https://csvoss.com/),

 [Gabriel Goh](https://gabgoh.github.io/),

 [Nick Cammarata](http://nickcammarata.com/),

 [Michael Petrov](https://twitter.com/mpetrov),

 [Ludwig Schubert](https://schubert.io/),

 [Chris Olah](https://colah.github.io/)

[OpenAI](https://openai.com/)

 When a neural network layer is divided into multiple branches, neurons

 self-organize into coherent groupings.

[Read Full Article](https://distill.pub/2020/circuits/branch-specialization/)

###

[Weight Banding](https://distill.pub/2020/circuits/weight-banding/)

### Authors

### Affiliations

[Michael Petrov](https://twitter.com/mpetrov),

 [Chelsea Voss](https://csvoss.com/),

 [Ludwig Schubert](https://schubert.io/),

 [Nick Cammarata](http://nickcammarata.com/),

 [Gabriel Goh](https://gabgoh.github.io/),

 [Chris Olah](https://colah.github.io/)

[OpenAI](https://openai.com/)

[Read Full Article](https://distill.pub/2020/circuits/weight-banding/)

#### This is a living document

 Expect more articles on this topic, along with critical comments from

 experts.

Get Involved

------------

 The Circuits thread is open to articles exploring individual features,

 circuits, and their organization within neural networks. Critical

 commentary and discussion of existing articles is also welcome. The thread

 is organized through the open `#circuits` channel on the

 [Distill slack](http://slack.distill.pub/). Articles can be

 suggested there, and will be included at the discretion of previous

 authors in the thread, or in the case of disagreement by an uninvolved

 editor.

 If you would like get involved but don't know where to start, small

 projects may be available if you ask in the channel.

About the Thread Format

-----------------------

 Part of Distill's mandate is to experiment with new forms of scientific

 publishing. We believe that that reconciling faster and more continuous

 approaches to publication with review and discussion is an important open

 problem in scientific publishing.

 Threads are collections of short articles, experiments, and critical

 commentary around a narrow or unusual research topic, along with a slack

 channel for real time discussion and collaboration. They are intended to

 be earlier stage than a full Distill paper, and allow for more fluid

 publishing, feedback and discussion. We also hope they'll allow for wider

 participation. Think of a cross between a Twitter thread, an academic

 workshop, and a book of collected essays.

 Threads are very much an experiment. We think it's possible they're a

 great format, and also possible they're terrible. We plan to trial two

 such threads and then re-evaluate our thought on the format.


Open Questions about Generative Adversarial Networks

Abstract: By some metrics, research on Generative Adversarial Networks (GANs) has progressed substantially in the past 2 years.     Practical improvements to image synthesis models are being made  almost too quickly to keep up with:

Odena et al., 2016

Miyato et al., 2017

Zhang et al., 2018

Brock et al., 2018

 However, by other metrics, less has happened. For instance, there is

 still widespread disagreement about how GANs should be evaluated.

 Given that current image synthesis benchmarks seem somewhat

saturated, we think now is a good time to reflect on research goals for

this sub-field.

 Lists of open problems have helped other fields with this.

 We also believe that writing this article has clarified our thinking about

 We assume a fair amount of background (or willingness to look things up)

 because we reference too many results to explain all those results in detail.

What are the Trade-Offs Between GANs and other Generative Models?

-----------------------------------------------------------------

 This statement shouldn't be taken too literally.

 that aren't easy to describe as belonging to just one of those clusters.

 I've also left out VAEs entirely;

 they're arguably no longer considered state-of-the-art at any tasks of record.

 .

 Roughly speaking, Flow Models apply a

 stack of invertible transformations to a sample from a prior

 so that exact log-likelihoods of observations can be computed.

 On the other hand, Autoregressive Models factorize the

 distribution over observations into conditional distributions

 at a time.)

 Recent research suggests that these models have different

 performance characteristics and trade-offs.

 to the model families is an interesting open question.

 At first glance, Flow Models seem like they might make GANs unnecessary.

 Flow Models allow for exact log-likelihood computation and exact inference,

 A lot of effort is spent on training GANs,

 so it seems like we should care about whether Flow Models make GANs obsolete

 image-to-image translation.

 .

 The GLOW model is trained to generate 256x256 celebrity faces using

 40 GPUs for 2 weeks and about 200 million parameters.

 In contrast, progressive GANs are trained on a similar face dataset

 to generate images with 16 times fewer pixels.

 This comparison isn't perfect,

 For instance, it's possible that the progressive growing

 technique could be applied to Flow Models as well.

 but it gives you a sense of things.

 Why are the Flow Models less efficient?

 We see two possible reasons:

 you will be penalized infinitely harshly!

 and this penalty is less harsh.

 Section 6.1 of  does some small experiments on expressivity, but at

 present we're not aware of any in-depth analysis of this question.

 It turns out that Autoregressive Models can be expressed as Flow Models

 (because they are both reversible) that are not parallelizable.

 Parallelizable is somewhat imprecise in this context.

 There may be ways around this limitation though.

 Thus, GANs are parallel and efficient but not reversible,

 Flow Models are reversible and parallel but not efficient, and

 Autoregressive models are reversible and efficient, but not parallel.

|  | Parallel | Efficient | Reversible |

| --- | --- | --- | --- |

| GANs | Yes | Yes | No |

| Flow Models | Yes | No | Yes |

| Autoregressive Models | No | Yes | Yes |

 This brings us to our first open problem:

Problem 1

What are the fundamental trade-offs between GANs and other generative models?

 This has been considered for hybrid GAN/Flow Models, but we think that

 this approach is still underexplored.

 to do better than chance if the generator does this.

What Sorts of Distributions Can GANs Model?

-------------------------------------------

 Most GAN research focuses on image synthesis.

 MNIST,

 CIFAR-10,

 STL-10,

 CelebA,

 and Imagenet.

 There is some folklore about which of these datasets is 'easiest' to model.

 'extremely regular'.

 Others have noted that 'a high number of classes is what makes

 ImageNet synthesis difficult for GANs'.

 synthesis model on CelebA generates images that seem

 Imagenet.

 trying to train GANs on ever larger and more complicated datasets.

 In particular, we've mostly studied how GANs perform on the datasets that

 happened to be laying around for object recognition.

 training a generative model, and then say something like 'this dataset will be

 easy for a GAN to model, but not a VAE'.

 There has been some progress on this topic,

 but we feel that more can be done.

 We can now state the problem:

Problem 2

 We might ask the following related questions as well:

 What do we mean by 'model the distribution'? Are we satisfied with a

 low-support representation, or do we want a true density model?

 Are there distributions that a GAN can never learn to model?

 Are there distributions that are learnable for a GAN in principle,

but are not

 efficiently learnable, for some reasonable model of

resource-consumption?

 Are the answers to these questions actually any different for GANs

than they are for other

 generative models?

 We propose two strategies for answering these questions:

 For example, in  the authors create a dataset of synthetic triangles.

 We feel that this

 angle is under-explored.

 allowing for systematic study.

 modify the assumptions to account

 for different properties of the dataset.

 what happens to them when the data distribution becomes multi-modal.

How Can we Scale GANs Beyond Image Synthesis?

---------------------------------------------

 Aside from applications like image-to-image

 translation

 and domain-adaptation

 most GAN successes have been in image synthesis.

 Attempts to use GANs beyond images have focused on three domains:

* **Text** -

 The discrete nature of text makes it difficult to apply GANs.

 This is because GANs rely on backpropagating a signal from the

discriminator through the generated content into the generator.

 There are two approaches to addressing this difficulty.

 The first is to have the GAN act only on continuous

 representations of the discrete data, as in.

 The second is use an actual discrete model and attempt to train the GAN using

 gradient estimation as in.

 Other, more sophisticated treatments exist,

 with likelihood-based language models.

* **Structured Data** -

 What about other non-euclidean structured data, like graphs?

 The study of this type of data is called geometric deep

 learning.

 GANs have had limited success here, but so have other deep learning techniques,

 so it's hard to tell how much the GAN aspect matters.

 We're aware of one attempt to use GANs in this space,

 which has the generator produce (and the discriminator 'critique') random walks

 that are meant to resemble those sampled from a source graph.

* **Audio** -

 Audio is the domain in which GANs are closest to achieving the success

 they've enjoyed with images.

 The first serious attempt at applying GANs to unsupervised audio synthesis

 is, in which the authors

 More recent work suggests GANs can even

 outperform autoregressive models on some perceptual metrics.

 Despite these attempts, images are clearly the easiest domain for GANs.

 This leads us to the statement of the problem:

Problem 3

How can GANs be made to perform well on non-image data?

Does scaling GANs to other domains require new training techniques,

 or does it simply require better implicit priors for each domain?

 but that it will require better implicit priors.

 in a given domain.

 For structured data or data that is not continuous, we're less sure.

 One approach might be to make both the generator and discriminator

 large-scale computational resources.

 Finally, this problem may just require fundamental research progress.

What can we Say About the Global Convergence of GAN Training?

-------------------------------------------------------------

 the generator and discriminator for opposing objectives.

 Under certain assumptions

 These assumptions are very strict.

 The referenced paper assumes (roughly speaking) that

 the equilibrium we are looking for exists and that

 we are already very close to it.

 ,

 this simultaneous optimization

 is locally asymptotically stable.

 But all neural networks have this problem!

 This brings us to our question:

Problem 4

When can we prove that GANs are globally convergent?

Which neural network convergence results can be applied to GANs?

 There has been nontrivial progress on this question.

 Broadly speaking, there are 3 existing techniques, all of which have generated

 promising results but none of which have been studied to completion:

* **Simplifying Assumptions** -

 For example, the simplified LGQ GAN —

 linear generator, Gaussian data, and quadratic discriminator — can be

shown to be globally convergent, if optimized with a special technique

 and some additional assumptions.

 As another example,  show under different simplifying assumptions that

 It seems promising to gradually relax those assumptions to see what happens.

 For example, we could move away from unimodal distributions.

* **Use Techniques from Normal Neural Networks** -

 to answer questions about convergence of GANs.

 For instance, it's argued in that the non-convexity

 of deep neural networks isn't a problem,

 A fact that practitioners already kind of suspected.

 because low-quality local minima of the loss function

 become exponentially rare as the network gets larger.

 Can this analysis be 'lifted into GAN space'?

 In fact, it seems like a generally useful heuristic to take analyses

 of deep neural networks used as classifiers and see if they apply to

GANs.

* **Game Theory** -

 The final strategy is to model GAN training using notions from game theory.

 but do so using unreasonably large resource constraints.

How Should we Evaluate GANs and When Should we Use Them?

--------------------------------------------------------

 Suggestions include:

* **Inception Score and FID** -

 Both these scores

 use a pre-trained image classifier and both have

 known issues .

 A common criticism is that these scores measure

 'sample quality' and don't really capture 'sample diversity'.

* **MS-SSIM** -

  propose using MS-SSIM to

* **AIS** -

  propose putting a Gaussian observation model on the outputs

 of a GAN and using annealed importance sampling to estimate

 the log likelihood under this model, but  show that

 .

* **Geometry Score** -

  suggest computing geometric properties of the generated data manifold

 and comparing those properties to the real data.

* **Precision and Recall** -

  attempt to measure both the 'precision' and 'recall' of GANs.

* **Skill Rating** -

  have shown that trained GAN discriminators can contain useful information

 with which evaluation can be performed.

 Those are just a small fraction of the proposed GAN evaluation schemes.

 *when to use GANs*.

 Thus, we have bundled those two questions into one:

Problem 5

When should we use GANs instead of other generative models?

How should we evaluate performance in those contexts?

 What should we use GANs for?

 If you want an actual density model, GANs probably aren't the best choice.

 , which means there may be substantial parts of the test

 set to which a GAN (implicitly) assigns zero likelihood.

 Rather than worrying too much about this,

 Though trying to fix this issue is a valid research agenda as well.

 GANs are likely to be well-suited to tasks with a perceptual flavor.

 all fall under this umbrella.

 How should we evaluate GANs on these perceptual tasks?

 Ideally, we would just use a human judge, but this is expensive.

 This is called a classifier two-sample test (C2STs)

 .

 (e.g., ) this will dominate the evaluation.

 One approach might be to make a critic that is blind to the dominant defect.

 creating an ordered list of the most important defects and

 critics that ignore them.

 more and more of the higher variance components.

 Finally, we could evaluate on humans despite the expense.

 This would allow us to measure the thing that we actually care about.

 the prediction is uncertain.

How does GAN Training Scale with Batch Size?

--------------------------------------------

 At first glance, it seems like the answer should be yes — after all,

 the discriminator in most GANs is just an image classifier.

 Larger batches can accelerate training if it is bottlenecked on

gradient noise.

 However, GANs have a separate bottleneck that classifiers don't: the

 training procedure can diverge.

 Thus, we can state our problem:

Problem 6

How does GAN training scale with batch size?

How big a role does gradient noise play in GAN training?

Can GAN training be modified so that it scales better with batch size?

 Can alternate training procedures make better use of large batches?

 asynchronous SGD interacts in a special way with GAN training.

What is the Relationship Between GANs and Adversarial Examples?

---------------------------------------------------------------

 It's well known that image classifiers suffer from adversarial examples:

 but are exponentially hard to learn robustly.

 Despite the large bodies of literature on GANs and adversarial examples,

 there doesn't seem to be much work on how they relate.

 Thus, we can ask the question:

Problem 7

How does the adversarial robustness of the discriminator affect GAN training?

 How can we begin to think about this problem?

 Consider a fixed discriminator **D**.

 An adversarial example for **D** would exist if

 there were a generator sample **G(z)** correctly classified as fake and

 a small perturbation **p** such that **G(z) + p** is classified as real.

 a new generator **G'** where **G'(z) = G(z) + p**.

 Is this concern realistic?

  shows that deliberate attacks on generative models can work,

 but we are more worried about something you might call an 'accidental attack'.

 There are reasons to believe that these accidental attacks are less likely.

 First, the generator is only allowed to make one gradient update before

 the discriminator is updated again.

 for every gradient step.

 We think this is a fruitful topic for further exploration.

Conditional Image Synthesis With Auxiliary Classifier GANs

Self-Attention Generative Adversarial Networks

Spectral Normalization for Generative Adversarial Networks

Large Scale GAN Training for High Fidelity Natural Image Synthesis

A style-based generator architecture for generative adversarial networks

Open Questions in Physics

Not especially famous, long-open problems which anyone can understand

Hilbert's Problems

Smale's Problems

Auto-Encoding Variational Bayes

NICE: Non-linear Independent Components Estimation

Density estimation using Real NVP

Glow: Generative Flow with Invertible 1x1 Convolutions

Normalizing Flows Tutorial

Pixel Recurrent Neural Networks

Conditional Image Generation with PixelCNN Decoders

PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

WaveNet: A Generative Model for Raw Audio

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Variational Inference with Normalizing Flows

Parallel WaveNet: Fast High-Fidelity Speech Synthesis

Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services

Flow-GAN: Bridging implicit and prescribed learning in generative models

Comparison of maximum likelihood and gan-based training of real nvps

Is Generator Conditioning Causally Related to GAN Performance?

Gradient-Based Learning Applied to Document Recognition

Learning Multiple Layers of Features from Tiny Images

An analysis of single-layer networks in unsupervised feature learning

Deep Learning Face Attributes in the Wild

ImageNet Large Scale Visual Recognition Challenge

PSA

Are GANs Created Equal? A Large-Scale Study

Disconnected Manifold Learning for Generative Adversarial Networks

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Improved training of wasserstein gans

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

MaskGAN: Better Text Generation via Filling in the ______

Long short-term memory

Geometric deep learning: going beyond Euclidean data

NetGAN: Generating Graphs via Random Walks

Synthesizing Audio with Generative Adversarial Networks

GANSynth: Adversarial Neural Audio Synthesis

OpenAI Five

Gradient descent GAN optimization is locally stable

Which Training Methods for GANs do actually Converge?

Understanding GANs: the LQG Setting

Global Convergence to the Equilibrium of GANs using Variational Inequalities

The Mechanics of n-Player Differentiable Games

Approximation and Convergence Properties of Generative Adversarial Learning

The Inductive Bias of Restricted f-GANs

On the Limitations of First-Order Approximation in GAN Dynamics

The Loss Surface of Multilayer Networks

GANGs: Generative Adversarial Network Games

Beyond Local Nash Equilibria for Adversarial Networks

An Online Learning Approach to Generative Adversarial Networks

Improved Techniques for Training GANs

GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium

A Note on the Inception Score

Towards GAN Benchmarks Which Require Generalization

Multiscale structural similarity for image quality assessment

On the Quantitative Analysis of Decoder-Based Generative Models

Annealed importance sampling

Geometry Score: A Method For Comparing Generative Adversarial Networks

Assessing Generative Models via Precision and Recall

Skill Rating for Generative Models

Discriminator Rejection Sampling

Revisiting Classifier Two-Sample Tests

Parametric Adversarial Divergences are Good Task Losses for Generative Modeling

Deconvolution and Checkerboard Artifacts

Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability

Deep reinforcement learning from human preferences

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Scaling sgd batch size to 32k for imagenet training

Train longer, generalize better: closing the generalization gap in large batch training of neural networks

Don't Decay the Learning Rate, Increase the Batch Size

An Empirical Model of Large-Batch Training

Science and research policy at the end of Moore’s law

In-datacenter performance analysis of a tensor processing unit

Improving GANs using optimal transport

Large Scale Distributed Deep Networks

Deep learning with Elastic Averaging SGD

Staleness-aware Async-SGD for Distributed Deep Learning

Faster Asynchronous SGD

The Unusual Effectiveness of Averaging in GAN Training

Intriguing properties of neural networks

Adversarial examples from computational constraints

Efficient noise-tolerant learning from statistical queries

Adversarial examples for generative models

Towards deep learning models resistant to adversarial attacks