 While deep neural networks have overwhelmingly established state-of-the-art

 results in many artificial intelligence problems, they can still be

 difficult to develop and debug.

 Recent research on deep learning understanding has focused on

 feature visualization ,

 theoretical guarantees ,

 model interpretability ,

 and generalization .

 In this work, we analyze deep neural networks from a complementary

 perspective, focusing on convolutional models.

 We are interested in understanding the extent to

 which input signals may affect output features, and mapping

 features at any part of the network to the region in the input that

 produces them. The key parameter to associate an output feature to an input

 region is the *receptive field* of the convolutional network, which is

 defined as the size of the region in the input that produces the feature.

 As our first contribution, we

 present a mathematical derivation and an efficient algorithm to compute

 receptive fields of modern convolutional neural networks.

 Previous work

 discussed receptive field

 computation for simple convolutional

 networks where there is a single path from the input to the output,

 providing recurrence equations that apply to this case.

 In this work, we revisit these derivations to obtain a closed-form

 expression for receptive field computation in the single-path case.

 Furthermore, we extend receptive field computation to modern convolutional

 networks where there may be multiple paths from the input to the output.

 To the best of our knowledge, this is the first exposition of receptive

 field computation for such recent convolutional architectures.

 for the computer vision task of object detection, it is important

 understanding a convolutional feature’s span is often required for that goal

 As our second contribution, we fill the void by introducing an

 [open-source library](https://github.com/google-research/receptive_field)

 can be easily employed to analyze a variety of models,

 as presented in this article.

 leading to more productive machine learning research.

Overview of the article

-----------------------

 We consider fully-convolutional neural networks, and derive their receptive

 input signal.

 appropriate.

 First, we derive closed-form expressions when the network has a

 single path from input to output (as in

 AlexNet

 or

 VGG ). Then, we discuss the

 more general case of arbitrary computation graphs with multiple paths from the

 input to the output (as in

 ResNet

 or

 Inception ). We consider

 potential alignment issues that arise in this context, and explain

 an algorithm to compute the receptive field size and locations.

 using our open-source library.

Problem setup

-------------

 output of the lll-th layer, with height hlhlh\_l, width wlwlw\_l and depth

 dldld\_l. We denote the input image by f0f0f\_0. The final output feature map

 corresponds to fLfLf\_{L}.

 111-dimensional input signals and feature maps. For higher-dimensional signals

 (e.g., 222D images), the

* klklk\_l: kernel size (positive integer)

* slsls\_l: stride (positive integer)

* plplp\_l: padding applied to the left side of the input feature map

 (non-negative integer)

 A more general definition of padding may also be considered: negative

 padding, interpreted as cropping, can be used in our derivations

 without any changes. In order to make the article more concise, our

 presentation focuses solely on non-negative padding.

* qlqlq\_l: padding applied to the right side of the input feature map

 (non-negative integer)

 Sorry, your browser does not support inline SVG.

Kernel Size (kl): 2

Left Padding (pl): 1

Right Padding (ql): 1

Stride (sl): 3

fl

fl-1

kl

kl

kl

sl

pl

ql

 We consider layers whose output features depend locally on input features:

 e.g., convolution, pooling, or elementwise operations such as non-linearities,

 addition and filter concatenation. These are commonly used in state-of-the-art

 networks. We define elementwise operations to

 have a “kernel size” of 111, since each output feature depends on a single

 location of the input feature maps.

 Our notation is further illustrated with the simple

 network below. In this case, L=4L=4L=4 and the model consists of a

 convolution, followed by ReLU, a second convolution and max-pooling.

 We adopt the convention where the first output feature for each layer is

 computed by placing the kernel on the left-most position of the input,

 including padding. This convention is adopted by all major deep learning

 libraries.

f0

f1

f2

f3

f4

**Convolution**

 Kernel Size (k1): 3

 Padding (p1, q1): 1

 Stride (s1): 2

Invalid setup! Decrease kernel

size or increase padding.

**ReLU**

 Kernel Size (k2): 1

 Padding (p2, q2): 0

 Stride (s2): 1

**Convolution**

 Kernel Size (k3): 1

 Padding (p3, q3): 0

 Stride (s3): 1

Invalid setup! Decrease kernel

size or increase padding.

**Max Pooling**

 Kernel Size (k4): 3

 Padding (p4, q4): 1

 Stride (s4): 2

Invalid setup! Decrease kernel

size or increase padding.

Single-path networks

--------------------

 In this section, we compute recurrence and closed-form expressions for

 fully-convolutional networks with a single path from input to output

 (e.g.,

 AlexNet

 or

 VGG ).

### Computing receptive field size

 Define rlrlr\_l as the receptive field size of

 other words, rlrlr\_l corresponds to the number of features in feature map

 flflf\_l which contribute to generate one feature in fLfLf\_{L}. Note

 that rL=1rL=1r\_{L}=1.

 input, and generates fLfLf\_{L} as output. Here is an illustration:

 Sorry, your browser does not support inline SVG.

Kernel Size (kL): 2

Padding (pL, qL): 0

Stride (sL): 3

kL

kL

fL-1

fL

 It is easy to see that kLkLk\_{L}

 feature from fLfLf\_{L} is directly connected to kLkLk\_{L} features from

 fL−1fL−1f\_{L-1}. So, rL−1=kLrL−1=kLr\_{L-1} = k\_{L}.

 fl−1fl−1f\_{l-1}.

 entire region where the

 which is why it needs to be deducted.

 As in the illustration below, note that, in some cases, the receptive

 field region may contain “holes”, i.e., some of the input features may be

 unused for a given layer.

 Sorry, your browser does not support inline SVG.

Kernel Size (kl): 1

Padding (pl, ql): 0

Stride (sl): 3

sl⋅rl

sl-1

fl-1

fl

rl=2

 will cover those from the left and the right of the region. For example, if we

 on each side, adding 444 in total. If klklk\_l is even, this works as well,

 since the left and right padding will add to kl−1kl−1k\_l-1.

 Due to border effects, note that the size of the region in the original

 image which is used to compute each output feature may be different. This

 happens if padding is used, in which case the receptive field for

 border features includes the padded region. Later in the article, we

 discuss how to compute the receptive field region for each feature,

 which can be used to determine exactly which image pixels are used for

 each output feature.

 Sorry, your browser does not support inline SVG.

Kernel Size (kl): 5

Padding (pl, ql): 0

Stride (sl): 3

sl⋅rl - (sl-1)

(kl-1)/2

(kl-1)/2

fl-1

fl

rl=2

 So, we obtain the general recurrence equation (which is

 [first-order,

 non-homogeneous, with variable

 ):

rl−1=sl⋅rl+(kl−sl) (1)(1)rl−1=sl⋅rl+(kl−sl) \begin{align}

 r\_{l-1} = s\_l \cdot r\_{l} + (k\_l - s\_l)

 \label{eq:rf\_recurrence}\

 \end{align}

 This equation can be used in a recursive algorithm to compute the receptive

 slsls\_l’s:

r0=L∑l=1((kl−1)l−1∏i=1si)+1(2)(2)r0=∑l=1L((kl−1)∏i=1l−1si)+1\begin{equation}

 r\_0 = \sum\_{l=1}^{L} \left((k\_l-1)\prod\_{i=1}^{l-1}

 s\_i\right) + 1 \label{eq:rf\_recurrence\_final} \end{equation}

 This expression makes intuitive sense, which can be seen by considering some

 special cases. For example, if all kernels are of size 1, naturally the

 receptive field is also of size 1. If all strides are 1, then the receptive

 increases proportionally for all layers below that one. Finally, note that

 padding does not need to be taken into account for this derivation.

### Computing receptive field region in input image

 in the output feature map, in many cases it is also critical to precisely

 localize the region which generated a feature. For example, given feature

 is addressed in this section.

 Let’s denote ululu\_l and vlvlv\_l the left-most

 each map is at coordinate 000).

 Note that uL=vLuL=vLu\_{L} = v\_{L} corresponds to the

 location of the desired feature in fLfLf\_{L}. The figure below illustrates a

 simple 2-layer network, where we highlight the region in f0f0f\_0 which is used

 to compute the first feature from f2f2f\_2. Note that in this case the region

 u0=−1,v0=4u0=−1,v0=4u\_0=-1, v\_0=4.

 Sorry, your browser does not support inline SVG.

Kernel Size (k1): 3

Padding (p1, q1): 1

Stride (s1): 3

Kernel Size (k2): 2

Padding (p2, q2): 0

Stride (s2): 1

u0 = -1

v0 = 4

f0

f1

f2

 compute ul−1,vl−1ul−1,vl−1u\_{l-1},v\_{l-1}?

 clearly be located at −pl−pl-p\_l, since the first feature will be generated by

 u\_{l-1}&= -p\_l + u\_{l}\cdot s\_l \label{eq:rf\_loc\_recurrence\_u} \\

 v\_{l-1}&= -p\_l + v\_{l}\cdot s\_l + k\_l -1

 \label{eq:rf\_loc\_recurrence\_v}

 \end{align}

 needed since in this case we want to find the right-most position.

 Note that these expressions are very similar to the recursion derived for the

 network parameters:

u0=uLL∏i=1si−L∑l=1pll−1∏i=1si(5)(5)u0=uL∏i=1Lsi−∑l=1Lpl∏i=1l−1si\begin{align}

 u\_0&= u\_{L}\prod\_{i=1}^{L}s\_i - \sum\_{l=1}^{L}

 p\_l\prod\_{i=1}^{l-1} s\_i

 \label{eq:rf\_loc\_recurrence\_final\_left}

 \end{align}

 and of the feature location in the output feature map (uLuLu\_{L}).

 And for the right-most feature location v0v0v\_0:

 v\_0&= v\_{L}\prod\_{i=1}^{L}s\_i -\sum\_{l=1}^{L}(1 + p\_l -

 k\_l)\prod\_{i=1}^{l-1} s\_i

 \label{eq:rf\_loc\_recurrence\_final\_right}

 \end{align}

 expression also depends on the kernel sizes (klklk\_l) of each layer.

**Relation between receptive field size and region.**

 You may be wondering that

 the receptive field size r0r0r\_0 must be directly related to u0u0u\_0 and

 1, which we leave as a follow-up exercise for the curious reader. To

 emphasize, this means that we can rewrite

v0=u0+r0−1(7)(7)v0=u0+r0−1\begin{align}

 v\_0&= u\_0 + r\_0 - 1

 \label{eq:rf\_loc\_recurrence\_final\_right\_rewrite}

 \end{align}

**Effective stride and effective padding.**

 To compute u0u0u\_0 and v0v0v\_0 in practice, it

 is convenient to define two other variables, which depend only on the paddings

 and strides of the different layers:

* *effective stride*

Sl=∏Li=l+1siSl=∏i=l+1LsiS\_l = \prod\_{i=l+1}^{L}s\_i: the stride between a

 given feature map flflf\_l and the output feature map fLfLf\_{L}

* *effective padding*

 the padding between a given feature map flflf\_l and the output feature map

 fLfLf\_{L}

 as:

u0=−P0+uL⋅S0(8)(8)u0=−P0+uL⋅S0\begin{align}

 u\_0&= -P\_0 + u\_{L}\cdot S\_0

 \label{eq:rf\_loc\_recurrence\_final\_left\_effective}

 \end{align}

 the output feature map uLuLu\_{L}. When one is interested in computing feature

 locations for a given network, it is handy to pre-compute three variables:

 mapping from any output feature location to the input region which influences

 it.

 It is also possible to derive recurrence equations for the effective stride and

 effective padding. It is straightforward to show that:

Sl−1=sl⋅SlPl−1=sl⋅Pl+pl(9)(10)(9)Sl−1=sl⋅Sl(10)Pl−1=sl⋅Pl+pl\begin{align}

 S\_{l-1}&= s\_l \cdot S\_l \label{eq:effective\_stride\_recurrence} \\

 P\_{l-1}&= s\_l \cdot P\_l + p\_l \label{eq:effective\_padding\_recurrence}

 \end{align}

 These expressions will be handy when deriving an algorithm to solve the case

 for arbitrary computation graphs, presented in the next section.

**Center of receptive field region.**

 It is also interesting to derive an

 expression for the center of the receptive field region which influences a

 particular output feature. This can be used as the location of the feature in

 the input image (as done for recent

 deep learning-based local features , for

 example).

 We define the center of the receptive field region for each layer lll as

 c\_0&= u\_{L}\prod\_{i=1}^{L}s\_i

 - \sum\_{l=1}^{L}

 - \sum\_{l=1}^{L}

 \left(p\_l - \frac{k\_l - 1}{2}\right)\prod\_{i=1}^{l-1} s\_i

 \nonumber \\&= -P\_0 + u\_{L}\cdot S\_0 + \left(\frac{r\_0 - 1}{2}\right)

 \label{eq:rf\_loc\_recurrence\_final\_center\_effective}

 \end{align}

 This expression can be compared to

 Note that the receptive field centers for the different output features are

 receptive field regions for the output features will be aligned to the first

 case all klklk\_l’s must be odd).

**Other network operations.**

 The derivations provided in this section cover most basic operations at the

 core of convolutional neural networks. A curious reader may be wondering

 about other commonly-used operations, such as dilation, upsampling, etc. You

 can find a discussion on these [in the appendix](#other-network-operations).

Arbitrary computation graphs

----------------------------

 Most state-of-the-art convolutional neural networks today (e.g.,

 ResNet  or

 Inception ) rely on models

 where each layer may have more than one input, which

 means that there might be several different paths from the input image to the

 final output feature map. These architectures are usually represented using

 directed acyclic computation graphs, where the set of nodes LL\mathcal{L}

 represents the layers and the set of edges EE\mathcal{E} encodes the

 connections between them (the feature maps flow through the edges).

 The computation presented in the previous section can be used for each of the

 possible paths from input to output independently. The situation becomes

 trickier when one wants to take into account all different paths to find the

 receptive field size of the network and the receptive field regions which

 correspond to each of the output features.

**Alignment issues.**

 The first potential issue is that one output feature may

 be computed using misaligned regions of the input image, depending on the

 used for the computation of each output feature may vary. As a consequence,

 **the receptive field size may not be shift-invariant**

  . This is illustrated in the

 figure below with a toy example, in which case the centers of the regions used

 in the input image are different for the two paths from input to output.

 Sorry, your browser does not support inline SVG.

Kernel Size (k1): 5

Left Pad (p1): 2

Right Pad (q1): 1

Stride (s1): 2

Kernel Size (k2): 3

Left Pad (p2): 0

Right Pad (q2): 0

Stride (s2): 1

Kernel Size (k3): 3

Left Pad (p3): 0

Right Pad (q3): 0

Stride (s3): 1

Add

 are convolutional, while the last layer performs a simple addition.

 The relative position between the receptive field regions of the left and

 right paths is inconsistent for different output features, which leads to a

 Also, note that the receptive field size for each output

 feature may be different. For the second feature from the left, 666 input

 samples are used, while only 555 are used for the third feature. This means

 aligned.

 surface normal estimation, colorization, etc), local feature matching and

 retrieval, among others.

 When the network is aligned, all different paths lead to output features being

 centered consistently in the same locations. All different paths must have the

 same effective stride. It is easy to see that the receptive field size will be

 the largest receptive field among all possible paths. Also, the effective

 padding of the network corresponds to the effective padding for the path with

 largest receptive field size, such that one can apply

 generated an output feature.

 The figure below gives one simple example of an aligned network. In this case,

 the two different paths lead to the features being centered at the same

 locations. The receptive field size is 333, the effective stride is 444 and

 the effective padding is 111.

 Sorry, your browser does not support inline SVG.

Kernel Size (k1): 1

Left Pad (p1): 0

Right Pad (q1): 0

Stride (s1): 4

Kernel Size (k2): 3

Left Pad (p2): 1

Right Pad (q2): 0

Stride (s2): 2

Kernel Size (k3): 1

Left Pad (p3): 0

Right Pad (q3): 0

Stride (s3): 2

Add

**Alignment criteria**

  . More precisely, for a network to be aligned at every

 layer, we need every possible pair of paths iii and jjj to have

 this to happen, we can see from

 satisfied:

 S\_l^{(i)}&= S\_l^{(j)} \label{eq:align\_crit\_1} \\

 \label{eq:align\_crit\_2}

 \end{align}

for all i,j,li,j,li,j,l.

**Algorithm for computing receptive field parameters: sketch.**

 field size and associated parameters for such computation graphs.

 Naturally, a brute-force approach is to use the expressions presented above to

 This method has a worst-case complexity of

 But we can do better. Start by topologically sorting the computation graph.

 The sorted representation arranges the layers in order of dependence: each

 layer’s output only depends on layers that appear before it.

 By visiting layers in reverse topological order, we ensure that all paths

 from a given layer lll to the output layer LLL have been taken into account

 when lll is visited. Once the input layer l=0l=0l=0 is reached, all paths

 have been considered and the receptive field parameters of the entire model

 are obtained. The complexity of this algorithm is

 which is much better than the brute-force alternative.

 As each layer is visited, some bookkeeping must be done in order to keep

 track of the network’s receptive field parameters. In particular, note that

 there might be several different paths from layer lll to the output layer

 LLL. In order to handle this situation, we keep track of the parameters

 for lll and update them if a new path with larger receptive field is found,

Discussion: receptive fields of modern networks

-----------------------------------------------

 In this section, we present the receptive field parameters of modern

 convolutional networks

 are drawn from the [TF-Slim

  , which were computed using the new open-source

 library (script

 The pre-computed parameters for

 AlexNet ,

 VGG ,

 ResNet ,

 Inception

 and

 MobileNet

 are presented in the table below.

 For a more

 comprehensive list, including intermediate network end-points, see

 [this

| --- | --- | --- | --- | --- |

| alexnet\_v2 | 195 | 32 | 64 | [2014](https://arxiv.org/abs/1404.5997v2) |

| vgg\_16 | 212 | 32 | 90 | [2014](https://arxiv.org/abs/1409.1556) |

| mobilenet\_v1 | 315 | 32 | 126 | [2017](https://arxiv.org/abs/1704.04861) |

| resnet\_v1\_50 | 483 | 32 | 239 | [2015](https://arxiv.org/abs/1512.03385) |

| inception\_v2 | 699 | 32 | 318 | [2015](https://arxiv.org/abs/1502.03167) |

| resnet\_v1\_101 | 1027 | 32 | 511 | [2015](https://arxiv.org/abs/1512.03385) |

| inception\_v3 | 1311 | 32 | 618 | [2015](https://arxiv.org/abs/1512.00567) |

| resnet\_v1\_152 | 1507 | 32 | 751 | [2015](https://arxiv.org/abs/1512.03385) |

| resnet\_v1\_200 | 1763 | 32 | 879 | [2015](https://arxiv.org/abs/1512.03385) |

| inception\_v4 | 2071 | 32 | 998 | [2016](https://arxiv.org/abs/1602.07261) |

 As models evolved, from

 AlexNet, to VGG, to ResNet and Inception, the receptive fields increased

 (which is a natural consequence of the increased number of layers).

 includes all of the input pixels.

 We can also relate the growth in receptive fields to increased

 classification accuracy. The figure below plots ImageNet

 top-1 accuracy as a function of the network’s receptive field size, for

 the same networks listed above. The circle size for each data point is

 proportional to the number of floating-point operations (FLOPs) for each

 architecture.

 google.charts.load('current', { 'packages': ['corechart'] });

 google.charts.setOnLoadCallback(drawSeriesChart);

 function drawSeriesChart() {

 var data = google.visualization.arrayToDataTable([

 ['alexnet\_v2', 195, 0.5720, 'alexnet', 1.38],

 ['vgg\_16', 212, 0.7150, 'vgg\_16', 30.71],

 ['inception\_v2', 699, 0.7390, 'inception', 3.88],

 ['inception\_v3', 1311, 0.7800, 'inception', 5.69],

 ['inception\_v4', 2071, 0.8020, 'inception', 12.27],

 ['inception\_resnet\_v2', 3039, 0.8040, 'inception\_resnet', 12.96],

 ['resnet\_v1\_50', 483, 0.7520, 'resnet', 6.97],

 ['resnet\_v1\_101', 1027, 0.7640, 'resnet', 14.40],

 ['resnet\_v1\_152', 1507, 0.7680, 'resnet', 21.82],

 ['mobilenet\_v1', 315, 0.7090, 'mobilenet', 1.14]

 ]);

 var options = {

 title: '',

 hAxis: { title: 'Receptive field size (pixels)', gridlines: { count: 10 } },

 vAxis: { title: 'ImageNet top-1 accuracy', format: 'percent' },

 sizeAxis: { maxSize: 10 },

 legend: { position: 'bottom', textStyle: { fontSize: 12 } },

 bubble: { textStyle: { fontSize: 11 } }

 };

 chart.draw(data, options);

 }

| --- | --- | --- | --- | --- |

| alexnet\_v2 | 195 | 0.572 | alexnet | 1.38 |

| vgg\_16 | 212 | 0.715 | vgg\_16 | 30.71 |

| inception\_v2 | 699 | 0.739 | inception | 3.88 |

| inception\_v3 | 1,311 | 0.78 | inception | 5.69 |

| inception\_v4 | 2,071 | 0.802 | inception | 12.27 |

| inception\_resnet\_v2 | 3,039 | 0.804 | inception\_resnet | 12.96 |

| resnet\_v1\_50 | 483 | 0.752 | resnet | 6.97 |

| resnet\_v1\_101 | 1,027 | 0.764 | resnet | 14.4 |

| resnet\_v1\_152 | 1,507 | 0.768 | resnet | 21.82 |

| mobilenet\_v1 | 315 | 0.709 | mobilenet | 1.14 |

0/0

 We observe a logarithmic relationship between

 classification accuracy and receptive field size, which suggests

 that large receptive fields are necessary for high-level

 recognition tasks, but with diminishing rewards.

 For example, note how MobileNets achieve high recognition performance even

 if using a very compact architecture: with depth-wise convolutions,

 the receptive field is increased with a small compute footprint.

 In comparison, VGG-16 requires 27X more FLOPs than MobileNets, but produces

 a smaller receptive field size; even if much more complex, VGG’s accuracy

 is only slightly better than MobileNet’s.

 This suggests that networks which can efficiently generate large receptive

 fields may enjoy enhanced recognition performance.

 residual connections, batch normalization, to name only a few.

 In other words, while we conjecture that a large receptive field is necessary,

 by no means it is sufficient.

 Additional experimentation is needed to confirm this hypothesis: for

 example, researchers may experimentally investigate how classification

 accuracy changes as kernel sizes and strides vary for different

 architectures. This may indicate if, at least for those architectures, a

 large receptive field is necessary.

 the feature, and consequently carry more weight.

 The relative importance of each input pixel defines the

 *effective receptive field* of the feature.

 Recent work

 provides a mathematical formulation and a procedure to measure effective

 receptive fields, experimentally observing a Gaussian shape,

 with the peak at the receptive field center. Better understanding the

 relative importance of input pixels in convolutional neural networks is

 an active research topic.

Visualizing higher-layer features of a deep network

Visualizing and Understanding Convolutional Networks

Global Optimality in Neural Network Training

On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport

Object Detectors Emerge in Deep Scene CNNs

Interpreting Deep Visual Representations via Network Dissection

In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Understanding Deep Learning Requires Rethinking Generalization

A Guide to Receptive Field Arithmetic for Convolutional Neural Networks

What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?

ImageNet Classification with Deep Convolutional Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Deep Residual Learning for Image Recognition

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Large-Scale Image Retrieval with Attentive Deep Local Features

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks

Computing Receptive Fields of Convolutional Neural Networks.html


Thread: Differentiable Self-organizing Systems

Alexander Mordvintsev
Ettore Randazzo
Eyvind Niklasson
Michael Levin
Sam Greydanus

Abstract: Self-organisation is omnipresent on all scales of biological life. From complex interactions between molecules       forming structures such as proteins, to cell colonies achieving global goals like exploration by means of the       individual cells collaborating and communicating, to humans forming collectives in society such as tribes,       governments or countries. The old adage “the whole is greater than the sum of its parts”, often ascribed to       Aristotle, rings true everywhere we look.

 Aristotle, rings true everywhere we look.

 communities.

Articles & Comments

-------------------

 self-organizing systems,

 interspersed with critical commentary from several experts in adjacent fields.

 The thread will be a living document, with new articles added over time.

 Articles and comments are presented below in chronological order:

###

[Growing Neural Cellular Automata](https://distill.pub/2020/growing-ca/)

### Authors

### Affiliations

[Alexander Mordvintsev](https://znah.net/),

 Ettore Randazzo,

 [Eyvind Niklasson](https://eyvind.me/),

 [Michael Levin](http://www.drmichaellevin.org/)

[Google](https://research.google/),

 [Allen Discovery Center](https://allencenter.tufts.edu/)

 arbitrary structure starting from a single cell.

 [Read Full Article](https://distill.pub/2020/growing-ca/)

###

[Self-classifying MNIST Digits](https://distill.pub/2020/selforg/mnist/)

### Authors

### Affiliations

Ettore Randazzo,

 [Alexander Mordvintsev](https://znah.net/),

 [Eyvind Niklasson](https://eyvind.me/),

 [Michael Levin](http://www.drmichaellevin.org/),

 [Sam Greydanus](https://greydanus.github.io/about.html)

[Google](https://research.google/),

 [Allen Discovery Center](https://allencenter.tufts.edu/),

 [Oregon State University and the ML Collective](http://mlcollective.org/)

 perturbations with a learned self-correcting classification behaviour.

 [Read Full Article](https://distill.pub/2020/selforg/mnist/)

###

[Self-Organising Textures](https://distill.pub/selforg/2021/textures/)

### Authors

### Affiliations

[Eyvind Niklasson](https://eyvind.me/),

 [Alexander Mordvintsev](https://znah.net/),

 Ettore Randazzo,

 [Michael Levin](http://www.drmichaellevin.org/)

[Google](https://research.google/),

 [Allen Discovery Center](https://allencenter.tufts.edu/),

 to robust and unexpected behaviors.

 [Read Full Article](https://distill.pub/selforg/2021/textures/)

###

### Authors

### Affiliations

[Ettore Randazzo](https://oteret.github.io/),

 [Alexander Mordvintsev](https://znah.net/),

 [Eyvind Niklasson](https://eyvind.me/),

 [Michael Levin](http://www.drmichaellevin.org/)

[Google](https://research.google/),

 [Allen Discovery Center](https://allencenter.tufts.edu/),

This work takes existing Neural CA models and shows how they can be

adversarially reprogrammed to perform novel tasks.

MNIST CA can be deceived into outputting incorrect classifications and

the patterns in Growing CA can be made to have their shape and colour

altered.

 [Read Full Article](https://distill.pub/selforg/2021/adversarial/)

#### This is a living document

 Expect more articles on this topic, along with critical comments from

 experts.

Get Involved

------------

 Critical

 commentary and discussion of existing articles is also welcome. The thread

 is organized through the open `#selforg` channel on the

 [Distill slack](http://slack.distill.pub/). Articles can be

 suggested there, and will be included at the discretion of previous

 authors in the thread, or in the case of disagreement by an uninvolved

 editor.

 If you would like get involved but don’t know where to start, small

 projects may be available if you ask in the channel.

About the Thread Format

-----------------------

 Part of Distill’s mandate is to experiment with new forms of scientific

 publishing. We believe that that reconciling faster and more continuous

 approaches to publication with review and discussion is an important open

 problem in scientific publishing.

 Threads are collections of short articles, experiments, and critical

 commentary around a narrow or unusual research topic, along with a slack

 channel for real time discussion and collaboration. They are intended to

 be earlier stage than a full Distill paper, and allow for more fluid

 publishing, feedback and discussion. We also hope they’ll allow for wider

 participation. Think of a cross between a Twitter thread, an academic

 workshop, and a book of collected essays.

 Threads are very much an experiment. We think it’s possible they’re a

 great format, and also possible they’re terrible. We plan to trial two

 such threads and then re-evaluate our thought on the format.

Thread Differentiable Self-organizing Systems.html


Feature Visualization

Chris Olah
Alexander Mordvintsev
Ludwig Schubert

Abstract: There is a growing sense that neural networks need to be interpretable to humans.   The field of neural network interpretability has formed in response to these concerns.   As it matures, two major threads of research have begun to coalesce: feature visualization and attribution.

![](Feature%20Visualization_files/neuron.png)

![](Feature%20Visualization_files/channel.png)

![](Feature%20Visualization_files/attribution-1.png)

![](Feature%20Visualization_files/attribution-2.jpg)

**Attribution**

 As a young field, neural network interpretability does not yet

have standardized terminology.

 Attribution has gone under many different names in the

literature — including “feature visualization”! — but recent work seems

to prefer terms like “attribution” and “saliency maps”.

 This article focuses on feature visualization.

 While feature visualization is a powerful tool, actually getting it to

 work involves a number of details.

 In this article, we examine the major issues and explore common

approaches to solving them.

 We find that remarkably simple methods can produce high-quality

visualizations. Along the way we introduce a few tricks for exploring

variation in what neurons react to, how they interact, and how to

improve the optimization process.

---

Feature Visualization by Optimization

-------------------------------------

 Neural networks are, generally speaking, differentiable with respect

to their inputs.

 If we want to find out what kind of input would cause a certain

behavior — whether that’s an internal neuron firing or the final output

behavior — we can use derivatives to iteratively tweak the input

 towards that goal .

Step 0

→

Step 4

→

Step 48

→

Step 2048

 While conceptually simple, there are subtle challenges in getting the

optimization to work. We will explore them, as well as common approaches

### Optimization Objectives

 What do we want examples of?

 This is the core question in working with examples, regardless of

whether we’re searching through a dataset to find the examples, or

optimizing images to create them from scratch.

 We have a wide variety of options in what we search for:

 #optimization-objectives .objectives {

 grid-column: text;

 grid-template-columns: repeat(1, 1fr);

 }

 #optimization-objectives .objective {

 display: grid;

 grid-template-columns: repeat(2, 1fr) 1.1fr;

 }

 #optimization-objectives .objectives figcaption {

 padding: 4px 8px;

 word-wrap: break-word;

 word-break: break-word;

 }

 #optimization-objectives .objective .objective-icon {

 padding: 8px;

 }

 @media (min-width: 512px) {

 #optimization-objectives .objectives {

 grid-template-columns: repeat(5, 1fr);

 grid-column-gap: 16px;

 }

 #optimization-objectives .objective {

 display: flex;

 flex-flow: column;

 }

 #optimization-objectives .objectives figcaption {

 padding: 0;

 padding-top: 4px;

 word-wrap: break-word;

 word-break: break-word;

 }

 }

`**n**` layer index

`**x,y**` spatial position

`**z**` channel index

`**k**` class index

![](Feature%20Visualization_files/neuron.png)

**Neuron**

`layern[x,y,z]`

![](Feature%20Visualization_files/channel.png)

**Channel**

`layern[:,:,z]`

![](Feature%20Visualization_files/layer.png)

**Layer**/DeepDream

`layern[:,:,:]2`

softmax

![](Feature%20Visualization_files/logits.png)

**Class Logits**

`pre_softmax[k]`

softmax

![](Feature%20Visualization_files/logits_post.png)

**Class Probability**

`softmax[k]`

 We used the channel objective to create most of the images in this article.

 after the softmax.

 One can see the logits as the evidence for each class, and the

probabilities as the likelihood of each class given the evidence.

 Unfortunately, the easiest way to increase the probability softmax

gives to a class is often to make the alternatives unlikely rather than

to make the class of interest likely .

 While the standard explanation is that maximizing probability

doesn’t work very well because you can just push down evidence for other

 classes, an alternate hypothesis is that it’s just harder to optimize

through the softmax function. We understand this has sometimes been an

issue in adversarial examples, and the solution is to optimize the

LogSumExp of the logits instead. This is equivalent to optimizing

softmax but generally more tractable. Our experience was that the

LogSumExp trick doesn’t seem better than dealing with the raw

probabilities.

 Regardless of why that happens, it can be fixed by very strong

regularization with generative models. In this case the probabilities

can be a very principled thing to optimize.

 and objectives used in optimization-based model inversion ,

 which help us understand what information a model keeps and what it

throws away.

 We are only at the beginning of understanding which objectives are

interesting, and there is a lot of room for more work in this area.

### Why visualize by optimization?

**Dataset Examples** show us what neurons respond to in practice

**Optimization** isolates the causes of behavior from mere correlations.

A neuron may not be detecting what you initially thought.

Baseball—or stripes?

*mixed4a, Unit 6*

Animal faces—or snouts?

*mixed4a, Unit 240*

Clouds—or fluffiness?

*mixed4a, Unit 453*

Buildings—or sky?

*mixed4a, Unit 492*

 Optimization also has the advantage of flexibility.

 For example, if we want to study how neurons jointly represent

information,

 we can easily ask how a particular example would need to be different

for an additional neuron to activate.

 This flexibility can also be helpful in visualizing how features

evolve as the network trains.

 If we were limited to understanding the model on the fixed examples in

 our dataset, topics like these ones would be much harder to explore.

 On the other hand, there are also significant challenges to

visualizing features with optimization.

 In the following sections we’ll examine techniques to get diverse

visualizations, understand how neurons interact, and avoid high

frequency artifacts.

---

Diversity

---------

 Do our examples show us the full picture?

 Dataset examples have a big advantage here.

 By looking through our dataset, we can find diverse examples.

 It doesn’t just give us ones activating a neuron intensely:

**Negative** optimized

**Minimum** activation examples

Slightly negative activation examples

**Positive** optimized

Slightly positive activation examples

**Maximum** activation examples

**Layer mixed 4a, unit 492**

[Reproduce in a

 In contrast, optimization generally gives us just one extremely

positive example — and if we’re creative, a very negative example as

well.

 Is there some way that optimization could also give us this diversity?

### Achieving Diversity with Optimization

 A given feature of a network may respond to a wide range of inputs.

 On the class level, for example, a classifier that has been trained to

 recognize dogs should recognize both closeups of their faces as well as

 wider profile images — even though those have quite different visual

appearances.

 Early work by Wei *et al.*

attempts to demonstrate this “intra-class” diversity by recording

activations over the entire training set, clustering them and optimizing

 for the cluster centroids, revealing the different facets of a class

that were learned.

 A different approach by Nguyen, Yosinski, and collaborators was to

search through the dataset for diverse examples and use those as

starting points for the optimization process .

 The idea is that this initiates optimization in different facets of

the feature so that the resulting example from optimization will

demonstrate that facet.

 In more recent work, they combine visualizing classes with a

generative model, which they can sample for diverse examples .

 Their first approach had limited success, and while the generative

model approach works very well — we’ll discuss it more in the section on

 For this article we use an approach based on ideas from artistic

style transfer. Following that work, we begin by computing the Gram

 Gi,j=∑x,ylayern[x, y, i]⋅layern[x, y, j]

 Gi,j=x,y∑layern[x, y, i]⋅layern[x, y, j]

 Cdiversity=−∑a∑b≠a vec(Ga)⋅vec(Gb)∣∣vec(Ga)∣∣ ∣∣vec(Gb)∣∣

 C\_{\text{diversity}} = - \sum\_{a} \sum\_{b\neq a} ~

\frac{\text{vec}(G\_a) \cdot

\text{vec}(G\_b)}{||\text{vec}(G\_a)||~||\text{vec}(G\_b)||}

 Cdiversity=−a∑b≠a∑ ∣∣vec(Ga)∣∣ ∣∣vec(Gb)∣∣vec(Ga)⋅vec(Gb)

 One possibility is to penalize the cosine similarity of different examples.

![](Feature%20Visualization_files/mixed4a_97_optimized.png)

Simple Optimization

![](Feature%20Visualization_files/mixed4a_97_diversity.png)

[Reproduce in a

![](Feature%20Visualization_files/mixed4a_97_examples.jpg)

Dataset examples

 Diverse feature visualizations allow us to more closely pinpoint what

activates a neuron, to the degree that we can make, and — by looking at

 For example, let’s examine this simple optimization result.

![](Feature%20Visualization_files/mixed4a_143_optimized.png)

Simple optimization

 Looking at it in isolation one might infer that this neuron activates

on the top of dog heads, as the optimization shows both eyes and only

downward curved edges.

 Looking at the optimization with diversity however, we see

optimization results which don’t include eyes, and also one which

includes upward curved edges. We thus have to broaden our expectation of

 what this neuron activates on to be mostly about the fur texture.

Checking this hypothesis against dataset examples shows that is broadly

correct. Note the spoon with a texture and color similar enough to dog

fur for the neuron to activate.

![](Feature%20Visualization_files/mixed4a_143_diversity.png)

 Optimization with diversity. *Layer mixed4a, Unit 143*

![](Feature%20Visualization_files/mixed4a_143_examples.jpg)

Dataset examples

 The effect of diversity can be even more striking in higher level

neurons, where it can show us different types of objects that stimulate a

 neuron.

 For example, one neuron responds to different kinds of balls, even

though they have a variety of appearances.

![](Feature%20Visualization_files/mixed5a_9_optimized.png)

Simple Optimization

![](Feature%20Visualization_files/mixed5a_9_diversity.png)

![](Feature%20Visualization_files/mixed5a_9_examples.jpg)

Dataset examples

 This simpler approach has a number of shortcomings:

 For one, the pressure to make examples different can cause unrelated

artifacts (such as eyes) to appear.

 Additionally, the optimization may make examples be different in an

unnatural way.

 For example, in the above example one might want to see examples of

soccer balls clearly separated from other types of balls like golf or

tennis balls.

 Dataset based approaches such as Wei *et al.*

 can split features apart more naturally — however they may not be as

helpful in understanding how the model will behave on different data.

 Diversity also starts to brush on a more fundamental issue: while the

examples above represent a mostly coherent idea, there are also neurons

that represent strange mixtures of ideas.

 Below, a neuron responds to two types of animal faces, and also to car

 bodies.

![](Feature%20Visualization_files/mixed4e_55_optimized.png)

Simple Optimization

![](Feature%20Visualization_files/mixed4e_55_diversity.png)

![](Feature%20Visualization_files/mixed4e_55_examples.jpg)

Dataset examples

---

Interaction between Neurons

---------------------------

 If neurons are not the right way to understand neural nets, what is?

 This framing unifies the concepts “neurons” and “combinations of

neurons” as “vectors in activation space”. It allows us to ask: Should

we expect the directions of the basis vectors to be any more

interpretable than the directions of other vectors in this space?

 More recently Bau, Zhou *et al.*

 found the directions of the basis vectors to be interpretable more

often than random directions.

 Our experience is broadly consistent with both results; we find that

random directions often seem interpretable, but at a lower rate than

basis directions.

[Reproduce in a

*mixed3a, random direction*

*mixed4c, random direction*

*mixed4d, random direction*

*mixed5a, random direction*

By jointly optimizing two neurons we can get a sense of how they interact.

[Reproduce in a

Neuron 1

Neuron 2

Jointly optimized

 These examples show us how neurons jointly represent images.

 The optimization objective is a linear

interpolation between the individual channel objectives. To get the

interpolations to look better, we also add a small alignment objective

that encourages lower layer activations to be similar. We additionally

use a combination of separate and shared image parameterizations to make

 it easier for the optimization algorithm to cause objects to line up,

while still giving it the freedom to create any image it needs to.

 This is similar to interpolating in the latent space of generative models.

[Reproduce in a

Layer 4a, Unit 476

Layer 4a, Unit 460

 This is only starting to scratch the surface of how neurons interact.

 The truth is that we have almost no clue how to select meaningful

directions, or whether there even exist particularly meaningful

directions.

 Independent of finding directions, there are also questions on how

directions interact — for example, interpolation can show us how a small

 number of directions interact, but in reality there are hundreds of

directions interacting.

---

The Enemy of Feature Visualization

----------------------------------

 If you want to visualize features, you might just optimize an image to

 make neurons fire.

 Unfortunately, this doesn’t really work.

 Instead, you end up with a kind of neural network optical

illusion — an image full of noise and nonsensical high-frequency

patterns that the network responds strongly to.

Even if you carefully tune learning rate, you’ll get noise.

Optimization results are enlarged to show detail and artifacts.

[Reproduce in a

Learning Rate

(0.05)

Step 1

Step 32

Step 128

Step 256

Step 2048

 but the image is dominated by these high frequency patterns.

 We don’t fully understand why these high frequency patterns form,

 but an important part seems to be strided convolutions and pooling

operations, which create high-frequency patterns in the gradient .

input

conv2d0

conv2d1

conv2d2

mixed3a

mixed3b

mixed4a

mixed4b

mixed4c

mixed4d

mixed4e

mixed5a

 These high-frequency patterns show us that, while optimization based

visualization’s freedom from constraints is appealing, it’s a

double-edged sword.

 Without any constraints on images, we end up with adversarial

examples.

 These are certainly interesting, but if we want to understand how

these models work in real life, we need to somehow move past them…

### The Spectrum of Regularization

 Dealing with this high frequency noise has been one of the primary

challenges and overarching threads of feature visualization research.

 If you want to get useful visualizations, you need to impose a more

natural structure using some kind of prior, regularizer, or constraint.

 In fact, if you look at most notable papers on feature visualization,

one of their main points will usually be an approach to regularization.

 Researchers have tried a lot of different things!

 In the middle we have three main families of regularization options.

 #feature-vis-history .row {

 }

 #feature-vis-history .line {

 width: 900px;

 }

 #feature-vis-history .row .info {

 display: inline-block;

 width: 320px;

 height: 56px;

 vertical-align: top;

 }

 #feature-vis-history .row .info img {

 width: 56px;

 height: 56px;

 border-radius: 5px;

 background: #EEE;

 }

 #feature-vis-history figcaption {

 line-height: 16px;

 vertical-align: top;

 }

 #feature-vis-history .row .info figcaption {

 width: 250px;

 margin-left: 8px;

 display: inline-block;

 }

 #feature-vis-history .row.header-row .info figcaption {

 margin-left: 0;

 }

 #feature-vis-history .header-row .sub-headers {

 display: inline-block;

 margin-top: 16px;

 margin-bottom: 10px;

 vertical-align: top;

 }

 #feature-vis-history .header-row .sub-headers figcaption {

 width: 100px; /\*104px;\*/

 display: inline-block;

 word-wrap: break-word;

 word-break: keep-all;

 }

 #feature-vis-history .row .spacer {

 display: inline-block;

 height: 48px;

 width: 45px;

 }

 #feature-vis-history .info p {

 margin-bottom: 4px;

 }

 #feature-vis-history .info .paper-title {

 font-weight: bold;

 }

 #feature-vis-history .info .paper-text {

 }

 @media (max-width: 1100px) {

 #feature-vis-history .row .spacer {

 display: inline-block;

 height: 48px;

 width: 24px;

 }

 #feature-vis-history .line {

 width: 750px;

 }

 }

 #feature-vis-history .row .category-check-container {

 /\*vertical-align: middle;\*/

 display: inline-block;

 margin: 8px;

 margin-left: 0px;

 margin-right: 70px;

 width: 32px;

 height: 32px;

 border-radius: 5px;

 border: 1px solid #CCC;

 }

 #feature-vis-history .row .category-check-container .category-check {

 margin: 6px;

 width: 20px;

 height: 20px;

 border-radius: 6px;

 }

 #feature-vis-history .row .category-check-container .set {

 background: #909092;

 }

 #feature-vis-history .line {

 margin-top: 4px;

 margin-bottom: 4px;

 margin-left: 67px;

 height: 1px;

 background: #EAEAEA;

 }

**Weak Regularization**

 avoids misleading correlations, but is less connected to real use.

**Unregularized**

**FrequencyPenalization**

**TransformationRobustness**

**Strong Regularization**

 gives more realistic examples at risk of misleading correlations.

**LearnedPrior**

**DatasetExamples**

![](Feature%20Visualization_files/Erhan2009.png)

Erhan, *et al.*, 2009

Introduced core idea. Minimal regularization.

![](Feature%20Visualization_files/Szegedy2013.png)

Szegedy, *et al.*, 2013

Adversarial examples. Visualizes with dataset examples.

![](Feature%20Visualization_files/Mahendran2015.png)

Mahendran & Vedaldi, 2015

Introduces total variation regularizer. Reconstructs input from representation.

![](Feature%20Visualization_files/Nguyen2015.png)

Nguyen, *et al.*, 2015

Explores counterexamples. Introduces image blurring.

![](Feature%20Visualization_files/Mordvintsev2015.png)

Mordvintsev, *et al.*, 2015

Introduced jitter & multi-scale. Explored GMM priors for classes.

![](Feature%20Visualization_files/Oygard2015.png)

Øygard, *et al.*, 2015

Introduces gradient blurring.

(Also uses jitter.)

![](Feature%20Visualization_files/Tyka2016.png)

Tyka, *et al.*, 2016

Regularizes with bilateral filters.

(Also uses jitter.)

![](Feature%20Visualization_files/Mordvintsev2016.png)

Mordvintsev, *et al.*, 2016

Normalizes gradient frequencies.

(Also uses jitter.)

![](Feature%20Visualization_files/Nguyen2016a.png)

Nguyen, *et al.*, 2016

Paramaterizes images with GAN generator.

![](Feature%20Visualization_files/Nguyen2016b.png)

Nguyen, *et al.*, 2016

Uses denoising autoencoder prior to make a generative model.

### Three Families of Regularization

 If we think about blurring in Fourier space, it is equivalent to

adding a scaled L2 penalty to the objective, penalizing each

Fourier-component based on its frequency.

 These techniques are in some ways very similar to the above and in

Frequency penalization directly targets high frequency noise

[Reproduce in a

L1

(-0.05)

Total Variation

(-0.25)

Blur

(-1)

Step 1

Step 32

Step 128

Step 256

Step 2048

 Even a small amount seems to be very effective in the case of images ,

 especially when combined with a more general regularizer for high-frequencies .

[Reproduce in a

Jitter

(1px)

Rotate

(5°)

Scale

(1.1×)

Step 1

Step 32

Step 128

Step 256

Step 2048

**Learned priors.**

 Our previous regularizers use very simple heuristics to keep examples

reasonable.

 A natural next step is to actually learn a model of the real data and

try to enforce that.

 With a strong model, this becomes similar to searching over the

dataset.

 This approach produces the most photorealistic visualizations, but it

may be unclear what came from the model being visualized and what came

from the prior.

 such as a GAN or VAE,

 and optimize within that latent space .

 this allows you to jointly optimize for the prior along with your objective .

 When one optimizes for the prior and the probability of a class, one

recovers a generative model of the data conditioned on that particular

class.

 Finally, Wei *et al.*

approximate a generative model prior, at least for the color

distribution, by penalizing distance between patches of the output and

the nearest patches retrieved from a database of image patches collected

 from the training data.

---

Preconditioning and Parameterization

------------------------------------

 It’s not clear this is really a regularizer:

 If it isn’t a regularizer, what does transforming the gradient like this do?

 You can think of it as doing steepest descent to optimize the same objective,

 Gradient blurring is equivalent

 to gradient descent in a different parameterization of image space,

where high frequency dimensions are stretched to make moving in those

 This changes which direction of descent will be steepest, and how fast

 the optimization moves in each direction, but it does not change what

the minimums are.

 If there are many local minima, it can stretch and shrink their basins

 of attraction, changing which ones the optimization process falls into.

 As a result, using the right preconditioner can make an optimization

problem radically easier.

 How can we choose a preconditioner that will give us these benefits?

 A good first guess is one that makes your data decorrelated and whitened.

 In the case of images this means doing gradient descent in the Fourier basis,

 This points to a profound fact about the Fourier transform.

 As long as a correlation is consistent across spatial

positions — such as the correlation between a pixel and its left

neighbor being the same across all positions of an image — the Fourier

coefficients will be independent variables.

 To see this, note that such a spatially consistent correlation can

be expressed as a convolution, and by the convolution theorem becomes

pointwise multiplication after the Fourier transform.

 with frequencies scaled so that they all have equal energy.

 Note that we have to be careful to get the colors to be

decorrelated, too. The Fourier transforms decorrelates spatially, but a

correlation will still exist between colors.

 To address this, we explicitly measure the correlation between

colors in the training set and use a Cholesky decomposition to

decorrelate them. Compare the directions of steepest decent before and

after decorrelating colors:

![](Feature%20Visualization_files/correlated_colors.jpeg)

 Correlated Colors

![](Feature%20Visualization_files/decorrelated_colors.jpeg)

 Decorrelated Colors

 Three directions of steepest descent under different notions of distance

Image

**L∞ metric**

used in adverserial work

**L2 metric**

regular gradient

**decorrelated space**

 All of these directions are valid descent directions for the same objective,

 but we can see they’re radically different.

 Notice that optimizing in the decorrelated space reduces high frequencies,

 while using L∞ increases them.

 It’s hard to do really fair comparisons because of hyperparameters, but the

 resulting visualizations seem a lot better — and develop faster, too.

[Reproduce in a

Learning Rate

(0.05)

Decorrelated Space

Transformation Robustness

Step 1

Step 32

Step 128

Step 256

Step 2048

 (Unless otherwise noted, the images in this article were optimizing in

 the decorrelated space and a suite of transformation robustness

techniques.

 Images were optimized for 2560 steps in a color-decorrelated

fourier-transformed space, using Adam at a learning rate of 0.05.

 We used each of following transformations in the given order at each

 step of the optimization:

 • Padding the input by 16 pixels to avoid edge artifacts

 • Jittering by up to 16 pixels

 • Jittering a second time by up to 8 pixels

 • Cropping the padding

)

 Is the preconditioner merely accelerating descent, bringing us to the

same place

 normal gradient descent would have brought us if we were patient

enough?

 Or is it also regularizing, changing which local minima we get

attracted to?

 It’s hard to tell for sure.

 On the one hand, gradient descent seems to continue improving as you

exponentially increase the number of optimization steps — it hasn’t

converged, it’s just moving very slowly.

 On the other hand, if you turn off all other regularizers, the

preconditioner seems to reduce high-frequency patterns.

---

Conclusion

----------

 Neural feature visualization has made great progress over the last few years.

 In the quest to make neural networks interpretable, feature visualization

 stands out as one of the most promising and developed research directions.

 By itself, feature visualization will never give a completely satisfactory

 understanding. We see it as one of the fundamental building blocks that,

 There remains still a lot of important work to be done in improving

feature visualization.

 Some issues that stand out include understanding neuron interaction,

finding which units are most meaningful for understanding neural net

activations, and giving a holistic view of the facets of a feature.

Going deeper with convolutions

Imagenet: A large-scale hierarchical image database

Visualizing higher-layer features of a deep network

Inceptionism: Going deeper into neural networks

Deep inside convolutional networks: Visualising image classification models and saliency maps

A neural algorithm of artistic style

Understanding deep image representations by inverting them

Understanding Intra-Class Knowledge Inside {CNN}

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Intriguing properties of neural networks

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Deconvolution and checkerboard artifacts

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Visualizing GoogLeNet Classes

Class visualization with bilateral filters

DeepDreaming with TensorFlow

Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

Tensorflow: Large-scale machine learning on heterogeneous distributed systems


Shan Carter
David Ha
Ian Johnson
Chris Olah

Four Experiments in Handwriting with a Neural Network

=====================================================

Let’s start with generating new strokes based on your handwriting input

-----------------------------------------------------------------------

Play/Pause

Clear

Length of prediction

20

Variation1

0.1

 dt-byline {

 font-size: 12px;

 line-height: 18px;

 display: block;

 border-top: 1px solid rgba(0, 0, 0, 0.1);

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 color: rgba(0, 0, 0, 0.5);

 padding-top: 12px;

 padding-bottom: 12px;

 }

 dt-article.centered dt-byline {

 text-align: center;

 }

 dt-byline a,

 dt-article dt-byline a {

 text-decoration: none;

 border-bottom: none;

 }

 dt-article dt-byline a:hover {

 text-decoration: underline;

 border-bottom: none;

 }

 dt-byline .authors {

 text-align: left;

 }

 dt-byline .name {

 display: inline;

 text-transform: uppercase;

 }

 dt-byline .affiliation {

 display: inline;

 }

 dt-byline .date {

 display: block;

 text-align: left;

 }

 dt-byline .year, dt-byline .month {

 display: inline;

 }

 dt-byline .citation {

 display: block;

 text-align: left;

 }

 dt-byline .citation div {

 display: inline;

 }

 @media(min-width: 768px) {

 dt-byline {

 }

 }

 @media(min-width: 1080px) {

 dt-byline {

 border-bottom: none;

 margin-bottom: 70px;

 }

 dt-byline a:hover {

 color: rgba(0, 0, 0, 0.9);

 }

 dt-byline .authors {

 display: inline-block;

 }

 dt-byline .author {

 display: inline-block;

 margin-right: 12px;

 /\*padding-left: 20px;\*/

 /\*border-left: 1px solid #ddd;\*/

 }

 dt-byline .affiliation {

 display: block;

 }

 dt-byline .author:last-child {

 margin-right: 0;

 }

 dt-byline .name {

 display: block;

 }

 dt-byline .date {

 border-left: 1px solid rgba(0, 0, 0, 0.1);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .year, dt-byline .month {

 display: block;

 }

 dt-byline .citation {

 border-left: 1px solid rgba(0, 0, 0, 0.15);

 padding-left: 15px;

 margin-left: 15px;

 display: inline-block;

 }

 dt-byline .citation div {

 display: block;

 }

 }

[Shan Carter](http://shancarter.com/)

[Google Brain](http://g.co/brain)

[David Ha](https://github.com/hardmaru)

[Google Brain](http://g.co/brain)

[Ian Johnson](https://github.com/enjalot)

[Google Cloud](http://cloud.google.com/)

[Chris Olah](http://colah.github.io/)

[Google Brain](http://g.co/brain)

Dec. 6

2016

[Citation:

Carter, et al., 2016](#citation)

 Neural networks are an extremely successful approach to machine

learning, but it’s tricky to understand why they behave the way they do.

 This has sparked a lot of interest and effort around trying to

understand and visualize them, which we think is so far just scratching

the surface of what is possible.

 and visualizing it in a number of ways. The model is quite simple (so

as to run well in the browser) so the generated output mostly produces

gibberish letters and words (albeit, gibberish that look like real

handwriting), but it is still useful for our purposes of exploring

visualization techniques.

In the end we don’t have some ultimate answer or visualization, but

 we do have some interesting ideas to share. Ultimately we hope they

make it easier to divine some meaning from the internals of these model.

Looking at the Output of the Model

----------------------------------

Our first experiment is the most obvious: when we want to see how

well someone has learned a task we usually ask them to demonstrate it.

So, let’s ask our model to write something for us and see how well it

does.

Play/Pause

Variation[1](#footnote1)

0.6

Most of the marks are gibberish, but many of them are surprisingly

convincing. Some real (or real-ish) words even start to appear. One

surprising thing you’ll notice is that the general style of handwriting

is more or less consistent within a sample. This is because the type of

architecture used for this model (LSTM) has a mechanism for remembering

previous strokes. It is therefore able to remember things like how loopy

 or jerky the handwriting is, or which letter preceeded the current one.

Even with this memory, the technique for generating samples from

neural networks is probabilistic, meaning each of these samples is one

of a much, much larger possibility space. Viewing them one at a time

feels unsatisfying — how can we infer much from one or two samples when

there are nearly infinite possibilities? If something looks wrong, how

do we know if it was because there is something fundamentally wrong with

 our model or if it was just dumb luck?

At each iteration our model could have produced many paths. Instead

 of picking one and throwing the rest away, let’s draw, say, 50 of them.

 Green strokes below are places where the model would have veered

rightward from the chosen stroke, orange is where it would have veered

leftward.

Generated sample 1

Generated sample 2

Generated sample 3

Generated sample 4

Generated sample 5

Straight line

Steps

8

Variation[1](#footnote1)

0.5

With this technique we are casting a light into a wider possibility

 space. You can see some areas where there was general consensus as to

what to do next. Other areas had more of a “anything can happen next”

feeling. Others seem to show a possible fork in the road. We can be

drawing an “a” or “g”. A few steps later it’s clear the model has

converged on a cursive “g”.

In addition to visualizing samples generated by the model, this

technique can also be applied to human-generated samples. Below we can

see what the model would have done at each point if it had taken over

for the person.

Validation sample 0

Validation sample 1

Validation sample 2

Validation sample 3

Validation sample 4

Validation sample 5

Validation sample 6

Steps

8

Variation[1](#footnote1)

0.5

It is obvious from these experiments that this model has learned

quite a lot about human handwriting. Which sort of raises the question,

can we extract that knowledge in any meaningful way, rather than just

blindly using it to mimic handwriting?

Examining the Internals of the Model

------------------------------------

Our model has 500 cells which act as a sort of memory which it will

 use as part of its input when deciding what to generate. If we can see

what those cells are doing as the model progresses we may be able to

gain some intuitive understanding about what the model is doing.

We begin by showing the activation of the cells over time. Each

column in the heatmap below represents one line segment of the

handwriting. Each row represents one cell of the model and is colored by

 its activations on that part of the stroke. By inspecting the diagram

you may be able to see some patterns in the way certain cells activate

for certain types of strokes. You can change which cell is used to color

 the strokes by clicking on the diagram.

Activation value

-4-2024

Step 0

Step 0

Cell 93

Activations at **step 0**

You may have been able to pick out one or two cells with

interesting patterns, but we have reason to believe that cells work in

tandem, such that an individual cell’s activity may not be as

interesting as a group of cells.

 to the activations of the cells over time can organize them, bringing

cells with similar behavior close together. This makes the diagram

easier to read. We use a few small tricks to achieve the best results.3

Activation value

-4-2024

Step 0

Step 0

Cell 55

Activations at **step 0**

 seems to predict whether the pen is about to be lifted. In the middle,

we see a variety of cells, including some tracking absolute position.

Another way to explore the activations is to give it a sample and

interactively see the activations. Below you can write and see the

activations of all the cells in real time.

Clear

Step 0

Cell 55

Activations at **step 0**

---

Conclusion

----------

The black box reputation of machine learning models is well

deserved, but we believe part of that reputation has been born from the

programming context into which they have been locked. The experience of

having an easily inspectable model available in the same programming

context as the interactive visualization environment (here, javascript)

proved to be very productive for prototyping and exploring new ideas for

 this post.

As we are able to move them more and more into the same programming

 context that user interface work is done, we believe we will see richer

 modes of human-ai interactions flourish. This could have a marked

impact on debugging and building models, for sure, but also in how the

models are used. Machine learning research typically seeks to mimic and

substitute humans, and increasingly it’s able to. What seems less

explored is using machine learning to augment humans. This sort of

complicated human-machine interaction is best explored when the full

capabilities of the model are available in the user interface context.

Experiments in Handwriting with a Neural Network.html


Chris Olah
Nick Cammarata
Ludwig Schubert
Gabriel Goh
Michael Petrov
Shan Carter

Abstract: This article is part of the Circuits thread,  a collection of short articles and commentary by an open scientific  collaboration delving into the inner workings of neural networks.

 a collection of short articles and commentary by an open scientific

collaboration delving into the inner workings of neural networks.

[Curve Detectors](https://distill.pub/2020/circuits/curve-detectors/)

input

softmax

[0](#conv2d0)

[1](#conv2d1)

[2](#conv2d2)

5a

4d

5b

[3b](#mixed3b)

4e

4c

4b

[3a](#mixed3a)

4a

 Over the course of these layers, we see the network go from raw pixels

 Along the way, we see a variety of interesting intermediate features,

 Firstly, it’s particularly easy to study:

 common for vision models to have on the order of 64 channels in their

initial convolutional layers, which are applied at many spatial

positions. So while there are many neurons, the number of unique neurons

 is orders of magnitude smaller. and the features seem

quite simple.

 Secondly, early vision seems most likely to be universal: to have the

same features and circuits form across different architectures and

tasks.

 Before we dive into detailed explorations of different parts of early

vision, we wanted to give a broader overview of how we presently

understand it.

 This article sketches out our understanding, as an annotated

collection of what we call “neuron groups.”

 We also provide illustrations of selected circuits at each layer.

 will not discuss the “bottleneck” neurons in mixed3a/mixed3b, which we

generally think of as low-rank connections to the previous layer.

 But our experience is that a thousand neurons is more than enough to

be disorienting when one begins studying a model.

 Our hope is that this article will help readers avoid this

disorientation by providing some structure and handholds for thinking

about them.

### Playing Cards with Neurons

 Dmitri Mendeleev is often accounted to have discovered the Periodic

Table by playing “chemical solitaire,” writing the details of each

element on a card and patiently fiddling with different ways of

classifying and organizing them.

 Some modern historians are skeptical about the cards, but Mendeleev’s

story is a compelling demonstration of that there can be a lot of value

in simply organizing phenomena, even when you don’t have a theory or

firm justification for that organization yet.

 Mendeleev is far from unique in this.

 For example, in biology, taxonomies of species preceded genetics and

the theory of evolution giving them a theoretical foundation.

 Our experience is that many neurons in vision models seem to fall into

 families of similar features.

 For example, it’s not unusual to see a dozen neurons detecting the

same feature in different orientations or colors.

 Perhaps even more strikingly, the same “neuron families” seem to recur

 across models!

 Of course, it’s well known that Gabor filters and color contrast

detectors reliably comprise neurons in the first layer of convolutional

neural networks, but we were quite surprised to see this generalize to

later layers.

 This article shares our working categorization of units in the first

five layers of InceptionV1 into neuron families.

 These families are ad-hoc, human defined collections of features that

seem to be similar in some way.

 We’ve found these helpful for communicating among ourselves and

breaking the problem of understanding InceptionV1 into smaller chunks.

 While there are some families we suspect are “real”, many others are

categories of convenience, or categories we have low-confidence about.

 The main goal of these families is to help researchers orient

themselves.

 In constructing this categorization, our understanding of individual

neurons was developed by looking at feature visualizations, dataset

examples, how a feature is built from the previous layer, how it is used

 by the next layer, and other analysis.

 It’s worth noting that the level of attention we’ve given to

individual neurons varies greatly: we’ve dedicated entire forthcoming

articles to detailed analysis some of these units, while many others

have only received a few minutes of cursory investigation.

 which correlates neurons with a pre-defined set of features and groups

them into categories like color, texture, and object.

 This has the advantage of removing subjectivity and being much more

scalable.

 At the same time, it also has downsides: correlation can be misleading

 and the pre-defined taxonomy may miss the true feature types.

[Net Dissect](http://netdissect.csail.mit.edu/)

 was very elegant work which advanced our ability to systematically talk

 about network features.

 However, to understand the differences between correlating features

with a pre-defined taxonomy and individually studying them, it may be

illustrative to consider how it classifies some features.

 Net Dissect doesn’t include the canonical InceptionV1, but it does

include a variant of it.

 we see many units which appear from dataset examples likely to be

familiar feature types like curve detectors, divot detectors, boundary

detectors, eye detector, and so forth, but are classified as weakly

correlated with another feature — often objects that it seems unlikely

could be detected at such an early layer.

 Or in another fun case, there is a feature (372) which is most

correlated with a cat detector, but appears to be detecting

left-oriented whiskers!

 frequency detectors — the fact that they are unanticipated makes them

impossible to include in a set of pre-defined features.

 The only way to discover them is the laborious process of manually

investigating each feature.

 In the future, you could imagine hybrid approaches, where a human

investigator is saved time by having many features sorted into a

#### Caveats

* This is a broad overview and our understanding of many of these

units is low-confidence. We fully expect, in retrospect, to realize we

misunderstood some units and categories.

* Many neuron groups are catch-all categories or convenient

organizational categories that we don’t think reflect fundamental

structure.

* Even for neuron groups we suspect do reflect a fundamental

structure (eg. some can be recovered from factorizing the layer’s weight

 matrices) the boundaries of these groups can be blurry and some neurons

 inclusion involve judgement calls.

### Presentation of Neurons

 In order to talk about neurons, we need to somehow represent them.

 We use small amounts of transformation robustness when visualizing

the first few layers, because it has a larger proportional affect on

their small receptive fields, and increase as we move to higher layers.

 For low layers, we use L2 regularization to push pixels towards

gray.

 For the first layer, we follow the convention of other papers and

just show the weights, which for the special case of the first layer are

 equivalent to feature visualization with the right L2 penalty.

 When we represent a neuron with a feature visualization, we don’t

intend to claim that the feature visualization captures the entirety of

the neuron’s behavior.

 Rather, the role of a feature visualization is like a variable name in

 understanding a program.

 It replaces an arbitrary number with a more meaningful symbol .

### Presentation of Circuits

 Although this article is focused on giving an overview of the features

 which exist in early vision, we’re also interested in understanding how

 they’re computed from earlier features.

 consisting of a neuron, the units it has the strongest (L2 norm)

weights to in the previous layer, and the weights between them.

 Some neurons in `mixed3a` and `mixed3b` are in

 branches consisting of a “bottleneck” 1x1 conv that greatly reduces the

 number of channels followed by a 5x5 conv. Although there is a ReLU

between them, we generally think of them as a low rank factorization of a

 single weight matrix and visualize the product of the two weights.

Additionally, some neurons in these layers are in a branch consisting of

 maxpooling followed by a 1x1 conv; we present these units as their

weights replicated over the region of their maxpooling.

 In some cases, we’ve also included a few neurons that have weaker

connections if they seem to have particular pedagogical value; in these

cases, we’ve mentioned doing so in the caption.

 Neurons are visually displayed by their feature visualizations, as

discussed above.

 Weights are represented using a color map with red as positive and

blue as negative.

positive (excitation)

positive (excitation)

negative (inhibition)

negative (inhibition)

Click on the feature visualization of any neuron to see more weights!

 At any point, you can click on a neuron’s feature visualization to see

 its weights to the 50 neurons in the previous layer it is most

connected to (that is, how it assembled from the previous layer, and

also the 50 neurons in the next layer it is most connected to (that is,

how it is used going forward).

 This allows further investigation, and gives you an unbiased view of

the weights if you’re concerned about cherry-picking.

---

`conv2d0`

---------

 The first conv layer of every vision model we’ve looked at is mostly

comprised of two kinds of features: color-contrast detectors and Gabor

filters.

 In contrast to other models, however, the features aren’t perfect

color contrast detectors and Gabor filters.

 For lack of a better word, they’re messy.

 We have no way of knowing, but it seems likely this is a result of the

 gradient not reaching the early layers very well during training.

 Note that InceptionV1 predated the adoption of modern techniques like

batch norm and Adam, which make it much easier to train deep models

well.

 The weights for the units in the first layer of the TF-Slim

 version of InceptionV1, which adds BatchNorm. (Units are sorted by the

first principal component of the adjacency matrix between the first and

second layers.) These features are typical of a well trained conv net.

Note how, unlike the canonical InceptionV1, these units have a crisp

division between black and white Gabors, color Gabors, color-contrast

units and color center-surround units.

 One subtlety that’s worth noting here is that Gabor filters almost

always come in pairs of weights which are negative versions of each

other, both in InceptionV1 and other vision models.

 A single Gabor filter can only detect edges at some offsets, but the

negative version fills in holes, allowing for the formation of complex

Gabor filters in the next layer.

### [**Gabor Filters** 44%](#group_conv2d0_gabor_filters)

Show all 28 neurons.

Collapse neurons.

Gabor filters are a simple edge

detector, highly sensitive to the alignment of the edge. They’re almost

universally found in the fist layer of vision models. Note that Gabor

filters almost always come in pairs of negative reciprocals.

### [**Color Contrast** 42%](#group_conv2d0_color_contrast)

Show all 27 neurons.

Collapse neurons.

These units detect a color one side of

their receptive field, and the opposite color on the other side. Compare

### [**Other Units** 14%](#group_conv2d0_other_units)

Units that don’t fit in another category.

---

`conv2d1`

---------

**Complex Gabors:**

 A nice example of this is the “Complex Gabor” feature family.

 Like simple Gabor filters, complex Gabors detect edges.

 But unlike simple Gabors, they are relatively invariant to the exact

position of the edge or which side is dark or light.

 This is achieved by being excited by multiple Gabor filters in similar

 orientations — and most critically, by being excited by “reciprocal

Gabor filters” that detect the same pattern with dark and light

switched.

 This can be seen as an early example of the “union over cases” motif.

 both positive (excitation) and negative (inhibition).

 Click on a neuron to see its forwards and backwards weights.

 Note that `conv2d1` is a 1x1 convolution, so there’s only a

 single weight — a single line, in this diagram — between each channel

in the previous and this one.

 There is a pooling layer between them, so the features it connects

to are pooled versions of the previous layer rather than original

features.

 This plays an important role in determining the features we observe:

in models with larger convolutions in their second layer, we often see a

 jump to crude versions of the larger more complex features we’ll see in

 the following layers.

 In addition to Complex Gabors, we see a variety of other features,

including

 more invariant color contrast detectors, Gabor-like features that are

less selective for a single orientation, and lower-frequency features.

### [**Low Frequency** 27%](#group_conv2d1_low_frequency)

Show all 17 neurons.

Collapse neurons.

### [**Gabor Like** 17%](#group_conv2d1_gabor_like)

Show all 11 neurons.

Collapse neurons.

These units respond to edges stimuli,

but seem to respond to a wider range of orientations, and also respond

to color contrasts that align with the edge. We haven’t studied them

very carefully.

### [**Color Contrast** 16%](#group_conv2d1_color_contrast)

These units detect a color on one side

of the receptive field, and a different color on the opposite side.

Composed of lower-level color contrast detectors, they often respond to

color transitions in a range of translation and orientation variations.

### [**Multicolor** 14%](#group_conv2d1_multicolor)

### [**Complex Gabor** 14%](#group_conv2d1_complex_gabor)

Like Gabor Filters, but fairly invariant

 to the exact position, formed by adding together multiple Gabor

detectors in the same orientation but different phases. We call these

### [**Color** 6%](#group_conv2d1_color)

Two of these units seem to track

brightness (bright vs dark), while the other two units seem to mostly

track hue, dividing the space of hues between them. One responds to

red/orange/yellow, while the other responds to purple/blue/turqoise.

Unfortunately, their circuits seem to heavily rely on the existence of a

### [**Other Units** 5%](#group_conv2d1_other_units)

Units that don’t fit in another category.

### [**hatch** 2%](#group_conv2d1_hatch)

---

`conv2d2`

---------

In `conv2d2` we see the emergence of very simple shape

predecessors.

This layer sees the first units that might be described as “line

detectors”, preferring a single longer line to a Gabor pattern and

accounting for about 25% of units.

We also see tiny curve detectors, corner detectors, divergence

detectors, and a single very tiny circle detector.

One fun aspect of these features is that you can see that they are

assembled from Gabor detectors in the feature visualizations, with

curves being built from small piecewise Gabor segments.

All of these units still moderately fire in response to incomplete

versions of their feature, such as a small curve running tangent to the

edge detector.

Since `conv2d2` is a 3x3 convolution, our understanding of

these shape precursor features (and some texture features) maps to

particular ways Gabor and lower-frequency edges are being spatially

assembled into new features.

At a high-level, we see a few primary patterns:

Line

Curve

Shifted Line

Gabor Texture

Corner / Lisp

Hatch Texture

Divergence

 We also begin to see various kinds of texture and color detectors

start to become a major constituent of the layer, including

color-contrast and color center surround features, as well as

Gabor-like, hatch, low-frequency and high-frequency textures.

 A handful of units look for different textures on different sides of

their receptive field.

### [**Color Contrast** 21%](#group_conv2d2_color_contrast)

Show all 40 neurons.

Collapse neurons.

These units detect a color on one side

of the receptive field, and a different color on the opposite side.

Composed of lower-level color contrast detectors, they often respond to

color transitions in a range of translation and orientation variations.

### [**Line** 17%](#group_conv2d2_line)

Show all 33 neurons.

Collapse neurons.

These units are beginning to look for a

single primary line. Some look for different colors on each side. Many

exhibit “combing” (small perpendicular lines along the main one), a very

 common but not presently understood phenomenon in line-like features

### [**Shifted Line** 8%](#group_conv2d2_shifted_line)

Show all 16 neurons.

Collapse neurons.

These units look for edges “shifted” to

the side of the receptive field instead of the middle. This may be

### [**Textures** 8%](#group_conv2d2_textures)

Show all 15 neurons.

Collapse neurons.

A broad category of units detecting repeating local structure.

### [**Other Units** 7%](#group_conv2d2_other_units)

Show all 14 neurons.

Collapse neurons.

Catch-all category for all other units.

### [**Color Center-Surround** 7%](#group_conv2d2_color_center_surround)

Show all 13 neurons.

Collapse neurons.

These units look for one color in the

middle and another (typically opposite) on the boundary. Genereally more

### [**Tiny Curves** 6%](#group_conv2d2_tiny_curves)

Show all 12 neurons.

Collapse neurons.

Very small curve (and one circle)

detectors. Many of these units respond to a range of curvatures all the

### [**Early Brightness Gradient** 6%](#group_conv2d2_early_brightness_gradient)

Show all 12 neurons.

Collapse neurons.

These units detect oriented gradients in

 brightness. They support a variety of similar units in the next layer.

### [**Gabor Textures** 6%](#group_conv2d2_gabor_textures)

Show all 12 neurons.

Collapse neurons.

### [**Texture Contrast** 4%](#group_conv2d2_texture_contrast)

### [**Hatch Textures** 3%](#group_conv2d2_hatch_textures)

### [**Color/Multicolor** 3%](#group_conv2d2_color_multicolor)

### [**Corners** 2%](#group_conv2d2_corners)

### [**Line Divergence** 1%](#group_conv2d2_line_divergence)

These units detect lines diverging from a point.

---

`mixed3a`

---------

`mixed3a` has a significant increase in the diversity of features we observe.

and will be discussed again in later articles in great detail.

Prior to `mixed3a`, color contrast detectors look for

transitions of a color to near complementary colors (eg. blue vs

yellow).

From this layer on, however, we’ll often see color detectors which

compare a color to the absence of color.

Additionally, black and white detectors can allow the detection of

greyscale images, which may be correlated with ImageNet categories (see

The circuit for our black and white detector is quite simple:

almost all of its large weights are negative, detecting the absence of colors.

inhibiting

positive (excitation)

negative (inhibition)

 The sixteen strongest magnitude weights to the previous layer are

shown.

 For simplicity, only one spatial weight for positive and negative

have been shown, but they all have almost identical structure.

 Click on a neuron to see its forwards and backwards weights.

But there’s lots of other interesting examples.

positive (excitation)

positive (excitation)

negative (inhibition)

negative (inhibition)

positive (excitation)

negative (inhibition)

 The circuit constructing a triangle detector.

 The choice of which neurons in the previous layer to show is slightly

cherrypicked for pedagogy.

 The six neurons with the highest magnitude weights to the triangle are

 shown, plus one other neuron with slightly weaker weights.

 (Left leaning edges have slightly higher weights than right ones, but

it seemed more illustrative to show two of both.) Click on neurons to

see the full weights.

 However, in practice, these triangle detectors (and other angle units)

 seem to often just be used as multi-edge detectors downstream,

 or in conjunction with many other units to detect convex boundaries.

 Below, we provide a taxonomized overview of all of them:

### [**Texture** 25%](#group_mixed3a_texture)

Show all 65 neurons.

Collapse neurons.

This is a broad, not very well defined

category for units that seem to look for simple local structures over a

wide receptive field, including mixtures of colors. Many live in a

branch consisting of a maxpool followed by a 1x1 conv, which

structurally encourages this.Maxpool

branches (ie. maxpool 5x5 stride 1 -> conv 1x1) have large receptive

fields, but can’t control where in in their receptive field each feature

 they detect is, nor the relative position of these features. In early

vision, this unstructured of feature detection makes them a good fit for

 textures.

### [**Color Center-Surround** 12%](#group_mixed3a_color_center_surround)

Show all 30 neurons.

Collapse neurons.

These units look for one color in the

center, and another (usually opposite) color surrounding it. They are

typically much more sensitive to the center color than the surrounding

one. In visual neuroscience, center-surround units are classically an

extremely low-level feature, but we see them in the later parts of early

### [**High-Low Frequency** 6%](#group_mixed3a_high_low_frequency)

Show all 15 neurons.

Collapse neurons.

 A detailed article on these is forthcoming.

### [**Brightness Gradient** 6%](#group_mixed3a_brightness_gradient)

Show all 15 neurons.

Collapse neurons.

These units detect brightness gradients.

 Among other things they will help detect specularity (shininess),

### [**Color Contrast** 5%](#group_mixed3a_color_contrast)

Show all 14 neurons.

Collapse neurons.

These units look for one color on one

side of their receptive field, and another (usually opposite) color on

the opposing side. They typically don’t care about the exact position or

### [**Complex Center-Surround** 5%](#group_mixed3a_complex_center_surround)

Show all 14 neurons.

Collapse neurons.

This is a broad, not very well defined

category for center-surround units that detect a pattern or complex

texture in their center.

### [**Line Misc.** 5%](#group_mixed3a_line_misc.)

Show all 14 neurons.

Collapse neurons.

Broad, low confidence organizational category.

### [**Lines** 5%](#group_mixed3a_lines)

Show all 14 neurons.

Collapse neurons.

Units used to detect extended lines,

often further excited by different colors on each side. A few are highly

 combed line detectors that aren’t obviously such at first glance. The

decision to include a unit was often decided by whether it seems to be

used by downstream client units as a line detector.

### [**Other Units** 5%](#group_mixed3a_other_units)

Show all 14 neurons.

Collapse neurons.

Catch-all category for all other units.

### [**Repeating patterns** 5%](#group_mixed3a_repeating_patterns)

Show all 12 neurons.

Collapse neurons.

### [**Curves** 4%](#group_mixed3a_curves)

Show all 11 neurons.

Collapse neurons.

These curve detectors detect

significantly larger radii curves than their predecessors. They will be

refined into more specific, larger curve detectors in the next layer.

### [**BW vs Color** 4%](#group_mixed3a_bw_vs_color)

These “black and white” detectors

respond to absences of color. Prior to this, color detectors contrast to

 the opposite hue, but from this point on we’ll see many compare to the

### [**Angles** 3%](#group_mixed3a_angles)

Units that detect multiple lines,

forming angles, triangles and squares. They generally respond to any of

the individual lines, and more strongly to them together.

### [**Fur Precursors** 3%](#group_mixed3a_fur_precursors)

These units are not yet highly selective

 for fur (they also fire for other high-frequency patterns), but their

 At the 224x224 image resolution, individual fur hairs are generally not

 detectable, but tufts of fur are. These units use Gabor textures to

detect those tufts in different orientations. The also detect lower

frequency edges or changes in lighting perpendicular to the tufts.

### [**Eyes / Small Circles** 2%](#group_mixed3a_eyes_small_circles)

### [**Crosses / Diverging Lines** 2%](#group_mixed3a_crosses_diverging_lines)

### [**Thick Lines** 1%](#group_mixed3a_thick_lines)

Low confidence organizational category.

### [**Line Ends** 1%](#group_mixed3a_line_ends)

---

`mixed3b`

---------

`mixed3b` straddles two levels of abstraction.

On the one hand, it has some quite sophisticated features that don’t

really seem like they should be characterized as “early” or “low-level”:

 object boundary detectors, early head detectors, and more sophisticated

 part of shape detectors.

On the other hand, it also has many units that still feel quite

low-level, such as color center-surround units.

 When you first look at the feature visualizations and dataset examples,

 you might think these are just another iteration of edge or curve detectors.

positive (excitation)

negative (inhibition)

High-low frequency detectors

Edges

End of Line

Color Contrasts

 We sometimes find it useful to think about the “goal” of early vision.

**Curve-based Features:**

 These include more sophisticated curves,

 [divots](#group_mixed3b_divots), and [“evolutes”](#group_mixed3b_evolute)

Curve

Circle

Spiral

Evolute

positive (excitation)

negative (inhibition)

 Again, these circuits only scratch the surface of `mixed3b`.

 Since it’s a larger layer with lots of families, we’ll go through a

couple particularly interesting and well understood families first:

### [**Boundary** 8%](#group_mixed3b_boundary)

Show all 36 neurons.

Collapse neurons.

These units use multiple cues to detect

the boundaries of objects. They vary in orientation, detecting

convex/concave/straight boundaries, and detecting artificial vs fur

foregrounds. Cues they rely on include line detectors, high-low

frequency detectors, and color contrast.

### [**Proto-Head** 3%](#group_mixed3b_proto_head)

Show all 12 neurons.

Collapse neurons.

The tiny eye detectors, along with

texture detectors for fur, hair and skin developed at the previous layer

 enable these early head detectors, which will continue to be refined in

 the next layer.

### [**Generic, Oriented Fur** 2%](#group_mixed3b_generic_oriented_fur)

We don’t typically think of fur as an

oriented feature, but it is. These units detect fur parting in various

ways, much like how hair on your head parts.

### [**Curves** 2%](#group_mixed3b_curves)

The third iteration of curve detectors.

They detect larger radii curves than their predecessors, and are the

first to not slightly fire for curves rotated 180 degrees. Compare to

### [**Divots** 2%](#group_mixed3b_divots)

Curve-like detectors for sharp corners or bumps.

### [**Square / Grid** 2%](#group_mixed3b_square_grid)

Units detecting grid patterns.

### [**Brightness Gradients** 1%](#group_mixed3b_brightness_gradients)

### [**Eyes** 1%](#group_mixed3b_eyes)

### [**Shallow Curves** 1%](#group_mixed3b_shallow_curves)

### [**Curve Shapes** 1%](#group_mixed3b_curve_shapes)

Simple shapes created by composing curves, such as spirals and S-curves.

### [**Circles / Loops** 1%](#group_mixed3b_circles_loops)

### [**Circle Cluster** 1%](#group_mixed3b_circle_cluster)

### [**Double Curves** 1%](#group_mixed3b_double_curves)

### [**Evolute** 0.2%](#group_mixed3b_evolute)

 One frustrating issue is that `mixed3b` has many units that

 don’t have a simple low-level articulation, but also are not yet very

specific to a high-level feature.

 For example, there are units which seem to be developing towards

detecting certain animal body parts, but still respond to many other

stimuli as well and so are difficult to describe.

### [**Color Center-Surround** 16%](#group_mixed3b_color_center_surround)

Show all 77 neurons.

Collapse neurons.

These units look for one color in the

center, and another color surrounding it. These units likely have many

subtleties about the range of hues, texture preferences, and

interactions that similar neurons in earlier layers may not have. Note

how many units detect the absence (or generic presence) of color,

### [**Complex Center-Surround** 15%](#group_mixed3b_complex_center_surround)

Show all 73 neurons.

Collapse neurons.

This is a broad, not very well defined

category for center-surround units that detect a pattern or complex

texture in their center.

### [**Texture** 9%](#group_mixed3b_texture)

Show all 44 neurons.

Collapse neurons.

This is a broad, not very well defined

category for units that seem to look for simple local structures over a

wide receptive field, including mixtures of colors.

### [**Other Units** 9%](#group_mixed3b_other_units)

Show all 42 neurons.

Collapse neurons.

Units that don’t fall in any other category.

### [**Color Contrast/Gradient** 5%](#group_mixed3b_color_contrast_gradient)

Show all 24 neurons.

Collapse neurons.

Units which respond to different colors

on each side. These units look for one color in the center, and another

color surrounding it. These units likely have many subtleties about the

range of hues, texture preferences, and interactions that similar

neurons in earlier layers may not have. Compare to earlier color

### [**Texture Contrast** 3%](#group_mixed3b_texture_contrast)

Show all 12 neurons.

Collapse neurons.

Units that detect one texture on one side and a different texture on the other.

### [**Other Fur** 2%](#group_mixed3b_other_fur)

Units which seem to detect fur but,

unlike the oriented fur detectors, don’t seem to detect it parting in a

particular way. Many of these seem to prefer a particular fur pattern.

### [**Lines** 2%](#group_mixed3b_lines)

### [**Cross / Corner Divergence** 2%](#group_mixed3b_cross_corner_divergence)

### [**Pattern** 2%](#group_mixed3b_pattern)

Low confidence category.

### [**Bumps** 2%](#group_mixed3b_bumps)

Low confidence category.

### [**Double Boundary** 1%](#group_mixed3b_double_boundary)

### [**Bar / Line-Like** 1%](#group_mixed3b_bar_line_like)

Low confidence category.

### [**Boundary Misc** 1%](#group_mixed3b_boundary_misc)

Boundary-related units we didn’t know what else to do with.

### [**Star** 1%](#group_mixed3b_star)

Low confidence category.

### [**Line Grad** 1%](#group_mixed3b_line_grad)

Low confidence category.

### [**Scales** 1%](#group_mixed3b_scales)

We don’t really understand these units.

### [**Curves misc.** 1%](#group_mixed3b_curves_misc.)

Low confidence organizational category.

### [**Shiny** 0.4%](#group_mixed3b_shiny)

Units that seem to detect shiny, specular surfaces.

### [**Pointy** 0.4%](#group_mixed3b_pointy)

Low confidence category.

---

Conclusion

----------

 The goal of this essay was to give a high-level overview of our

present understanding of early vision in InceptionV1.

 Every single feature discussed in this article is a potential topic of

 deep investigation.

 For example, are curve detectors really curve detectors? What types of

 curves do they fire for? How do they behave on various edge cases? How

are they built?

 Over the coming articles, we’ll do deep dives rigorously investigating

 these questions for a few features, starting with curves.

 Is there a better taxonomy, or another way to understand the space of features?

 Why do features often seem to form in families?

 To what extent do the same features families form across models?

 Is there a “periodic table of low-level visual features”, in some sense?

 To what extent do later features admit a similar taxonomy?

 We think these could be interesting questions for future work.

 a collection of short articles and commentary by an open scientific

collaboration delving into the inner workings of neural networks.

[Curve Detectors](https://distill.pub/2020/circuits/curve-detectors/)

Going deeper with convolutions

Network dissection: Quantifying interpretability of deep visual representations

Visualizing higher-layer features of a deep network

Feature Visualization

Inceptionism: Going deeper into neural networks

The Building Blocks of Interpretability

TensorFlow-Slim: A lightweight library for defining, training and evaluating complex models in TensorFlow

An Overview of Early Vision in InceptionV1.html