An Overview of Early Vision in InceptionV1

Chris Olah
Nick Cammarata
Ludwig Schubert
Gabriel Goh
Michael Petrov
Shan Carter

Abstract: This article is part of the Circuits thread,  a collection of short articles and commentary by an open scientific  collaboration delving into the inner workings of neural networks.

 a collection of short articles and commentary by an open scientific

collaboration delving into the inner workings of neural networks.

[Curve Detectors](https://distill.pub/2020/circuits/curve-detectors/)

input

softmax

[0](#conv2d0)

[1](#conv2d1)

[2](#conv2d2)

5a

4d

5b

[3b](#mixed3b)

4e

4c

4b

[3a](#mixed3a)

4a

 Over the course of these layers, we see the network go from raw pixels

 Along the way, we see a variety of interesting intermediate features,

 Firstly, it's particularly easy to study:

 common for vision models to have on the order of 64 channels in their

initial convolutional layers, which are applied at many spatial

positions. So while there are many neurons, the number of unique neurons

 is orders of magnitude smaller. and the features seem

quite simple.

 Secondly, early vision seems most likely to be universal: to have the

same features and circuits form across different architectures and

tasks.

 Before we dive into detailed explorations of different parts of early

vision, we wanted to give a broader overview of how we presently

understand it.

 This article sketches out our understanding, as an annotated

collection of what we call "neuron groups."

 We also provide illustrations of selected circuits at each layer.

 will not discuss the "bottleneck" neurons in mixed3a/mixed3b, which we

generally think of as low-rank connections to the previous layer.

 But our experience is that a thousand neurons is more than enough to

be disorienting when one begins studying a model.

 Our hope is that this article will help readers avoid this

disorientation by providing some structure and handholds for thinking

about them.

### Playing Cards with Neurons

 Dmitri Mendeleev is often accounted to have discovered the Periodic

Table by playing "chemical solitaire," writing the details of each

element on a card and patiently fiddling with different ways of

classifying and organizing them.

 Some modern historians are skeptical about the cards, but Mendeleev's

story is a compelling demonstration of that there can be a lot of value

in simply organizing phenomena, even when you don't have a theory or

firm justification for that organization yet.

 Mendeleev is far from unique in this.

 For example, in biology, taxonomies of species preceded genetics and

the theory of evolution giving them a theoretical foundation.

 Our experience is that many neurons in vision models seem to fall into

 families of similar features.

 For example, it's not unusual to see a dozen neurons detecting the

same feature in different orientations or colors.

 Perhaps even more strikingly, the same "neuron families" seem to recur

 across models!

 Of course, it's well known that Gabor filters and color contrast

detectors reliably comprise neurons in the first layer of convolutional

neural networks, but we were quite surprised to see this generalize to

later layers.

 This article shares our working categorization of units in the first

five layers of InceptionV1 into neuron families.

 These families are ad-hoc, human defined collections of features that

seem to be similar in some way.

 We've found these helpful for communicating among ourselves and

breaking the problem of understanding InceptionV1 into smaller chunks.

 While there are some families we suspect are "real", many others are

categories of convenience, or categories we have low-confidence about.

 The main goal of these families is to help researchers orient

themselves.

 In constructing this categorization, our understanding of individual

neurons was developed by looking at feature visualizations, dataset

examples, how a feature is built from the previous layer, how it is used

 by the next layer, and other analysis.

 It's worth noting that the level of attention we've given to

individual neurons varies greatly: we've dedicated entire forthcoming

articles to detailed analysis some of these units, while many others

have only received a few minutes of cursory investigation.

 which correlates neurons with a pre-defined set of features and groups

them into categories like color, texture, and object.

 This has the advantage of removing subjectivity and being much more

scalable.

 At the same time, it also has downsides: correlation can be misleading

 and the pre-defined taxonomy may miss the true feature types.

[Net Dissect](http://netdissect.csail.mit.edu/)

 was very elegant work which advanced our ability to systematically talk

 about network features.

 However, to understand the differences between correlating features

with a pre-defined taxonomy and individually studying them, it may be

illustrative to consider how it classifies some features.

 Net Dissect doesn't include the canonical InceptionV1, but it does

include a variant of it.

 we see many units which appear from dataset examples likely to be

familiar feature types like curve detectors, divot detectors, boundary

detectors, eye detector, and so forth, but are classified as weakly

correlated with another feature — often objects that it seems unlikely

could be detected at such an early layer.

 Or in another fun case, there is a feature (372) which is most

correlated with a cat detector, but appears to be detecting

left-oriented whiskers!

 frequency detectors — the fact that they are unanticipated makes them

impossible to include in a set of pre-defined features.

 The only way to discover them is the laborious process of manually

investigating each feature.

 In the future, you could imagine hybrid approaches, where a human

investigator is saved time by having many features sorted into a

#### Caveats

* This is a broad overview and our understanding of many of these

units is low-confidence. We fully expect, in retrospect, to realize we

misunderstood some units and categories.

* Many neuron groups are catch-all categories or convenient

organizational categories that we don't think reflect fundamental

structure.

* Even for neuron groups we suspect do reflect a fundamental

structure (eg. some can be recovered from factorizing the layer's weight

 matrices) the boundaries of these groups can be blurry and some neurons

 inclusion involve judgement calls.

### Presentation of Neurons

 In order to talk about neurons, we need to somehow represent them.

 We use small amounts of transformation robustness when visualizing

the first few layers, because it has a larger proportional affect on

their small receptive fields, and increase as we move to higher layers.

 For low layers, we use L2 regularization to push pixels towards

gray.

 For the first layer, we follow the convention of other papers and

just show the weights, which for the special case of the first layer are

 equivalent to feature visualization with the right L2 penalty.

 When we represent a neuron with a feature visualization, we don't

intend to claim that the feature visualization captures the entirety of

the neuron's behavior.

 Rather, the role of a feature visualization is like a variable name in

 understanding a program.

 It replaces an arbitrary number with a more meaningful symbol .

### Presentation of Circuits

 Although this article is focused on giving an overview of the features

 which exist in early vision, we're also interested in understanding how

 they're computed from earlier features.

 consisting of a neuron, the units it has the strongest (L2 norm)

weights to in the previous layer, and the weights between them.

 Some neurons in `mixed3a` and `mixed3b` are in

 branches consisting of a "bottleneck" 1x1 conv that greatly reduces the

 number of channels followed by a 5x5 conv. Although there is a ReLU

between them, we generally think of them as a low rank factorization of a

 single weight matrix and visualize the product of the two weights.

Additionally, some neurons in these layers are in a branch consisting of

 maxpooling followed by a 1x1 conv; we present these units as their

weights replicated over the region of their maxpooling.

 In some cases, we've also included a few neurons that have weaker

connections if they seem to have particular pedagogical value; in these

cases, we've mentioned doing so in the caption.

 Neurons are visually displayed by their feature visualizations, as

discussed above.

 Weights are represented using a color map with red as positive and

blue as negative.

positive (excitation)

positive (excitation)

negative (inhibition)

negative (inhibition)

Click on the feature visualization of any neuron to see more weights!

 At any point, you can click on a neuron's feature visualization to see

 its weights to the 50 neurons in the previous layer it is most

connected to (that is, how it assembled from the previous layer, and

also the 50 neurons in the next layer it is most connected to (that is,

how it is used going forward).

 This allows further investigation, and gives you an unbiased view of

the weights if you're concerned about cherry-picking.

---

`conv2d0`

---------

 The first conv layer of every vision model we've looked at is mostly

comprised of two kinds of features: color-contrast detectors and Gabor

filters.

 In contrast to other models, however, the features aren't perfect

color contrast detectors and Gabor filters.

 For lack of a better word, they're messy.

 We have no way of knowing, but it seems likely this is a result of the

 gradient not reaching the early layers very well during training.

 Note that InceptionV1 predated the adoption of modern techniques like

batch norm and Adam, which make it much easier to train deep models

well.

 The weights for the units in the first layer of the TF-Slim

 version of InceptionV1, which adds BatchNorm. (Units are sorted by the

first principal component of the adjacency matrix between the first and

second layers.) These features are typical of a well trained conv net.

Note how, unlike the canonical InceptionV1, these units have a crisp

division between black and white Gabors, color Gabors, color-contrast

units and color center-surround units.

 One subtlety that's worth noting here is that Gabor filters almost

always come in pairs of weights which are negative versions of each

other, both in InceptionV1 and other vision models.

 A single Gabor filter can only detect edges at some offsets, but the

negative version fills in holes, allowing for the formation of complex

Gabor filters in the next layer.

### [**Gabor Filters** 44%](#group_conv2d0_gabor_filters)

Show all 28 neurons.

Collapse neurons.

Gabor filters are a simple edge

detector, highly sensitive to the alignment of the edge. They're almost

universally found in the fist layer of vision models. Note that Gabor

filters almost always come in pairs of negative reciprocals.

### [**Color Contrast** 42%](#group_conv2d0_color_contrast)

Show all 27 neurons.

Collapse neurons.

These units detect a color one side of

their receptive field, and the opposite color on the other side. Compare

### [**Other Units** 14%](#group_conv2d0_other_units)

Units that don't fit in another category.

---

`conv2d1`

---------

**Complex Gabors:**

 A nice example of this is the "Complex Gabor" feature family.

 Like simple Gabor filters, complex Gabors detect edges.

 But unlike simple Gabors, they are relatively invariant to the exact

position of the edge or which side is dark or light.

 This is achieved by being excited by multiple Gabor filters in similar

 orientations — and most critically, by being excited by "reciprocal

Gabor filters" that detect the same pattern with dark and light

switched.

 This can be seen as an early example of the "union over cases" motif.

 both positive (excitation) and negative (inhibition).

 Click on a neuron to see its forwards and backwards weights.

 Note that `conv2d1` is a 1x1 convolution, so there's only a

 single weight — a single line, in this diagram — between each channel

in the previous and this one.

 There is a pooling layer between them, so the features it connects

to are pooled versions of the previous layer rather than original

features.

 This plays an important role in determining the features we observe:

in models with larger convolutions in their second layer, we often see a

 jump to crude versions of the larger more complex features we'll see in

 the following layers.

 In addition to Complex Gabors, we see a variety of other features,

including

 more invariant color contrast detectors, Gabor-like features that are

less selective for a single orientation, and lower-frequency features.

### [**Low Frequency** 27%](#group_conv2d1_low_frequency)

Show all 17 neurons.

Collapse neurons.

### [**Gabor Like** 17%](#group_conv2d1_gabor_like)

Show all 11 neurons.

Collapse neurons.

These units respond to edges stimuli,

but seem to respond to a wider range of orientations, and also respond

to color contrasts that align with the edge. We haven't studied them

very carefully.

### [**Color Contrast** 16%](#group_conv2d1_color_contrast)

These units detect a color on one side

of the receptive field, and a different color on the opposite side.

Composed of lower-level color contrast detectors, they often respond to

color transitions in a range of translation and orientation variations.

### [**Multicolor** 14%](#group_conv2d1_multicolor)

### [**Complex Gabor** 14%](#group_conv2d1_complex_gabor)

Like Gabor Filters, but fairly invariant

 to the exact position, formed by adding together multiple Gabor

detectors in the same orientation but different phases. We call these

### [**Color** 6%](#group_conv2d1_color)

Two of these units seem to track

brightness (bright vs dark), while the other two units seem to mostly

track hue, dividing the space of hues between them. One responds to

red/orange/yellow, while the other responds to purple/blue/turqoise.

Unfortunately, their circuits seem to heavily rely on the existence of a

### [**Other Units** 5%](#group_conv2d1_other_units)

Units that don't fit in another category.

### [**hatch** 2%](#group_conv2d1_hatch)

---

`conv2d2`

---------

In `conv2d2` we see the emergence of very simple shape

predecessors.

This layer sees the first units that might be described as "line

detectors", preferring a single longer line to a Gabor pattern and

accounting for about 25% of units.

We also see tiny curve detectors, corner detectors, divergence

detectors, and a single very tiny circle detector.

One fun aspect of these features is that you can see that they are

assembled from Gabor detectors in the feature visualizations, with

curves being built from small piecewise Gabor segments.

All of these units still moderately fire in response to incomplete

versions of their feature, such as a small curve running tangent to the

edge detector.

Since `conv2d2` is a 3x3 convolution, our understanding of

these shape precursor features (and some texture features) maps to

particular ways Gabor and lower-frequency edges are being spatially

assembled into new features.

At a high-level, we see a few primary patterns:

Line

Curve

Shifted Line

Gabor Texture

Corner / Lisp

Hatch Texture

Divergence

 We also begin to see various kinds of texture and color detectors

start to become a major constituent of the layer, including

color-contrast and color center surround features, as well as

Gabor-like, hatch, low-frequency and high-frequency textures.

 A handful of units look for different textures on different sides of

their receptive field.

### [**Color Contrast** 21%](#group_conv2d2_color_contrast)

Show all 40 neurons.

Collapse neurons.

These units detect a color on one side

of the receptive field, and a different color on the opposite side.

Composed of lower-level color contrast detectors, they often respond to

color transitions in a range of translation and orientation variations.

### [**Line** 17%](#group_conv2d2_line)

Show all 33 neurons.

Collapse neurons.

These units are beginning to look for a

single primary line. Some look for different colors on each side. Many

exhibit "combing" (small perpendicular lines along the main one), a very

 common but not presently understood phenomenon in line-like features

### [**Shifted Line** 8%](#group_conv2d2_shifted_line)

Show all 16 neurons.

Collapse neurons.

These units look for edges "shifted" to

the side of the receptive field instead of the middle. This may be

### [**Textures** 8%](#group_conv2d2_textures)

Show all 15 neurons.

Collapse neurons.

A broad category of units detecting repeating local structure.

### [**Other Units** 7%](#group_conv2d2_other_units)

Show all 14 neurons.

Collapse neurons.

Catch-all category for all other units.

### [**Color Center-Surround** 7%](#group_conv2d2_color_center_surround)

Show all 13 neurons.

Collapse neurons.

These units look for one color in the

middle and another (typically opposite) on the boundary. Genereally more

### [**Tiny Curves** 6%](#group_conv2d2_tiny_curves)

Show all 12 neurons.

Collapse neurons.

Very small curve (and one circle)

detectors. Many of these units respond to a range of curvatures all the

### [**Early Brightness Gradient** 6%](#group_conv2d2_early_brightness_gradient)

Show all 12 neurons.

Collapse neurons.

These units detect oriented gradients in

 brightness. They support a variety of similar units in the next layer.

### [**Gabor Textures** 6%](#group_conv2d2_gabor_textures)

Show all 12 neurons.

Collapse neurons.

### [**Texture Contrast** 4%](#group_conv2d2_texture_contrast)

### [**Hatch Textures** 3%](#group_conv2d2_hatch_textures)

### [**Color/Multicolor** 3%](#group_conv2d2_color_multicolor)

### [**Corners** 2%](#group_conv2d2_corners)

### [**Line Divergence** 1%](#group_conv2d2_line_divergence)

These units detect lines diverging from a point.

---

`mixed3a`

---------

`mixed3a` has a significant increase in the diversity of features we observe.

and will be discussed again in later articles in great detail.

Prior to `mixed3a`, color contrast detectors look for

transitions of a color to near complementary colors (eg. blue vs

yellow).

From this layer on, however, we'll often see color detectors which

compare a color to the absence of color.

Additionally, black and white detectors can allow the detection of

greyscale images, which may be correlated with ImageNet categories (see

The circuit for our black and white detector is quite simple:

almost all of its large weights are negative, detecting the absence of colors.

inhibiting

positive (excitation)

negative (inhibition)

 The sixteen strongest magnitude weights to the previous layer are

shown.

 For simplicity, only one spatial weight for positive and negative

have been shown, but they all have almost identical structure.

 Click on a neuron to see its forwards and backwards weights.

But there's lots of other interesting examples.

positive (excitation)

positive (excitation)

negative (inhibition)

negative (inhibition)

positive (excitation)

negative (inhibition)

 The circuit constructing a triangle detector.

 The choice of which neurons in the previous layer to show is slightly

cherrypicked for pedagogy.

 The six neurons with the highest magnitude weights to the triangle are

 shown, plus one other neuron with slightly weaker weights.

 (Left leaning edges have slightly higher weights than right ones, but

it seemed more illustrative to show two of both.) Click on neurons to

see the full weights.

 However, in practice, these triangle detectors (and other angle units)

 seem to often just be used as multi-edge detectors downstream,

 or in conjunction with many other units to detect convex boundaries.

 Below, we provide a taxonomized overview of all of them:

### [**Texture** 25%](#group_mixed3a_texture)

Show all 65 neurons.

Collapse neurons.

This is a broad, not very well defined

category for units that seem to look for simple local structures over a

wide receptive field, including mixtures of colors. Many live in a

branch consisting of a maxpool followed by a 1x1 conv, which

structurally encourages this.Maxpool

branches (ie. maxpool 5x5 stride 1 -> conv 1x1) have large receptive

fields, but can't control where in in their receptive field each feature

 they detect is, nor the relative position of these features. In early

vision, this unstructured of feature detection makes them a good fit for

 textures.

### [**Color Center-Surround** 12%](#group_mixed3a_color_center_surround)

Show all 30 neurons.

Collapse neurons.

These units look for one color in the

center, and another (usually opposite) color surrounding it. They are

typically much more sensitive to the center color than the surrounding

one. In visual neuroscience, center-surround units are classically an

extremely low-level feature, but we see them in the later parts of early

### [**High-Low Frequency** 6%](#group_mixed3a_high_low_frequency)

Show all 15 neurons.

Collapse neurons.

 A detailed article on these is forthcoming.

### [**Brightness Gradient** 6%](#group_mixed3a_brightness_gradient)

Show all 15 neurons.

Collapse neurons.

These units detect brightness gradients.

 Among other things they will help detect specularity (shininess),

### [**Color Contrast** 5%](#group_mixed3a_color_contrast)

Show all 14 neurons.

Collapse neurons.

These units look for one color on one

side of their receptive field, and another (usually opposite) color on

the opposing side. They typically don't care about the exact position or

### [**Complex Center-Surround** 5%](#group_mixed3a_complex_center_surround)

Show all 14 neurons.

Collapse neurons.

This is a broad, not very well defined

category for center-surround units that detect a pattern or complex

texture in their center.

### [**Line Misc.** 5%](#group_mixed3a_line_misc.)

Show all 14 neurons.

Collapse neurons.

Broad, low confidence organizational category.

### [**Lines** 5%](#group_mixed3a_lines)

Show all 14 neurons.

Collapse neurons.

Units used to detect extended lines,

often further excited by different colors on each side. A few are highly

 combed line detectors that aren't obviously such at first glance. The

decision to include a unit was often decided by whether it seems to be

used by downstream client units as a line detector.

### [**Other Units** 5%](#group_mixed3a_other_units)

Show all 14 neurons.

Collapse neurons.

Catch-all category for all other units.

### [**Repeating patterns** 5%](#group_mixed3a_repeating_patterns)

Show all 12 neurons.

Collapse neurons.

### [**Curves** 4%](#group_mixed3a_curves)

Show all 11 neurons.

Collapse neurons.

These curve detectors detect

significantly larger radii curves than their predecessors. They will be

refined into more specific, larger curve detectors in the next layer.

### [**BW vs Color** 4%](#group_mixed3a_bw_vs_color)

These "black and white" detectors

respond to absences of color. Prior to this, color detectors contrast to

 the opposite hue, but from this point on we'll see many compare to the

### [**Angles** 3%](#group_mixed3a_angles)

Units that detect multiple lines,

forming angles, triangles and squares. They generally respond to any of

the individual lines, and more strongly to them together.

### [**Fur Precursors** 3%](#group_mixed3a_fur_precursors)

These units are not yet highly selective

 for fur (they also fire for other high-frequency patterns), but their

 At the 224x224 image resolution, individual fur hairs are generally not

 detectable, but tufts of fur are. These units use Gabor textures to

detect those tufts in different orientations. The also detect lower

frequency edges or changes in lighting perpendicular to the tufts.

### [**Eyes / Small Circles** 2%](#group_mixed3a_eyes_small_circles)

### [**Crosses / Diverging Lines** 2%](#group_mixed3a_crosses_diverging_lines)

### [**Thick Lines** 1%](#group_mixed3a_thick_lines)

Low confidence organizational category.

### [**Line Ends** 1%](#group_mixed3a_line_ends)

---

`mixed3b`

---------

`mixed3b` straddles two levels of abstraction.

On the one hand, it has some quite sophisticated features that don't

really seem like they should be characterized as "early" or "low-level":

 object boundary detectors, early head detectors, and more sophisticated

 part of shape detectors.

On the other hand, it also has many units that still feel quite

low-level, such as color center-surround units.

 When you first look at the feature visualizations and dataset examples,

 you might think these are just another iteration of edge or curve detectors.

positive (excitation)

negative (inhibition)

High-low frequency detectors

Edges

End of Line

Color Contrasts

 We sometimes find it useful to think about the "goal" of early vision.

**Curve-based Features:**

 These include more sophisticated curves,

 [divots](#group_mixed3b_divots), and ["evolutes"](#group_mixed3b_evolute)

Curve

Circle

Spiral

Evolute

positive (excitation)

negative (inhibition)

 Again, these circuits only scratch the surface of `mixed3b`.

 Since it's a larger layer with lots of families, we'll go through a

couple particularly interesting and well understood families first:

### [**Boundary** 8%](#group_mixed3b_boundary)

Show all 36 neurons.

Collapse neurons.

These units use multiple cues to detect

the boundaries of objects. They vary in orientation, detecting

convex/concave/straight boundaries, and detecting artificial vs fur

foregrounds. Cues they rely on include line detectors, high-low

frequency detectors, and color contrast.

### [**Proto-Head** 3%](#group_mixed3b_proto_head)

Show all 12 neurons.

Collapse neurons.

The tiny eye detectors, along with

texture detectors for fur, hair and skin developed at the previous layer

 enable these early head detectors, which will continue to be refined in

 the next layer.

### [**Generic, Oriented Fur** 2%](#group_mixed3b_generic_oriented_fur)

We don't typically think of fur as an

oriented feature, but it is. These units detect fur parting in various

ways, much like how hair on your head parts.

### [**Curves** 2%](#group_mixed3b_curves)

The third iteration of curve detectors.

They detect larger radii curves than their predecessors, and are the

first to not slightly fire for curves rotated 180 degrees. Compare to

### [**Divots** 2%](#group_mixed3b_divots)

Curve-like detectors for sharp corners or bumps.

### [**Square / Grid** 2%](#group_mixed3b_square_grid)

Units detecting grid patterns.

### [**Brightness Gradients** 1%](#group_mixed3b_brightness_gradients)

### [**Eyes** 1%](#group_mixed3b_eyes)

### [**Shallow Curves** 1%](#group_mixed3b_shallow_curves)

### [**Curve Shapes** 1%](#group_mixed3b_curve_shapes)

Simple shapes created by composing curves, such as spirals and S-curves.

### [**Circles / Loops** 1%](#group_mixed3b_circles_loops)

### [**Circle Cluster** 1%](#group_mixed3b_circle_cluster)

### [**Double Curves** 1%](#group_mixed3b_double_curves)

### [**Evolute** 0.2%](#group_mixed3b_evolute)

 One frustrating issue is that `mixed3b` has many units that

 don't have a simple low-level articulation, but also are not yet very

specific to a high-level feature.

 For example, there are units which seem to be developing towards

detecting certain animal body parts, but still respond to many other

stimuli as well and so are difficult to describe.

### [**Color Center-Surround** 16%](#group_mixed3b_color_center_surround)

Show all 77 neurons.

Collapse neurons.

These units look for one color in the

center, and another color surrounding it. These units likely have many

subtleties about the range of hues, texture preferences, and

interactions that similar neurons in earlier layers may not have. Note

how many units detect the absence (or generic presence) of color,

### [**Complex Center-Surround** 15%](#group_mixed3b_complex_center_surround)

Show all 73 neurons.

Collapse neurons.

This is a broad, not very well defined

category for center-surround units that detect a pattern or complex

texture in their center.

### [**Texture** 9%](#group_mixed3b_texture)

Show all 44 neurons.

Collapse neurons.

This is a broad, not very well defined

category for units that seem to look for simple local structures over a

wide receptive field, including mixtures of colors.

### [**Other Units** 9%](#group_mixed3b_other_units)

Show all 42 neurons.

Collapse neurons.

Units that don't fall in any other category.

### [**Color Contrast/Gradient** 5%](#group_mixed3b_color_contrast_gradient)

Show all 24 neurons.

Collapse neurons.

Units which respond to different colors

on each side. These units look for one color in the center, and another

color surrounding it. These units likely have many subtleties about the

range of hues, texture preferences, and interactions that similar

neurons in earlier layers may not have. Compare to earlier color

### [**Texture Contrast** 3%](#group_mixed3b_texture_contrast)

Show all 12 neurons.

Collapse neurons.

Units that detect one texture on one side and a different texture on the other.

### [**Other Fur** 2%](#group_mixed3b_other_fur)

Units which seem to detect fur but,

unlike the oriented fur detectors, don't seem to detect it parting in a

particular way. Many of these seem to prefer a particular fur pattern.

### [**Lines** 2%](#group_mixed3b_lines)

### [**Cross / Corner Divergence** 2%](#group_mixed3b_cross_corner_divergence)

### [**Pattern** 2%](#group_mixed3b_pattern)

Low confidence category.

### [**Bumps** 2%](#group_mixed3b_bumps)

Low confidence category.

### [**Double Boundary** 1%](#group_mixed3b_double_boundary)

### [**Bar / Line-Like** 1%](#group_mixed3b_bar_line_like)

Low confidence category.

### [**Boundary Misc** 1%](#group_mixed3b_boundary_misc)

Boundary-related units we didn't know what else to do with.

### [**Star** 1%](#group_mixed3b_star)

Low confidence category.

### [**Line Grad** 1%](#group_mixed3b_line_grad)

Low confidence category.

### [**Scales** 1%](#group_mixed3b_scales)

We don't really understand these units.

### [**Curves misc.** 1%](#group_mixed3b_curves_misc.)

Low confidence organizational category.

### [**Shiny** 0.4%](#group_mixed3b_shiny)

Units that seem to detect shiny, specular surfaces.

### [**Pointy** 0.4%](#group_mixed3b_pointy)

Low confidence category.

---

Conclusion

----------

 The goal of this essay was to give a high-level overview of our

present understanding of early vision in InceptionV1.

 Every single feature discussed in this article is a potential topic of

 deep investigation.

 For example, are curve detectors really curve detectors? What types of

 curves do they fire for? How do they behave on various edge cases? How

are they built?

 Over the coming articles, we'll do deep dives rigorously investigating

 these questions for a few features, starting with curves.

 Is there a better taxonomy, or another way to understand the space of features?

 Why do features often seem to form in families?

 To what extent do the same features families form across models?

 Is there a "periodic table of low-level visual features", in some sense?

 To what extent do later features admit a similar taxonomy?

 We think these could be interesting questions for future work.

 a collection of short articles and commentary by an open scientific

collaboration delving into the inner workings of neural networks.

[Curve Detectors](https://distill.pub/2020/circuits/curve-detectors/)

Going deeper with convolutions

Network dissection: Quantifying interpretability of deep visual representations

Visualizing higher-layer features of a deep network

Feature Visualization

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

The Building Blocks of Interpretability

TensorFlow-Slim: A lightweight library for defining, training and evaluating complex models in TensorFlow


Communicating with Interactive Articles

Fred Hohman
Matthew Conlen
Jeffrey Heer
Duen Horng (Polo) Chau

Abstract: Computing has changed how people communicate. The transmission  of news, messages, and ideas is instant. Anyone’s voice can be heard. In  fact, access to digital communication technologies such as the Internet  is so fundamental to daily life that their disruption by government is  condemned by the United Nations Human Rights Council . But while the technology to distribute our ideas has grown in leaps and bounds, the interfaces have remained largely the same.

### Contents

[Introduction](#introduction)

[Interactive Articles: Theory & Practice](#interactive-articles)

* [Connecting People and Data](#connecting-people-and-data)

* [Making Systems Playful](#making-systems-playful)

* [Prompting Self-Reflection](#prompting-self-reflection)

* [Personalizing Reading](#personalizing-reading)

* [Reducing Cognitive Load](#reducing-cognitive-load)

[Challenges for Authoring Interactives](#challenges)

[Critical Reflections](#critical-reflections)

[Looking Forward](#looking-forward)

 Computing has changed how people communicate. The transmission

of news, messages, and ideas is instant. Anyone's voice can be heard. In

 fact, access to digital communication technologies such as the Internet

 is so fundamental to daily life that their disruption by government is

 Parallel to the development of the internet, researchers like

Alan Kay and Douglas Engelbart worked to build technology that would

empower individuals and enhance cognition. Kay imagined the Dynabook

 in the hands of children across the world. Engelbart, while best

remembered for his "mother of all demos," was more interested in the

ability of computation to augment human intellect .

 Neal Stephenson wrote speculative fiction that imagined interactive

paper that could display videos and interfaces, and books that could

teach and respond to their readers .

 More recent designs (though still historical by personal computing

 standards) point to a future where computers are connected and assist

people in decision-making and communicating using rich graphics and

 unfortunately, many others have not. The most popular publishing

platforms, for example WordPress and Medium, choose to prioritize social

 features and ease-of-use while limiting the ability for authors to

communicate using the dynamic features of the web.

 In the spirit of previous computer-assisted cognition

technologies, a new type of computational communication medium has

emerged that leverages active reading techniques to make ideas more

accessible to a broad range of people. These interactive articles build

 In this work, for the the first time, we connect the dots between

interactive articles such as those featured in this journal and

publications like *The New York Times* and the techniques,

theories, and empirical evaluations put forth by academic researchers

across the fields of education, human-computer interaction, information

visualization, and digital journalism. We show how digital designers are

 operationalizing these ideas to create interactive articles that help

boost learning and engagement for their readers compared to static

alternatives.

 Conducting

 novel research requires deep understanding and expertise in a specific

area. Once achieved, researchers continue contributing new knowledge for

 future researchers to use and build upon. Over time, this consistent

addition of new knowledge can build up, contributing to what some have

called research debt. Not everyone is an expert in every field, and it

can be easy to lose perspective and forget the bigger picture. Yet

research should be understood by many. Interactive articles can be used

to distill the latest progress in various research fields and make their

 methods and results accessible and understandable to a broader

audience. #### Opportunities

 * Engage and excite broader audience with latest research progress

* Remove research debt, onboard new researchers

* Make faster and clearer research progress

 #### Challenges

 * No clear incentive structure for researchers

* Little funding for bespoke research dissemination and communication

 companion piece to a traditional research paper that uses interactive

visualizations to let readers adjust a machine learning model's behavior

 PhD thesis that contributes a programming language abstraction for

understanding how programs access the context or environment in which

they execute, and walks readers through the work using two simple

 crash course in complex systems science, created by leading experts,

practitioners, and students in the field, with accompanying interactive

 Interactive articles are applicable to variety of domains, such as

research dissemination, journalism, education, and policy and decision

 Today there is a growing excitement around the use of interactive

articles for communication since they offer unique capabilities to help

people learn and engage with complex ideas that traditional media lacks.

 After describing the affordances of interactive articles, we provide

critical reflections from our own experience with open-source,

interactive publishing at scale. We conclude with discussing practical

challenges and open research directions for authoring, designing, and

publishing interactive articles.

 This style of communication — and the platforms which support

it — are still in their infancy. When choosing where to publish this

work, we wanted the medium to reflect the message. Journals like *Distill*

 are not only pushing the boundaries of machine learning research but

also offer a space to put forth new interfaces for dissemination. This

work ties together the theory and practice of authoring and publishing

interactive articles. It demonstrates the power that the medium has for

providing new representations and interactions to make systems and ideas

 more accessible to broad audiences.

Interactive Articles: Theory & Practice

---------------------------------------

 Interactive articles draw from and connect many types of media,

from static text and images to movies and animations. But in contrast to

 these existing forms, they also leverage interaction techniques such as

 details-on demand, belief elicitation, play, and models and simulations

 to enhance communication.

 While the space of possible designs is far too broad to be solved

with one-size-fits-all guidelines, by connecting the techniques used in

these articles back to underlying theories presented across disparate

fields of research we provide a missing foundation for designers to use

when considering the broad space of interactions that could be added to a

 born-digital article.

 We draw from a corpus of over sixty interactive articles to

highlight the breadth of techniques available and analyze how their

authors took advantage of a digital medium to improve the reading

experience along one or more dimensions, for example, by reducing the

overall cognitive load, instilling positive affect, or improving

information recall.

| Title *arrow\_downward* | Publication or Author  | Tags  | Year  |

| --- | --- | --- | --- |

 Because diverse communities create interactive content, this

medium goes by many different names and has not yet settled on a

 In newsrooms, data journalists, developers, and designers work together

 to make complex news and investigative reporting clear and engaging

using interactive stories . Educators

 use interactive textbooks as an alternative learning format to give

students hands-on experience with learning material .

 Besides these groups, others such as academics, game developers,

web developers, and designers blend editorial, design, and programming

 While these all slightly differ in their technical approach and target

audience, they all largely leverage the interactivity of the modern web.

 We focus on five unique affordances of interactive articles,

listed below. In-line videos and example interactive graphics are

presented alongside this discussion to demonstrate specific techniques.

### Connecting People and Data

 an audience which finds content to be aesthetically pleasing is more

likely to have a positive attitude towards it. This in turn means people

 will spend more time engaging with content and ultimately lead to

improved learning outcomes. While engagement itself may not be an end

goal of most research communications, the ability to influence both

audience attitude and the amount of time that is spent is a useful lever

 to improve learning: we know from education research that both time

spent  and emotion  are predictive of learning outcomes.

 Animations can also be used to improve engagement .

 While there is debate amongst researchers if animations in general are

able to more effectively convey the same information compared to a well

 while the series of still images may be more effective for answering

specific questions like, "Does a horse lift all four of its feet off the

 ground when it runs?" watching the animation in slow motion gives the

viewer a much more visceral sense of how it runs. A more modern example

 1878, Eadweard Muybridge settled Leland Stanford's hotly debated

question of whether all four feet of a horse lifted off the ground

during a trot using multiple cameras to capture motion in stop-motion

 Passively, animation can be used to add drama to a graphic

displaying important information, but which readers may otherwise find

dry. Scientific data which is inherently time varying may be shown using

 an animation to connect viewers more closely with the original data, as

 compared to seeing an abstracted static view. For example, Ed Hawkins

designed "Climate Spirals," which shows the average global temperature

change over time . This

presentation of the data resonated with a large public audience, so much

 so that it was displayed at the opening ceremony at the 2016 Rio

Olympics. In fact, many other climate change visualizations of this same

 dataset use animation to build suspense and highlight the recent spike

in global temperatures .

 Preview,

 click to play

 Full Video.

 By adding variation over time, authors have access to a new

dimension to encode information and an even wider design space to work

in. Consider the animated graphic in *The New York Times* story

"Extensive Data Shows Punishing Reach of Racism for Black Boys," which

 by utilizing animation, it became possible for the authors to design a

unit visualization in which each data point shown represented an

individual, reminding readers that the data in this story was about real

 peoples' lives.

 Unit visualizations have also been used to evoke empathy in

 Using person-shaped glyphs (as opposed to abstract symbols like circles

 or squares) has been shown not to produce additional empathic responses

 using visualizations. Correll argues that much of the power of

visualization comes from abstraction, but quantization stymies empathy .

 He instead suggests anthropomorphizing data, borrowing journalistic and

 rhetoric techniques to create novel designs or interventions to foster

empathy in readers when viewing visualizations .

 Regarding the format of interactive articles, an ongoing debate

within the data journalism community has been whether articles which

utilize scroll-based graphics (scrollytelling) are more effective than

those which use step-based graphics (slideshows). McKenna et al.

 found that their study participants largely preferred content to be

displayed with a step- or scroll-based navigation as opposed to

traditional static articles, but did not find a significant difference

in engagement between the two layouts. In related work, Zhi et al. found

 that performance on comprehension tasks was better in slideshow layouts

 than in vertical scroll-based layouts .

 Both studies focused on people using desktop (rather than mobile)

devices. More work is needed to evaluate the effectiveness of various

layouts on mobile devices, however the interviews conducted by MckEnna

et al. suggest that additional features, such as supporting navigation

through swipe gestures, may be necessary to facilitate the mobile

reading experience.

readers play the role of a pirate commander, giving them a unique look

at the economics that led to rise in piracy off the coast of Somalia. Playing

 Preview,

 click to play

 Full Video.

 the reading experience becomes closer to that of playing a game. For

example, the critically acclaimed explorable explanation "Parable of the

 Polygons" puts play at the center of the story, letting a reader

manually run an algorithm that is later simulated in the article to

demonstrate how a population of people with slight personal biases

against diversity leads to social segregation .

### Making Systems Playful

 Interactive articles utilize an underlying computational

infrastructure, allowing authors editorial control over the

computational processes happening on a page. This access to computation

allows interactive articles to engage readers in an experience they

could not have with traditional media. For example, in "Drawing Dynamic

Visualizations", Victor demonstrates how an interactive visualization

can allow readers to build an intuition about the behavior of a system,

leading to a fundamentally different understanding of an underlying

 Complex systems often require extensive setup to allow for proper

study: conducting scientific experiments, training machine learning

models, modeling social phenomenon, digesting advanced mathematics, and

researching recent political events, all require the configuration of

sophisticated software packages before a user can interact with a system

 at all, even just to tweak a single parameter. This barrier to entry

can deter people from engaging with complex topics, or explicitly

prevent people who do not have the necessary resources, for example,

computer hardware for intense machine learning tasks. Interactive

articles drastically lower these barriers.

 Science that utilizes physical and computational experiments

requires systematically controlling and changing parameters to observe

their effect on the modeled system. In research, dissemination is

typically done through static documents, where various figures show and

compare the effect of varying particular parameters. However, efforts

have been made to leverage interactivity in academic publishing,

 gives readers control over the reporting of the research findings and

shows great promise in helping readers both digest new ideas and learn

about existing fields that are built upon piles of research debt .

 Beyond reporting statistics, interactive articles are extremely

powerful when the studied systems can be modeled or simulated in

real-time with interactive parameters without setup, e.g., in-browser

sandboxes. Consider the example in [4](#simulation-vis)

 of a Boids simulation that models how birds flock together. Complex

systems such as these have many different parameters that change the

resulting simulation. These sandbox simulations allow readers to play

with parameters to see their effect without worrying about technical

overhead or other experimental consequences.

Interact with live simulations—no setup required. This

 Boids visualization models the movement of a flock of birds, and

exposes parameters that a reader can manipulate to change the behavior

of the simulation. Boid Count

 At the top, drag the slider to change the number of boids in the

simulation. Underneath, adjust the different parameters to find

interesting configurations.

 This is a standout design pattern within interactive articles, and

 many examples exist ranging in complexity. "How You Will Die" visually

simulates the average life expectancy of different groups of people,

where a reader can choose the gender, race, and age of a person .

 "On Particle Physics" allows readers to experiment with accelerating

different particles through electric and magnetic fields to build

intuition behind electromagnetism foundations such as the Lorentz force

and Maxwell's equations — the experiments backing these simulations

cannot be done without multi-million dollar machinery .

 "Should Prison Sentences Be Based On Crimes That Haven't Been Committed

 Yet?" shows the outcome of calculating risk assessments for recidivism

where readers adjust the thresholds for determining who gets parole .

 reader uses their own live video camera to train a machine learning

image classifier in-browser without any extra computational resources. Playing

 Preview,

 click to play

 Full Video.

 The dissemination of modern machine learning techniques has been

bolstered by interactive models and simulations. Three articles, "How to

 show the effect that hyperparameters and different dimensionality

reduction techniques have on creating low dimensional embeddings of

high-dimensional data. A popular approach is to demonstrate how machine

 Other examples are aimed at technical readers who wish to learn about

specific concepts within deep learning. Here, interfaces allow readers

to choose model hyperparameters, datasets, and training procedures that,

 once selected, visualize the training process and model internals to

inspect the effect of varying the model configuration .

 Interactive articles commonly communicate a single idea or concept

 using multiple representations. The same information represented in

different forms can have different impact. For example, in mathematics

often a single object has both an algebraic and a geometric

representation. A clear example of this is the definition of a circle .

 Both are useful, inform one another, and lead to different ways of

thinking. Examples of interactive articles that demonstrate this include

 various media publications' political election coverage that break down

 the same outcome in multiple ways, for example, by voter demographics,

geographical location, and historical perspective .

 as people can process information through both a visual channel and

auditory channel simultaneously. Popular video creators such as

3Blue1Brown  and Primer

 exemplify these principles by using rich animation and simultaneous

narration to break down complex topics. These videos additionally take

advantage of the Redundancy Principle by including complementary

information in the narration and in the graphics rather than repeating

the same information in both channels .

 Preview,

 click to play

 Full Video.

 While these videos are praised for their approachability and

rich exposition, they are not interactive. One radical extension from

traditional video content is also incorporating user input into the

video while narration plays. A series of these interactive videos on

"Visualizing Quaternions" lets a reader listen to narration of a live

animation on screen, but at any time the viewer can take control of the

video and manipulate the animation and graphics while simultaneously

listening to the narration .

 Utilizing multiple representations allows a reader to see

different abstractions of a single idea. Once these are familiar and

known, an author can build interfaces from multiple representations and

let readers interact with them simultaneously, ultimately leading to

interactive experiences that demonstrate the power of computational

communication mediums. Next, we discuss such experiences where

interactive articles have transformed communication and learning by

making live models and simulations of complex systems and phenomena

accessible.

### Prompting Self-Reflection

 Asking a student to reflect on material that they are studying and

 explain it back to themselves — a learning technique called

self-explanation — is known to have a positive impact on learning

outcomes . By generating explanations

 and refining them as new information is obtained, it is hypothesized

that a student will be more engaged with the processes which they are

studying . When writing for an

interactive environment, components can be included which prompt readers

 to make a prediction or reflection about the material and cause them to

 engage in self-explanation .

 While these prompts may take the form of text entry or other

standard input widgets, one of the most prominent examples of this

 In these visualizations, readers are prompted to complete a trendline

on a chart, causing them to generate an explanation based on their

current beliefs for why they think the trend may move in a certain

direction. Only after readers make their prediction are they shown the

actual data. Kim et al. showed that using visualizations as a prompt is

an effective way to encourage readers to engage in self explanation and

improve their recall of the information . [5](#you-draw-it)

 shows one these visualizations for CO₂ emissions from burning fossil

fuels. After clicking and dragging to guess the trend, your guess will

be compared against the actual data.

Complete the trend of CO₂ emissions from burning fossil fuels. Letting

 a reader first guess about data and only showing the ground truth

afterwards challenges a reader's prior beliefs and has been shown to

readers are tasked to type the names of celebrities with challenging

spellings. After submitting a guess, a visualization shows the reader's

entry against everyone else's, scaled by the frequency of different

spellings. Playing

 Preview,

 click to play

 Full Video.

 In the case of "You Draw It," readers were also shown the

predictions that others made, adding a social comparison element to the

experience. This additional social information was not shown to

necessarily be effective for improving recall .

 However, one might hypothesize that this social aspect may have other

benefits such as improving reader engagement, due to the popularity of

 Prompting readers to remember previously presented material, for

 example through the use of quizzes, can be an effective way to improve

 While testing may call to mind stressful educational experiences for

many, quizzes included in web articles can be low stakes: there is no

need to record the results or grade readers. The effect is enhanced if

feedback is given to the quiz-takers, for example by providing the

correct answer after the user has recorded their response .

 Preview,

 click to play

 Full Video.

 assuming readers are willing to participate in the process. The idea of

 spaced repetition has been a popular foundation for memory building

applications, for example in the Anki flash card system. More recently,

authors have experimented with building spaced repetition directly into

### Personalizing Reading

 Content personalization — automatically modifying text and

multimedia based on a reader's individual features or input (e.g.,

demographics or location) — is a technique that has been shown to

increase engagement and learning within readers  and support behavioral change .

 The PersaLog system gives developers tools to build personalized

content and presents guidelines for personalization based on user

research from practicing journalists .

 Other work has shown that "personalized spatial analogies," presenting

distance measurements in regions where readers are geographically

familiar with, help people more concretely understand new distance

measurements within news stories .

 reader enters their birthplace and birth year and is shown multiple

visualizations describing the impact of climate on their hometown. Playing

 Preview,

 click to play

 Full Video.

 Personalization alone has also been used as the standout feature

of multiple interactive articles. Both "How Much Hotter Is Your Hometown

 Than When You Were Born?"  and "Human Terrain"

 use a reader's location to drive stories relating to climate change and

 population densities respectively. Other examples ask for explicit

reader input, such as a story that visualizes a reader's net worth to

challenge a reader's assumptions if they are wealthy or not (relative to

 In this visualization, professions are plotted to determine their

likelihood of being automated against their average annual wage. The

article encourages readers to use the search bar to type in their own

profession to highlight it against the others.

 An interactive medium has the potential to offer readers an

experience other than static, linear text. Non-linear stories, where a

reader can choose their own path through the content, have the potential

 to provide a more personalized experience and focus on areas of user

 a technology focused news television program. Non-linear stories

present challenges for authors, as they must consider the myriad

possible paths through the content, and consider the different possible

experiences that the audience would have when pursuing different

branches.

 [8](#matuschak2019quantum): In "[Quantum Country](https://quantum.country/)" ,

the interactive textbook uses spaced repetition and allows a reader to

opt-in and save their progress while reading through dense material and

mathematical notion over time. Playing

 Preview,

 click to play

 Full Video.

 Another technique interactive articles often use is segmenting

content into small pieces to be read in-between or alongside other

graphics. While we have already discussed cognitive load theory, the

Segmenting Theory, the idea that complex lessons are broken into

smaller, bit-sized parts , also

 supports personalization within interactive articles. Providing a

reader the ability to play, pause, and scrub content allows the reader

to move at their own speed, comprehending the information at a speed

that works best for them. Segmenting also engages a reader's essential

processing without overloading their cognitive system .

 Multiple studies have been conducted showing that learners perform

 better when information is segmented, whether it be only within an

animation  or within an interface with textual descriptions .

 One excellent example of using segmentation and animation to

personalize content delivery is "A Visual Introduction to Machine

Learning," which introduces fundamental concepts within machine learning

 in bite-sized pieces, while transforming a single dataset into a

trained machine learning model .

Extending this idea, in "Quantum Country," an interactive textbook

covering quantum computing, the authors implemented a user account

system, allowing readers to save their position in the text and consume

### Reducing Cognitive Load

 Authors must calibrate the detail at which to discuss ideas and

content to their readers expertise and interest to not overload them.

When topics become multifaceted and complex, a balance must be struck

between a high-level overview of a topic and its lower-level details.

One interaction technique used to prevent a cognitive overload within a

reader is "details-on-demand."

 Details-on-demand has become an ubiquitous design pattern. For

example, modern operating systems offer to fetch dictionary definitions

when a word is highlighted. When applied to visualization, this

technique allows users to select parts of a dataset to be shown in more

detail while maintaining a broad overview. This is particularly useful

when a change of view is not required, so that users can inspect

elements of interest on a point-by-point basis in the context of the

whole . Below we highlight areas

where details-on-demand has been successfully applied to reduce the

amount of information present within an interface at once.

#### Data Visualization

 Successful visualizations not only provide the base representations and

 techniques for these three steps, but also bridge the gaps between them

 . In practice, the

solidified standard for details-on-demand in data visualization

manifests as a tooltip, typically summoned on a cursor mouseover, that

presents extra information in an overlay. Given that datasets often

contain multiple attributes, tooltips can show the other attributes that

#### Illustration

 Details-on-demand is also used in illustrations, interactive

textbooks, and museum exhibits, where highlighted segments of a figure

can be selected to display additional information about the particular

segment. For example, in "How does the eye work?" readers can select

segments of an anatomical diagram of the human eye to learn more about

specific regions, e.g., rods and cones .

 Another example is "Earth Primer," an interactive textbook on tablets

that allows readers to inspect the Earth's interior, surface, and biomes

#### Mathematical Notation

 Formal mathematics, a historically static medium, can benefit from

 details-on-demand, for example, to elucidate a reader with intuition

about a particular algebraic term, present a geometric interpretation of

 an equation, or to help a reader retain high-level context while

 For example, in "Why Momentum Really Works," equation layout is done

using Gestalt principles plus annotation to help a reader easily

identify specific terms. In "Colorized

 Math Equations," the Fourier transform equation is presented in both

mathematical notation and plain text, but the two are linked through a

mouseover that highlights which term in the equation corresponds to

which word in the text .

Another example that visualizes mathematics and computation is the

"Image Kernels" tutorial where a reader can mouseover a real image and

observe the effect and exact computation for applying a filter over the

image .

 Instead of writing down long arithmetic sums, the interface allows

 one of Maxwell's equations is shown. Click the two buttons to reveal,

or remind yourself, what each notation mark and variable represent.

Enhancing mathematics design with annotation and interactivity. Working

 with equations requires a sharp working memory. Optional interactivity

can help people remember specific notation and variable defintions, only

#### Text

 While not as pervasive, text documents and other long-form textual

 mediums have also experimented with letting readers choose a variable

level of detail to read. This idea was explored as early as the 1960s in

 StretchText, a hypertext feature that allows a reader to reveal a more

descriptive or exhaustive explanation of something by expanding or

 One challenge that has limited this technique's adoption is the burden

it places on authors to write multiple versions of their content. For

example, drag the slider in [9](#details-text)

 to read descriptions of the Universal Approximation Theorem in

increasing levels of detail. For other examples of details-on-demand for

 text, such as application in code documentation, see this small

collection of examples .

 networks can approximate any function that exists. However, we do not

have a guaranteed way to obtain such a neural network for every

function.

#### Previewing Content

 Details-on-demand can also be used as a method for previewing

content without committing to another interaction or change of view. For

 example, when hovering over a hyperlink on Wikipedia, a preview card is

 shown that can contain an image and brief description; this gives

readers a quick preview of the topic without clicking through and

 within hypertext that present information about a particular topic in a

 location that does not obscure the source material. Both older and

modern preview techniques use perceptually-based animation and simple

tooltips to ensure their interactions are natural and lightweight

feeling to readers .

Challenges for Authoring Interactives

-------------------------------------

*If interactive articles provide clear benefits over other

mediums for communicating complex ideas, then why aren't they more

prevalent?*

 Unfortunately, creating interactive articles today is difficult.

Domain-specific diagrams, the main attraction of many interactive

articles, must be individually designed and implemented, often from

scratch. Interactions need to be intuitive and performant to achieve a

nice reading experience. Needless to say, the text must also be

well-written, and, ideally, seamlessly integrated with the graphics.

 The act of creating a successful interactive article is closer to

building a website than writing a blog post, often taking significantly

more time and effort than a static article, or even an academic

 Most interactive articles are created using general purpose

web-development frameworks which, while expressive, can be difficult to

work with for authors who are not also web developers. Even for expert

web developers, current tools offer lower levels of abstraction than may

 be desired to prototype and iterate on designs.

 can help authors start writing quickly and even enable rapid iteration

through various designs (for example, letting an author quickly compare

between sequencing content using a "scroller" or "stepper" based

layout). However, Idyll does not offer any design guidance, help authors

 think through where interactivity would be most effectively applied,

nor highlight how their content could be improved to increase its

readability and memorability. For example, Idyll encodes no knowledge of

 the positive impact of self-explanation, instead it requires authors to

 be familiar with this research and how to operationalize it.

 To design an interactive article successfully requires a diverse

set of editorial, design, and programming skills. While some individuals

 are able to author these articles on their own, many interactive

articles are created by a collective team consisting of multiple members

 with specialized skills, for example, data analysts, scripters,

editors, journalists, graphic designers, and typesetters, as outlined in

 requires one to clone its source code using git, install

project-specific dependencies using a terminal, and be comfortable

editing HTML files. All of this complexity is incidental to task of

editing text.

 Publishing to the web brings its own challenges: while interactive

 articles are available to anyone with a browser, they are burdened by

rapidly changing web technologies that could break interactive content

after just a few years. For this reason, easy and accessible interactive

 article archival is important for authors to know their work can be

 Authoring interactive articles also requires designing for a diverse

set of devices, for example, ensuring bespoke content can be adapted for

 desktop and mobile screen sizes with varying connection speeds, since

accessing interactive content demands more bandwidth.

 There are other non-technical limitations for publishing

interactive articles. For example, in non-journalism domains, there is a

 mis-aligned incentive structure for authoring and publishing

interactive content: why should a researcher spend time on an "extra"

interactive exposition of their work when they could instead publish

more papers, a metric by which their career depends on? While different

groups of people seek to maximize their work's impact, legitimizing

interactive artifacts requires buy-in from a collective of communities.

 Making interactive articles accessible to people with disabilities

 is an open challenge. The dynamic medium exacerbates this problem

compared to traditional static writing, especially when articles combine

 multiple formats like audio, video, and text. Therefore, ensuring

interactive articles are accessible to everyone will require alternative

 modes of presenting content (e.g. text-to-speech, video captioning,

data physicalization, data sonification) and careful interaction design.

 It is also important to remember that not everything needs to be

interactive. Authors should consider the audience and context of their

work when deciding if use of interactivity would be valuable. In the

worst case, interactivity may be distracting to readers or the

functionality may go unused, the author having wasted their time

implementing it. However, even in a domain where the potential

Critical Reflections

--------------------

 We write this article not as media theorists, but as

practitioners, researchers, and tool builders. While it has never been

easier for writers to share their ideas online, current publishing tools

 largely support only static authoring and do not take full advantage of

 the fact that the web is a dynamic medium. We want that to change, and

we are not alone. Others from the explorable explanations community have

 identified design patterns that help share complex ideas through play .

 an annually published digital magazine that showcases the expository

power that interactive dynamic media can have when effectively combined .

 In late 2018, we invited writers to respond to a call for proposals for

 our first issue focusing on exploring scientific and technological

phenomena that stand to shape society at large. We sought to cover

topics that would benefit from using the interactive or otherwise

dynamic capabilities of the web. Given the challenges of authoring

interactive articles, we did not ask authors to submit fully developed

pieces. Instead, we accepted idea submissions, and collaborated with the

 authors over the course of four months to develop the issue, offering

technical, design, and editorial assistance collectively to the authors

that lacked experience in one of these areas. For example, we helped a

writer implement visualizations, a student frame a cohesive narrative,

and a scientist recap history and disseminate to the public. Multiple

views from one article are shown in [10](#parametric).

 The article used techniques like animation, data visualizations,

explanatory diagrams, margin notes, and interactive simulations to

explain how biases occur in machine learning systems.

 We see *The Parametric Press* as a crucial connection between

 the often distinct worlds of research and practice. The project serves

as a platform through which to operationalize the theories put forth by

education, journalism, and HCI researchers. Tools like Idyll which are

designed in a research setting need to be validated and tested to ensure

 that they are of practical use; *The Parametric Press* facilitates

 this by allowing us to study its use in a real-world setting, by

authors who are personally motivated to complete their task of

constructing a high-quality interactive article, and only have secondary

 concerns and care about the tooling being used, if at all.

|  | Research | Practice |

| --- | --- | --- |

 As researchers we can treat the project as a series of case

studies, where we were observers of the motivation and workflows which

were used to craft the stories, from their initial conception to their

publication. Motivation to contribute to the project varied by author.

Where some authors had personal investment in an issue or dataset they

wanted to highlight and raise awareness to broadly, others were drawn

towards the medium, recognizing its potential but not having the

expertise or support to communicate interactively. We also observed how

research software packages like Apparatus , Idyll , and D3

 fit into the production of interactive articles, and how authors must

combine these disparate tools to create an engaging experience for

readers. In one article, "On Particle Physics," an author combined two

tools in a way that allowed him to create and embed dynamic graphics

directly into his article without writing any code beyond basic markup.

One of the creators of Apparatus had not considered this type of

 fantastic! Reading that article, I had no idea that Apparatus was used.

 This is a very exciting proof-of-concept for unconventional

explorable-explanation workflows."*

 We were able to provide editorial guidance to the authors drawing

on our knowledge of empirical studies done in the multimedia learning

and information visualization communities to recommend graphical

structures and page layouts, helping each article's message be

communicated most effectively. One of the most exciting outcomes of the

project is that we saw authors develop interactive communication skills

like any other skill: through continued practice, feedback, and

iteration. We also observed the challenges that are inherent in

publishing dynamic content on the web and identified the need for

improved tooling in this area, specifically around the archiving of

interactive articles. Will an article's code still run a year from now?

Ten years from now? To address interactive content archival, we set up a

 system to publish a digital archive of all of our articles at the time

that they are first published to the site. At the top of each article on

 *The Parametric Press* is an archive link that allows readers to

download a WARC (Web ARChive) file that can "played back" without

requiring any web infrastructure. While our first iteration of the

project relied on ad-hoc solutions to these problems, we hope to show

how digital works such as ours can be published confidently knowing that

 they will be preserved indefinitely.

 As practitioners we pushed the boundaries of the current

generation of tools designed to support the creation of interactive

articles on the web. We found bugs and limitations in Idyll, a tool

which was originally designed to support the creation of one-off

articles that we used as a content management system to power an entire

magazine issue. We were forced to write patches and plugins to work

 We were also forced to craft designs under a more realistic set of

constraints than academics usually deal with: when creating a

visualization it is not enough to choose the most effective visual

encodings, the graphics also had to be aesthetically appealing, adhere

to a house style, have minimal impact on page load time and runtime

performance, be legible on both mobile and desktop devices, and not be

overly burdensome to implement. Any extra hour spent implementing one

graphic was an hour that was not spent improving some other part of the

issue, such as the clarity of the text, or the overall site design.

 There are relatively few outlets that have the skills, technology,

 and desire to publish interactive articles. From its inception, one of

the objectives of *The Parametric Press* is to showcase the new

forms of media and publishing that are possible with tools available

today, and inspire others to create their own dynamic writings. For

 told us he had wanted to write this interactive article for years yet

never had the opportunity, support, or incentive to create it. His

article drew wide interest and critical acclaim.

 We also wanted to take the opportunity as an independent

publication to serve as a concrete example for others to follow, to

represent a set of best practices for publishing interactive content. To

 that end, we made available all of the software that runs the site,

including reusable components, custom data visualizations, and the

publishing engine itself.

Looking Forward

---------------

 A diverse community has emerged to meet these challenges,

exploring and experimenting with what interactive articles could be. The

 [Explorable Explanations community](https://explorabl.es/)

is a "disorganized 'movement' of artists, coders & educators who

want to reunite play and learning." Their online hub contains 170+

interactive articles on topics ranging from art, natural sciences,

social sciences, journalism, and civics. The curious can also find

tools, tutorials, and meta-discussion around learning, play, and

 Many interactive articles are self-published due to a lack of

platforms that support interactive publishing. Creating more outlets

that allow authors to publish interactive content will help promote

their development and legitimization. The few existing examples,

 help, but currently target a narrow group of authors, namely those who

have programming skills. Such platforms should also provide clear paths

to submission, quality and editorial standards, and authoring

guidelines. For example, news outlets have clear instructions for

pitching written pieces, yet this is under-developed for interactive

articles. Lastly, there is little funding available to support the

development of interactive articles and the tools that support them.

Researchers do not receive grants to communicate their work, and

practitioners outside of the largest news outlets are not able to afford

 the time and implementation investment. Providing more funding for

enabling interactive articles incentivizes their creation and can

contribute to a culture where readers expect digital communications to

better utilize the dynamic medium.

 We have already discussed the breadth of skills required to author

 an interactive article. Can we help lower the barrier to entry? While

there have been great, practical strides in this direction ,

 there is still opportunity for creating tools to design, develop, and

evaluate interactive articles in the wild. Specific features should

include supporting mobile-friendly adaptations of interactive graphics

(for example ),

 creating content for different platforms besides just the web, and

tools that allow people to create interactive content without code.

 The usefulness of interactive articles is predicated on the

assumption that these interactive articles actually facilitate

communication and learning. There is limited empirical evaluation of the

 effectiveness of interactive articles. The problem is exacerbated by

the fact that large publishers are unwilling to share internal metrics,

 provided one of the few available data points, stating that only a

fraction of readers interact with non-static content, and suggested that

 designers should move away from interactivity .

 However, other research found that many readers, even those on mobile

devices, are interested in utilizing interactivity when it is a core

part of the article's message . This statement from *The New York Times*

 has solidified as a rule-of-thumb for designers and many choose not to

utilize interactivity because of it, despite follow-up discussion that

contextualizes the original point and highlights scenarios where

interactivity can be beneficial .

 This means designers are potentially choosing a suboptimal presentation

 of their story due to this anecdote. More research is needed in order

to identify the cases in which interactivity is worth the cost of

creation.

 We believe in the power and untapped potential of interactive

articles for sparking reader's desire to learn and making complex ideas

accessible and understandable to all.

Report on the role of digital access providers

A personal computer for children of all ages

Augmenting human intellect: A conceptual framework

The diamond age

The knowledge navigator

Getting it out of our system

PLATO

PhET interactive simulations

Explorable explanations

How y’all, youse and you guys talk

Snow fall: The avalanche at tunnel creek

Why outbreaks like coronavirus spread exponentially, and how to 'flatten the curve'

Attacking discrimination with smarter machine learning

Coeffects: Context-aware programming languages

Complexity explained

What's really warming the world

You draw it: How family income predicts children’s college chances

The Uber Game

Let's learn about waveforms

The book of shaders

EconGraphs

To build a better ballot

The atlas of redistricting

Is it better to rent or buy

Explorable explanation

Increasing the transparency of research papers with explorable multiverse analyses

Workshop on Visualization for AI Explainability

Exploranation: A new science communication paradigm

Research debt

More than telling a story: Transforming data into visually shared stories

"Concrete" computer manipulatives in mathematics education

Cybertext: Perspectives on ergodic literature

Interactive non-fiction: Towards a new approach for storytelling in digital journalism

Active essays on the web

Newsgames: Journalism at play

Simply bells and whistles? Cognitive effects of visual aesthetics in digital longforms

Learning as a function of time

Emotional design in multimedia learning

Hooked on data videos: assessing the effect of animation and pictographs on viewer engagement

Animation: Can it facilitate?

Animated transitions in statistical data graphics

Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering

La perception de la causalité.(Etudes Psychol.), Vol. VI

The illusion of life: Disney animation

The horse in motion

Emergent tool use from multi-agent autocurricula

Climate spirals

Earth's relentless warming sets a brutal new record in 2017

Global temperature

It's not your imagination. Summers are getting hotter.

Extensive data shows punishing reach of racism for black boys

Disagreements

Gun deaths in america

The fallen of World War II

Showing people behind data: Does anthropomorphizing visualizations elicit more empathy for human rights data?

Beyond memorability: Visualization recognition and recall

What makes a visualization memorable?

What if the data visualization is actually people

Ethical dimensions of visualization research

A walk among the data

Visual narrative flow: Exploring factors shaping data visualization story reading experiences

Linking and layout: Exploring the integration of text and visualization in storytelling

Video games and learning

Cutthroat Capitalism: The Game

Combining software games with education: Evaluation of its educational effectiveness

Narrative visualization: Telling stories with data

Parable of the polygons

Drawing dynamic visualizations

How to read a book: The classic guide to intelligent reading

Scientific communication as sequential art

How you will die

On Particle Physics

Should prison sentences be based on crimes that haven’t been committed yet

How to use t-SNE effectively

The beginner's guide to dimensionality reduction

Understanding UMAP

Tensorflow.js: Machine learning for the web and beyond

Designing (and learning from) a teachable machine

Experiments in handwriting with a neural network

Direct-manipulation visualization of deep networks

Gan lab: Understanding complex deep generative models using interactive visual experimentation

Using artificial intelligence to augment human intelligence

Who will win the presidency

Who will be president

Live results: Presidential election

Multimedia learning

3Blue1Brown

Primer

Revising the redundancy principle in multimedia learning

Visualizing quaternions: An explorable video series

Self-explanations: How students study and use examples in learning to solve problems

Self-explaining expository texts: The dual processes of generating inferences and repairing mental models

Explaining the gap: Visualizing one's predictions improves recall and comprehension of data

You draw it: Just how bad is the drug overdose epidemic

You draw it: What got better or worse during Obama's presidency

They draw it!

Data through others' eyes: The impact of visualizing others' expectations on visualization interpretation

The Gyllenhaal experiment

How do you draw a circle? We analyzed 100,000 drawings to show how culture shapes our instincts

Recitation as a factor in memorizing

The power of testing memory: Basic research and implications for educational practice

Khan Academy

The instructional effect of feedback in test-like events

The critical importance of retrieval for learning

How to remember anything for forever-ish

Quantum country

Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice.

Authoring and generation of individualized patient education materials

PersaLog: Personalization of news article content

Generating personalized spatial analogies for distances and areas

How much hotter is your hometown than when you were born

Human terrain

Are you rich? This income-rank quiz might change how you see yourself

Quiz: Let us predict whether you’re a democrat or a republican

Find Out If Your Job Will Be Automated

Booze calculator: What's your drinking nationality

Click 1,000: How the pick-your-own-path episode was made

E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning

Multimedia learning in an interactive self-explaining environment: What works in the design of agent-based microworlds?

Pictorial aids for learning by doing in a multimedia geology simulation game.

A visual introduction to machine learning

Beyond guidelines: What can we learn from the visual information seeking mantra?

The eyes have it: A task by data type taxonomy for information visualizations

Information visualization and visual data mining

How the recession shaped the economy, in 255 charts

How does the eye work

Earth primer

Progressive growing of gans for improved quality, stability, and variation

A style-based generator architecture for generative adversarial networks

Why momentum really works

Colorized math equations

Image kernels

Stretchtext – hypertext note #8

On variable level-of-detail documents

Call for proposals winter/spring 2019

A UI that lets readers control how much information they see

Wikipedia Preview Card

Fluid links for informed and incremental link transitions

Reading and writing fluid hypertext narratives

Idyll: A markup language for authoring and publishing interactive articles on the web

Apparatus: A hybrid graphics editor and programming environment for creating interactive diagrams

Observable

LOOPY: a tool for thinking in systems

Webstrates: shareable dynamic media

Neural networks and deep learning

How I make explorable explanations

Explorable explanations: 4 more design patterns

Emerging and recurring data-driven storytelling techniques: Analysis of a curated collection of recent stories

Issue 01: Science & Society

The myth of the impartial machine

D3 data-driven documents

Launching the Parametric Press

Techniques for flexible responsive visualization design

A Comparative Evaluation of Animation and Small Multiples for Trend Visualization on Mobile Phones

Visualizing ranges over time on mobile phones: a task-based crowdsourced evaluation

Why we are doing fewer interactives

Capture & analysis of active reading behaviors for interactive articles on the web

In defense of interactive graphics


Weight Banding

Michael Petrov
Chelsea Voss
Ludwig Schubert
Nick Cammarata
Gabriel Goh
Chris Olah

Abstract: This article is part of the Circuits thread,  an experimental format collecting invited short articles and critical  commentary delving into the inner workings of neural networks.

![](Weight%20Banding_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

Introduction

------------

 Open up any ImageNet conv net and look at the weights in the last

layer. You'll find a uniform spatial pattern to them, dramatically

unlike anything we see elsewhere in the network. No individual weight is

 unusual, but the uniformity is so striking that when we first

discovered it we thought it must be a bug. Just as different biological

tissue types jump out as distinct under a microscope, the weights in

this final layer jump out as distinct when visualized with NMF. We call

this phenomenon *weight banding*.

Microscope slides of different tissues

Muscle tissue

Epithelial tissue

Typical layer

Layer with weight banding

NMF of weights at different layers

 and small circuits. In contrast, weight banding is an example of what

we call a "structural phenomenon," a larger-scale pattern in the

circuits and features of a neural network. Other examples of structural

 In the case of weight banding, we think of it as a structural

phenomenon because the pattern appears at the scale of an entire layer.

 In addition to describing weight banding, we'll explore when and

why it occurs. We find that there appears to be a causal link between

whether a model uses global average pooling or fully connected layers at

 the end, suggesting that weight banding is part of an algorithm for

preserving information about larger scale structure in images.

Establishing causal links like this is a step towards closing the loop

between practical decisions in training neural networks and the

phenomena we observe inside them.

Where weight banding occurs

---------------------------

 In order to see the bands, we need to visualize the spatial

structure of the weights, as shown below. We typically do this using

 Visualizing Weights. For each neuron, we take the weights connecting it

 to the previous layer. We then use NMF to reduce the number of

dimensions corresponding to channels in the previous layer down to 3

factors, which we can map to RGB channels. Since which factor is which

is arbitrary, we use a heuristic to make the mapping consistent across

neurons. This reveals a very prominent pattern of horizontalThe

 stripes aren't always perfectly horizontal - sometimes they exhibit a

slight preference for extra weight in the center of the central band, as

 seen in some examples below. stripes.

[2](#figure-2).

 These common networks have pooling operations before their fully

 connected layers and consistently show banding at their last

 convolutional layers.

86 weights hidden

InceptionV1

mixed 5b

470 weights hidden

ResNet50

block 4 unit 3

470 weights hidden

VGG19

conv5

 Interestingly, AlexNet does not exhibit this phenomenon.

[3](#figure-3).

 AlexNet does not have a pooling operation before its fully connected

 layers and does not show banding at its last convolutional

 layer.

 To make it easier to look for groups of similar weights, we

 sorted the neurons at each layer by similarity of their reduced

 forms.

160 weights hidden

AlexNet

conv5

 Unlike most modern vision models, AlexNet does not use global

average pooling. Instead, it has a fully connected layer directly

connected to its final convolutional layer, allowing it to treat

different positions differently. If one looks at the weights of this

fully connected layer, the weights strongly vary as a function of the

global y position.

 The horizontal stripes in weight banding mean that the filters

don't care about horizontal position, but are strongly encoding relative

 vertical position. Our hypothesis is that weight banding is a learned

way to preserve spatial information as it gets lost through various

pooling operations.

 In the next section, we will construct our own simplified vision

network and investigate variations on its architecture in order to

understand exactly which conditions are necessary to produce weight

banding.

What affects banding

--------------------

 We'd like to understand which architectural decisions affect

weight banding. This will involve trying out different architectures and

 seeing whether weight banding persists.

 Since we will only want to change a single architectural parameter

 at a time, we will need a consistent baseline to apply our

modifications to. Ideally, this baseline would be as simple as possible.

 We created a simplified network architecture with 6 groups of

convolutions, separated by L2 pooling layers. At the end, it has a

global average pooling operation that reduces the input to 512 values

that are then fed to a fully connected layer with 1001 outputs.

[4](#figure-4). Our simplified vision network architecture.

 This simplified network reliably produces weight banding in its last layer

 (and usually in the two preceding layers as well).

416 weights hidden

simplified model (`5b`), baseline

 In the rest of this section, we'll experiment with modifying this

architecture and its training settings and seeing if weight banding is

preserved.

### Rotating images 90 degrees

 To rule out bugs in training or some strange numerical problem, we decided

 to do a training run with the input rotated by 90 degrees. This sanity check

 yielded a very clear result showing *vertical* banding in the resulting

416 weights hidden

[6](#figure-6). simplified model (`5b`), 90º rotation

### Fully connected layer without global average pooling

 We remove the global average pooling step in our simplified model,

 allowing the fully connected layer to see all spatial positions at

once. This model did **not** exhibit weight banding, but

used 49x more parameters in the fully connected layer and overfit to the

 training set. This is pretty strong evidence that the use of aggressive

 pooling after the last convolutions in common models causes weight

banding. This result is also consistent with AlexNet not showing this

banding phenomenon (since it also does not have global average pooling).

416 weights hidden

### Average pooling along x-axis only

 We average out each row of the final convolutional layer, so that

vertical absolute position is preserved but horizontal absolute position

 is not.Since this model has 7x7 spatial

positions in the final convolutional layer, this modification increases

the number of parameters in the fully connected layer by 7x, but not the

[8](#figure-8).

416 weights hidden

 simplified model (`5a`), x-axis pooling

416 weights hidden

simplified model (`5b`), x-axis pooling

### Approaches where weight banding persisted

* Global average pooling with learned spatial masks. By applying

several different spatial masks and global average pooling, we can allow

 the model to preserve some spatial information. Intuitively, each mask

can select for a different subset of spatial positions.

 We tried experimental runs using each of 3, 5, or 16 different

masks.

 The masks that were learned corresponded to large-scale global

structure, but banding was still strongly present.

 `5b`.

* Adding a 7x7x512 mask with learned weights after `5b`. The hope was that a

 mask would help each `5b` neuron focus on the right parts of the 7x7 image

 without a convolution.

* Adding CoordConv channels to the inputs

 of `5a` and `5b`.

* Splitting the output of `5b` into 16 7x7x32 channel groups and feeding

 concatenated into the input of the final 1001-class fully connected layer.

 by VGG).

Confirming banding interventions in common architectures

--------------------------------------------------------

 In the previous section, we observed two interventions that

clearly affected weight banding: rotating the dataset by 90º and

removing the global average pooling before the fully connected layer.

 To confirm that these effects hold beyond our simplified model, we

 decided to make the same interventions to three

 common architectures (InceptionV1, ResNet50, VGG19) and train them

 from

 scratch.

With one exception, the effect holds in all three models.

#### InceptionV1

[9](#figure-9). Inception V1, layer `mixed_5c`, 5x5 convolution

86 weights hidden

baseline

86 weights hidden

90º rotation

86 weights hidden

global average pooling layer removed

#### ResNet50

[10](#figure-10). ResNet50, last 3x3 convolutional layer

470 weights hidden

baseline

470 weights hidden

90º rotation

470 weights hidden

global average pooling layer removed

#### VGG19

[11](#figure-11). VGG19, last 3x3 convolutional layer.

470 weights hidden

baseline

470 weights hidden

90º rotation

470 weights hidden

global average pooling layer removed

 The one exception is VGG19, where the removal of the pooling

operation before its set of fully connected layers did not eliminate

weight banding as expected; these weights look fairly similar to the

baseline. However, it clearly responds to rotation.

Conclusion

----------

 Once we really understand neural networks, one would expect us to

be able to leverage that understanding to design more effective neural

networks architectures. Early papers, like Zeiler et al,

 emphasized this quite strongly, but it's unclear whether there have yet

 been any significant successes in doing this. This hints at significant

 limitations in our work. It may also be a missed opportunity: it seems

likely that if interpretability was useful in advancing neural network

capabilities, it would become more integrated into other research and

get attention from a wider range of researchers.

 It's unclear whether weight banding is "good" or "bad."On

 one hand, the 90º rotation experiment shows that weight banding is a

product of the dataset and is encoding useful information into the

weights. However, if spatial information could flow through the network

in a different, more efficient way, then perhaps the channels would be

able to focus on encoding relationships between features without needing

 to track spatial positions. We don't have any

recommendation or action to take away from it. However, it is an example

 of a consistent link between architecture decisions and the resulting

trained weights. It has the right sort of flavor for something that

could inform architectural design, even if it isn't particularly

actionable itself.

 More generally, weight banding is an example of a large-scale

structure. One of the major limitations of circuits has been how

small-scale it is. We're hopeful that larger scale structures like

weight banding may help circuits form a higher-level story of neural

networks.

![](Weight%20Banding_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

Muscle Tissue: Cardiac Muscle

Epithelial Tissues: Stratified Squamous Epithelium

Deconvolution and Checkerboard Artifacts

Going Deeper with Convolutions

Deep Residual Learning for Image Recognition

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Classification with Deep Convolutional Neural Networks

An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

Visualizing and Understanding Convolutional Networks