![](images/multiple-pages.svg)

 This article is part of the [Circuits thread](/2020/circuits/), an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.

[Naturally Occurring Equivariance in Neural Networks](/2020/circuits/equivariance/)
[Curve Circuits](/2020/circuits/curve-circuits/)

Introduction
------------

Some of the neurons in vision models are features that we aren’t particularly surprised to find. [Curve detectors](https://distill.pub/2020/circuits/curve-detectors/), for example, are a pretty natural feature for a vision system to have. In fact, they had already been discovered in the animal visual cortex. It’s easy to imagine how curve detectors are built up from earlier edge detectors, and it’s easy to guess why curve detection might be useful to the rest of the neural network.

High-low frequency detectors, on the other hand, seem more surprising. They are not a feature that we would have expected *a priori* to find. Yet, when systematically characterizing the [early layers](https://distill.pub/2020/circuits/early-vision/) of InceptionV1, we found a full [fifteen neurons](https://distill.pub/2020/circuits/early-vision/#group_mixed3a_high_low_frequency) of `mixed3a` that appear to detect a high frequency pattern on one side, and a low frequency pattern on the other.

By “high frequency” and “low frequency” here, we mean [spatial frequency](https://en.wikipedia.org/wiki/Spatial_frequency) — just like when we take the
 Fourier transform of an image.

 One worry we might have about the [circuits](https://distill.pub/2020/circuits/zoom-in/) approach to studying neural networks is that we might only be able to understand a limited set of highly-intuitive features.

 High-low frequency detectors demonstrate that it’s possible to understand at least somewhat unintuitive features.

[Function
----------](#function)

 How can we be sure that “high-low frequency detectors” are actually detecting directional transitions from low to high spatial frequency?
 We will rely on three methods:

* [**Feature visualization**](#feature-visualization) allows us to establish a causal link between each neuron and its function.
* [**Dataset examples**](#dataset-examples) show us where the neuron fires in practice.
* [**Synthetic tuning curves**](#tuning-curves) show us how variation affects the neuron’s response.

 Later on in the article, we dive into the mechanistic details of how they are both [implemented](#implementation) and [used](#usage). We will be able to understand the algorithm that implements them, confirming that they detect high to low frequency transitions.

[### Feature Visualization](#feature-visualization)

 A [feature visualization](https://distill.pub/2017/feature-visualization) is a synthetic input
 optimized to elicit maximal activation of a single, specific neuron.
 Feature visualizations are constructed starting from random noise, so each and every pixel in a feature visualization
 that’s *changed* from random noise is there because it caused the neuron to activate more strongly. This
 establishes a causal link! The behavior shown in the
 feature visualization is behavior that causes the neuron to fire:

 .gallery {
 display: grid;
 grid-template-columns: repeat(auto-fit, minmax(28px, 64px));
 grid-gap: 0.5rem;
 justify-content: start;
 }

 ul.gallery {
 padding-left: 0;
 }

 .gallery img,
 .gallery-img {
 max-width: 100%;
 width: unset;
 object-fit: none;
 object-position: center;
 border-radius: 8px;
 }

 @media screen and (min-width: 768px) {
 .gallery {
 grid-template-columns: repeat(7, minmax(28px, 96px));
 justify-content: left;
 }
 }
 @media screen and (min-width: 1180px) {
 .gallery {
 grid-gap: 1rem;
 }
 }

[![](diagrams/1.1-feature-vis/neuron-136.png "Unit 136")
3a:136](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_136.html)
[![](diagrams/1.1-feature-vis/neuron-108.png "Unit 108")
3a:108](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_108.html)
[![](diagrams/1.1-feature-vis/neuron-132.png "Unit 132")
3a:132](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_132.html)
[![](diagrams/1.1-feature-vis/neuron-88.png "Unit 88")
3a:88](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_88.html)
[![](diagrams/1.1-feature-vis/neuron-110.png "Unit 110")
3a:110](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_110.html)
[![](diagrams/1.1-feature-vis/neuron-180.png "Unit 180")
3a:180](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_180.html)
[![](diagrams/1.1-feature-vis/neuron-153.png "Unit 153")
3a:153](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_153.html)
[![](diagrams/1.1-feature-vis/neuron-186.png "Unit 186")
3a:186](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_186.html)
[![](diagrams/1.1-feature-vis/neuron-86.png "Unit 86")
3a:86](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_86.html)
[![](diagrams/1.1-feature-vis/neuron-117.png "Unit 117")
3a:117](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_117.html)
[![](diagrams/1.1-feature-vis/neuron-112.png "Unit 112")
3a:112](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_112.html)
[![](diagrams/1.1-feature-vis/neuron-70.png "Unit 70")
3a:70](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_70.html)
[![](diagrams/1.1-feature-vis/neuron-106.png "Unit 106")
3a:106](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_106.html)
[![](diagrams/1.1-feature-vis/neuron-113.png "Unit 113")
3a:113](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_113.html)

[1](#figure-1):
 Feature visualizations of a variety of high-low frequency detectors from InceptionV1′s [`mixed3a`](https://microscope.openai.com/models/inceptionv1/mixed3a_0?models.op.feature_vis.type=neuron&models.op.technique=feature_vis) layer.

 From their feature visualizations, we observe that all of these high-low frequency detectors share these same
 characteristics:

* **Detection of adjacent high and low frequencies.** The detectors respond to *high frequency* on one side, and *low frequency* on the other side.
* **Rotational equivariance.**
 The detectors are rotationally [equivariant](https://distill.pub/2020/circuits/equivariance/): each unit detects a high-low frequency change along a particular angle, with different units spanning the full 360º of possible orientations.
 We will see this in more detail when we [construct a tuning curve](#tuning-curves) with synthetic examples, and also when we look at the weights [implementing](#implementation) these detectors.

We can use a [diversity term](https://distill.pub/2017/feature-visualization/#diversity) in our feature visualizations to jointly optimize for the activation of a neuron while encouraging different activation patterns in a batch of visualizations.

 We are thus reasonably confident that if high-low frequency detectors were also sensitive to other patterns, we would see signs of them in these feature visualizations. Instead, the frequency contrast remains an invariant aspect of all these visualizations. (Although other patterns form along the boundary, these are likely outside the neuron’s effective receptive field.)

![](diagrams/1.1-feature-vis/fv-mixed3a-136-diversity-0.png "Unit mixed3a:136 optimized with a diversity objective.")
![](diagrams/1.1-feature-vis/fv-mixed3a-136-diversity-1.png "Unit mixed3a:136 optimized with a diversity objective.")
![](diagrams/1.1-feature-vis/fv-mixed3a-136-diversity-2.png "Unit mixed3a:136 optimized with a diversity objective.")
![](diagrams/1.1-feature-vis/fv-mixed3a-136-diversity-3.png "Unit mixed3a:136 optimized with a diversity objective.")
![](diagrams/1.1-feature-vis/fv-mixed3a-136-diversity-4.png "Unit mixed3a:136 optimized with a diversity objective.")
![](diagrams/1.1-feature-vis/fv-mixed3a-136-diversity-5.png "Unit mixed3a:136 optimized with a diversity objective.")
![](diagrams/1.1-feature-vis/fv-mixed3a-136-diversity-6.png "Unit mixed3a:136 optimized with a diversity objective.")

[1-2](#figure-1-2):
 Feature visualizations of high-low frequency detector mixed3a:136 from InceptionV1′s [`mixed3a`](https://microscope.openai.com/models/inceptionv1/mixed3a_0?models.op.feature_vis.type=neuron&models.op.technique=feature_vis)
 layer, optimized with a diversity objective. You can learn more about feature visualization and the diversity objective [here](https://distill.pub/2017/feature-visualization/#diversity).

[### Dataset Examples](#dataset-examples)

 We generate dataset examples by sampling from a natural data distribution (in this case, the training set) and selecting the images that cause the neurons to maximally activate.

 Checking against these examples helps ensure we’re not misreading the feature visualizations.

[![](diagrams/1.1-feature-vis/neuron-136.png)](https://microscope.openai.com/models/inceptionv1/mixed3a_0/136)
![](diagrams/1.2-dataset-examples/placeholder.png)

[2](#1.2.0-dataset-examples):
 Crops taken from Imagenet where [`mixed3a` 136](https://microscope.openai.com/models/inceptionv1/mixed3a_0/136) activated maximally,
 argmaxed over spatial locations.

A wide range of real-world situations can cause high-low frequency detectors to fire. Oftentimes it’s a highly-textured, in-focus foreground object against a blurry background — for example, the foreground might be the microphone’s latticework, the hummingbird’s tiny head feathers, or the small rubber dots on the Lenovo ThinkPad [pointing stick](https://en.wikipedia.org/wiki/Pointing_stick) — but not always: we also observe that it fires for the MP3 player’s brushed metal finish against its shiny screen, or the text of a watermark.

In all cases, we see one area with high frequency and another area with low frequency. Although they often fire at an object boundary,

 they can also fire in cases where there is a frequency change without an object boundary.

 High-low frequency detectors are therefore not the same as [boundary detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_boundary).

[### Synthetic Tuning Curves](#tuning-curves)

**Tuning curves** show us how a neuron’s response changes with respect to a parameter.

 They are a standard method in neuroscience, and we’ve found them very helpful for studying artificial neural networks as well. For example, we used them to demonstrate [how the response of curve detectors changes](https://distill.pub/2020/circuits/curve-detectors/#radial-tuning-curve) with respect to orientation.

 Similarly, we can use tuning curves to show how high-low frequency detectors respond.

 To construct such a curve, we’ll need a set of *synthetic stimuli* which cause high-low frequency detectors to fire.

 We generate images with a high-frequency pattern on one side and a low-frequency pattern on the other. Since we’re interested in orientation, we’ll rotate this pattern to create a 1D family of stimuli:

![](diagrams/1.4-tuning-curves/orientation.png)

 The first axis of variation of our synthetic stimuli is *orientation*.

 But what frequency should we use for each side? How steep does the difference in frequency need to be?
 To explore this, we’ll add a second dimension varying the ratio between the two frequencies:

![](diagrams/1.4-tuning-curves/ratio.png)

 The second axis of variation of our synthetic stimuli is the *frequency ratio*.

 (Adding a second dimension will also help us see whether the results for the first dimension are robust.)

 Now that we have these two dimensions, we sample the synthetic stimuli and plot each neuron’s responses to them:

 Each high-low frequency detector exhibits a clear preference for a limited range of orientations.

 As we [previously found](https://distill.pub/2020/circuits/curve-detectors/#synthetic-curves) with curve detectors, high-low frequency detectors are rotationally [equivariant](https://distill.pub/2020/circuits/equivariance/): each one selects for a given orientation, and together they span the full 360º space.

[Implementation
--------------](#implementation)
How are high-low frequency detectors built up from lower-level neurons?

 One could imagine many different circuits which could implement this behavior. To give just one example, it seems like there are at least two different ways that the oriented nature of these units could form.

* **Equivariant→Equivariant Hypothesis.** The first possibility is that the previous layer already has precursor features which detect oriented transitions from high frequency to low frequency. The extreme version of this hypothesis would be that the high-low frequency detector is just an identity passthrough of some lower layer neuron. A more moderate version would be something like what we see with curve detectors, where [early curve detectors](https://distill.pub/2020/circuits/early-vision/#group_conv2d2_tiny_curves) become refined into the larger and more sophisticated [late curve detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3a_curves). Another example would be how edge detection is built up from simple [Gabor filters](https://distill.pub/2020/circuits/early-vision/#group_conv2d0_gabor_filters) which were already oriented.

 We call this [Equivariant→Equivariant](https://distill.pub/2020/circuits/equivariance/#equivariant-to-equivariant) because the equivariance over orientation was already there in the previous layer.
* **Invariant→Equivariant Hypothesis.** Alternatively, previous layers might not have anything like high-low frequency detectors. Instead, the orientation might come from spatial arrangements in the neuron’s weights that govern where it is excited by low-frequency and high-frequency features.

To resolve this question — and more generally, to understand how these detectors are implemented — we can look at the weights.

Let’s look at a single detector. Glancing at the weights from [`conv2d2`](https://distill.pub/2020/circuits/early-vision/#conv2d2) to [`mixed3a` 110](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_110.html), most of them can be roughly divided into two categories: those that activate on the left and inhibit on the right, and those that do the opposite.

[4](#figure-4):
 Six neurons from conv2d2 contributing weights to mixed3a 110.

 The same also holds for each of the other high-low frequency detectors — but, of course, with different spatial patternsAs an aside: The 1-2-1 pattern on each column of weights is curiously reminiscent of the structure of the [Sobel filter](https://en.wikipedia.org/wiki/Sobel_operator). on the weights, implementing the different orientations.

 Surprisingly, across all high-low frequency detectors, the two clusters of neurons that we get for each are actually the *same* two clusters! One cluster appears to detect textures with a generally high frequency, and one cluster appears to detect textures with a generally low frequency.

![](diagrams/HF-LF-clusters-amd-weight-structure.png)

[5](#figure-5):
 The strongest weights on any high-low frequency detector (here shown: [`mixed3a` 110](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_110.html), [`mixed3a` 136](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_136.html), and [`mixed3a` 112](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_112.html)) can be divided into roughly two clusters. Each cluster contributes its weights in similar ways.

 Top row: underlying neurons [`conv2d2` 119](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_119.html), [`conv2d2` 102](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_102.html), [`conv2d2` 123](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_123.html), [`conv2d2` 90](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_90.html), [`conv2d2` 89](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_89.html), [`conv2d2` 163](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_163.html), [`conv2d2` 98](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_98.html), and [`conv2d2` 188](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_188.html).

 This is exactly what we would expect to see if the Invariant→Equivariant hypothesis is true: each high-low frequency detector composes the same two components in different spatial arrangements, which then in turn govern the detector’s orientation.

 These two different clusters are really striking.

 In the next section, we’ll investigate them in more detail.

### High and Low Frequency Factors

It would be nice if we could confirm that these two clusters of neurons are real. It would also be nice if we could create a simpler way to represent them for circuit analysis later.

Factorizing the connectionsBetween two adjacent layers, “connections” reduces to the weights
 between the two layers. Sometimes we are interested in observing connectivity between layers that may not be
 directly adjacent. Because our model, a deep convnet, is non-linear, we will need to approximate the
 connections. A simple approach that we take is to linearize the model by removing the non-linearities. While
 this is not a great approximation of the model’s behavior, it does give a reasonable intuition for
 counterfactual influence: had the neurons in the intermediate layer fired, how it would have affected neurons in
 the downstream layers. We treat positive and negative influences separately. between lower layers and the high-low frequency detectors is one way that we can check whether these two clusters are meaningful, and investigate their significance. Performing a one-sided non-negative matrix factorization (NMF)We [require](/2020/circuits/visualizing-weights/#one-simple-trick) that the channel factor be positive, but allow the spatial factor to have both positive and negative values. separates the connections into two factors.

Each factor corresponds to a vector over neurons. Feature visualization can also be used to visualize these linear combinations of neurons. Strikingly, one clearly displays a generic high-frequency image, whereas the other does the same with a low-frequency image.In InceptionV1 in particular, it’s possible that we recover these two factors so crisply in part due to the *[3x3 bottleneck](https://microscope.openai.com/models/inceptionv1/mixed3a_3x3_bottleneck_0?models.op.feature_vis.type=neuron&models.op.technique=feature_vis)* between conv2d2 and mixed3a. Because of this, we’re not here looking at direct weights between conv2d2 and mixed3a, but rather the “expanded weights,” which are a product of a 1x1 convolution (which reduces down to a small number of neurons) combined with a 3x3 convolution. This structure is very similar to the factorization we apply. However, as we see later in [Universality](#universality), we recover similar factors for other models where this bottleneck doesn’t exist. NMF makes it easy to see this abstract circuit across many models which may not have an architecture that more explicitly reifies it. We’ll call these the *HF-factor* and the *LF-factor*:

 .upstream-neurons {
 display: grid;
 grid-gap: 1em;
 margin-bottom: 1em;
 }

 h5 {
 margin-bottom: 0px;
 }

 .upstream-neurons .row {
 display: grid;
 grid-template-columns: 1fr min-content 1fr min-content 1fr min-content 1fr min-content 1fr min-content 1fr min-content min-content;
 grid-column-gap: .25em;
 column-gap: .25em;
 align-items: center;
 }

 .units,
 .weights {
 display: grid;
 grid-template-columns: repeat(6, 1fr);
 grid-gap: 0.5rem;
 grid-column-start: 3;
 }

 img.fv {
 display: block;
 max-width: 100%;
 border-radius: 8px;
 }

 img.full {
 width: unset;
 object-fit: none;
 object-position: center;
 image-rendering: optimizeQuality;
 }

 img.weight {
 width: 100%;
 image-rendering: pixelated;
 align-self: center;
 border: 1px solid #ccc;
 }

 .layer-label {
 grid-row-start: span 2;
 }

 .layer-label label {
 display: inline-block;
 /\* transform: rotate(-90deg); \*/
 }

 .annotation {
 font-size: 1.5em;
 font-weight: 200;
 color: #666;
 margin-bottom: 0.2em;
 }

 .equal-sign {
 padding: 0 0.25em;
 }

 .ellipsis {
 padding: 0 0.25em;
 /\* vertically align the ellipsis \*/
 position: relative;
 bottom: 0.5ex;
 }

 .unit {
 display: block;
 min-width: 50px;
 }

 .factor {
 box-shadow: 0 0 8px #888;
 }

 .unit .bar {
 display: block;
 margin-top: 0.5em;
 background-color: #CCC;
 height: 4px;
 }

 .row h4 {
 border-bottom: 1px solid #ccc;
 }

#### mixed3a → conv2d2

![](diagrams/2.1-upstream-nmf/conv2d2-hi.png)
=
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=0-channel_index=119.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=0-channel_index=102.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=0-channel_index=123.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=0-channel_index=90.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=0-channel_index=173.png)
+
…
HF-factor

 × 0.93

 × 0.73

 × 0.66

 × 0.59

 × 0.55

![](diagrams/2.1-upstream-nmf/conv2d2-lo.png)
=
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=1-channel_index=89.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=1-channel_index=163.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=1-channel_index=98.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=1-channel_index=188.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d2-component=1-channel_index=158.png)
+
…
LF-factor

 × 0.44

 × 0.41

 × 0.38

 × 0.36

 × 0.34

#### mixed3a → conv2d1

![](diagrams/2.1-upstream-nmf/conv2d1-hi.png)
=
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=0-channel_index=30.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=0-channel_index=41.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=0-channel_index=55.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=0-channel_index=51.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=0-channel_index=16.png)
+
…
HF-factor

 × 0.86

 × 0.81

 × 0.64

 × 0.53

 × 0.52

![](diagrams/2.1-upstream-nmf/conv2d1-lo.png)
=
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=1-channel_index=4.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=1-channel_index=46.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=1-channel_index=1.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=1-channel_index=33.png)
+
![](diagrams/2.2-upstream-neurons/layer_name=conv2d1-component=1-channel_index=29.png)
+
…
LF-factor

 × 0.49

 × 0.48

 × 0.45

 × 0.43

 × 0.42

[6](#figure-6):
 NMF recovers the neurons that contribute to the two NMF factors plus the weighted amount they contribute to
 each factor. Here shown: NMF against both `conv2d2` and a deeper layer, `conv2d1`. The
 left side of the equal sign shows feature visualizations of the NMF factors.

The feature visualizations are suggestive, but how can we be sure that these factors really correspond to high and low frequency in general, rather than specific high or low frequency patterns? One thing we can do is to create synthetic stimuli again, but now plotting the responses of those two NMF factors.

 Since our factors don’t correspond to an edge, our synthetic stimuli will only have one frequency region for each stimulus. To add a second dimension and again demonstrate robustness, we also vary the rotation of that region. (The frequency texture is not exactly rotationally invariant because we construct the stimulus out of orthogonal cosine waves.)

Unlike last time, these activations now mostly ignore the image’s orientation, but are sensitive to its frequency. We can average these results over all orientations in order to produce a simple tuning curve of how each factor responds to frequency. As predicted, the HF-factor responds to high frequency and the LF-factor responds to low frequency.

[8](#figure-8):
 Tuning curve for HF-factor and LF-factor from `conv2d2` against images with synthetic frequency, averaged across orientation. Wavelength as a proportion of the full input image ranges from 1:1 to 1:10.

 Now that we’ve confirmed what these factors are, let’s look at how they’re combined into high-low frequency detectors.

### Construction of High-Low Frequency Detectors

 NMF factors the weights into both a channel factor and a spatial factor. So far, we’ve looked at the two parts of the channel factor. The spatial factor shows the spatial weighting that combines the HF and LF factors into high-low frequency detectors.

 Unsurprisingly, these weights basically reproduce the same pattern that we’d previously been seeing in [Figure 5](#figure-5) from its two different clusters of neurons: where the HF-factor inhibits, the LF-factor activates — and vice versa.

 As an aside, the HF-factor here for InceptionV1 (as well as some of its NMF components, like [`conv2d2` 123](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_123.html)) also appears to be lightly activated by bright greens and magentas. This might be responsible for the feature visualizations of these high-low frequency detectors showing only greens and magentas on the high-frequency side.

 .upstream-nmf {
 display: grid;
 grid-row-gap: 2rem;
 margin-bottom: 2rem;
 }

 .upstream-nmf .row {
 display: grid;
 grid-template-columns: min-content 1fr 6fr;
 grid-column-gap: 1rem;
 grid-row-gap: .5rem;

 }

 .units,
 .weights {
 display: grid;
 grid-template-columns: repeat(6, 1fr);
 grid-gap: 0.5rem;
 grid-column-start: 3;
 }

 img.fv {
 max-width: 100%;
 border-radius: 8px;
 }

 div.units img.full {
 margin-left: 1px;
 }

 img.full {
 width: unset;
 object-fit: none;
 object-position: center;
 image-rendering: optimizeQuality;
 }

 img.weight {
 width: 100%;
 image-rendering: pixelated;
 align-self: center;
 border: 1px solid #ccc;
 }

 .annotated-image {
 display: grid;
 grid-auto-flow: column;
 align-items: center;
 }

 .annotated-image span {
 writing-mode: vertical-lr;
 }

 .layer-label {
 grid-row-start: span 2;
 border-right: 1px solid #aaa;
 text-align: end;
 }

 .layer-label label {
 display: inline-block;
 margin-right: .5em;
 writing-mode: vertical-lr;
 }

 .layer-label.hidden {
 border-color: transparent;
 }

 .layer-label.hidden label {
 visibility: hidden;
 }

mixed3a

![](diagrams/1.1-feature-vis/neuron-136.png "Unit 136")
![](diagrams/1.1-feature-vis/neuron-108.png "Unit 108")
![](diagrams/1.1-feature-vis/neuron-132.png "Unit 132")
![](diagrams/1.1-feature-vis/neuron-88.png "Unit 88")
![](diagrams/1.1-feature-vis/neuron-110.png "Unit 110")
![](diagrams/1.1-feature-vis/neuron-180.png "Unit 180")

mixed3a → conv2d2

HF-factor
![](diagrams/2.1-upstream-nmf/conv2d2-hi.png)

![](diagrams/2.1-upstream-nmf/neuron=136-layer=maxpool1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=108-layer=maxpool1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=132-layer=maxpool1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=88-layer=maxpool1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=110-layer=maxpool1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=180-layer=maxpool1-factor=1.png)

LF-factor
![](diagrams/2.1-upstream-nmf/conv2d2-lo.png)

![](diagrams/2.1-upstream-nmf/neuron=136-layer=maxpool1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=108-layer=maxpool1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=132-layer=maxpool1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=88-layer=maxpool1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=110-layer=maxpool1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=180-layer=maxpool1-factor=0.png)

mixed3a → conv2d1

HF-factor
![](diagrams/2.1-upstream-nmf/conv2d1-hi.png)

![](diagrams/2.1-upstream-nmf/neuron=136-layer=conv2d1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=108-layer=conv2d1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=132-layer=conv2d1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=88-layer=conv2d1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=110-layer=conv2d1-factor=1.png)
![](diagrams/2.1-upstream-nmf/neuron=180-layer=conv2d1-factor=1.png)

LF-factor
![](diagrams/2.1-upstream-nmf/conv2d1-lo.png)

![](diagrams/2.1-upstream-nmf/neuron=136-layer=conv2d1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=108-layer=conv2d1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=132-layer=conv2d1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=88-layer=conv2d1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=110-layer=conv2d1-factor=0.png)
![](diagrams/2.1-upstream-nmf/neuron=180-layer=conv2d1-factor=0.png)

[9](#figure-9):
 Using NMF factorization on the weights connecting six high-low frequency detectors in InceptionV1 to the
 two directly
 preceding convolutional layers, `conv2d2` and `conv2d1`.

Their spatial arrangement is very clear, with LF factors activating
 areas in which high-low frequency detectors expect low frequencies, and inhibiting areas in which they expect high frequencies. The two
 factors
 are very close to symmetric. Weight magnitudes normalized between -1  and 1.

High-low frequency detectors are therefore built up by circuits that arrange high frequency detection on one side and low frequency detection on the other.

 There are some exceptions that aren’t fully captured by the NMF factorization perspective. For example, [`conv2d2` 181](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/conv2d2_181.html) is a texture contrast detector that appears to already have spatial structure.

 This is the kind of feature that we would expect to be involved through an Equivariant→Equivariant circuit.

 If that were the case, however, we would expect its weights to the high-low frequency detector [`mixed3a` 70](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3a_70.html) to be a solid positive stripe down the middle.

 What we instead observe is that it contributes as a component of high frequency detection, though perhaps with a slight positive overall bias.

 Although `conv2d2` 181 has a spatial structure, perhaps it responds more strongly to high frequency patterns.

![](diagrams/2d2-181-3a-70-weight.png)

 The weights from `conv2d2` 181 to `mixed3a` 70 are consistent with `conv2d2` 181 contributing via the HF-factor, not via the existing spatial structure of its texture contrast detection.

Now that we understand how they are constructed, how are high-low frequency detectors used by higher-level features?

[Usage
-----](#usage)

[`mixed3b`](https://distill.pub/2020/circuits/early-vision/#mixed3b) is the next layer immediately after the high-low frequency detectors. Here, high-low frequency detectors contribute to a variety of features. Their most important role seems to be supporting [boundary detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_boundary), but they also contribute to [bumps](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_bumps) and [divots](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_divots), [line-like](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_bar_line_like) and [curve-like](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_curves_misc.) shapes,
 and at least one each of [center-surrounds](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_281.html), [patterns](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_372.html), and [textures](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_276.html).

![](diagrams/usage-1.png)

[10](#figure-10):
 Examples of neurons that high-low frequency detectors contribute to: (1) [`mixed3b` 345](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_345.html) (a boundary detector), (2) [`mixed3b` 276](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_276.html) (a center-surround texture detector), (3) [`mixed3b` 314](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_314.html) (a double boundary detector), and (4) [`mixed3b` 365](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_365.html) (an hourglass shape detector).

 These aren’t the only contributors to these neurons – for example, `mixed3b` 276 also relies heavily on certain center-surrounds and textures – but they are strong contributors.

 Oftentimes, downstream features appear to ignore the “polarity” of a high-low frequency detector, responding roughly the same way regardless of which side is high frequency. For example, the vertical boundary detector [`mixed3b` 345](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_345.html) (see above) is strongly excited by high-low frequency detectors that detect frequency change across a vertical line in either direction.

 Whereas activation from a high-low frequency detector can help detect boundaries between different objects, inhibition from a high-low frequency detector can also add structure to an object detector by detecting regions that must be contiguous along some direction — essentially, indicating the absence of a boundary.

![](diagrams/usage-2.png)

[11](#figure-11):
 Some of [`mixed3b` 314](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_314.html)’s weights, extracted for emphasis. Orientation doesn’t matter so much for how these weights are used by [`mixed3b` 314](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_314.html), but their 180º-invariant orientation does!

You may notice that strong excitation (left) is correlated with the presence of a **boundary** at a particular angle, whereas strong inhibition (right) is correlated with **object continuity** where a boundary might otherwise have been.

As we’ve mentioned, by far the primary downstream contribution of high-low frequency detectors is to *boundary detectors*. Of the top 20 neurons in `mixed3b` with the highest L2-norm of weights across all high-low frequency detectors, eight of those 20 neurons participate in boundary detection of some sort: [double boundary detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_double_boundary), [miscellaneous boundary detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_boundary_misc), and especially [object boundary detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_boundary).

#### Role in object boundary detection

[Object boundary detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_boundary) are neurons which detect boundaries between objects, whether that means the boundary between one object and another or the transition from foreground to background. They are different from edge detectors or curve detectors: although they are sensitive to edges (indeed, some of their strongest weights are contributed by lower-level edge detectors!), object boundary detectors are also sensitive to other indicators such as color contrast and high-low frequency detection.

![](diagrams/usage-boundary.png)

[12](#figure-12): `mixed3b` 345 is a boundary detector activated by high-low frequency detectors, edges, color contrasts, and end-of-line
 detectors. It is specifically sensitive to vertically-oriented high-low frequency detectors, regardless of their
 orientation, and along a vertical line of positive weights.

 High-low frequency detectors contribute to these object boundary detectors by providing one piece of evidence that an object has ended and something else has begun. Some examples of object boundary detectors are shown below, along with their weights to a selection of high-low frequency detectors, grouped by orientation (ignoring polarity).

In particular, note how similar the weights are within each grouping! This shows us again that the later layers ignore the high-low frequency detectors’ polarity. Furthermore, the arrangement of excitatory and inhibitory weights contributes to each boundary detector’s overall shape, following the principles outlined above.

 /\* TODO: Optimize smaller breakpoints by hand \*/

![](diagrams/usage-boundaries.png)

[13](#figure-13): Four examples of object boundary detectors that high-low frequency detectors contribute to: [`mixed3b` 345](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_345.html), [`mixed3b` 376](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_376.html), [`mixed3b` 368](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_368.html), and [`mixed3b` 151](https://storage.googleapis.com/distill-circuits/inceptionv1-weight-explorer/mixed3b_151.html).

 Beyond `mixed3b`, high-low frequency detectors ultimately play a role in detecting more sophisticated object shapes in `mixed4a` and beyond, by continuing to contribute to the detection of boundaries and contiguity.

 So far, the scope of our investigation has been limited to InceptionV1.

 How common are high-low frequency detectors in convolutional neural networks generally?

Universality
------------

### High-Low Frequency Detectors in Other Networks

It’s always good to ask if what we see is the rule or an interesting exception — and high-low frequency detectors seem to be the rule.
 High-low frequency detectors similar to ones in InceptionV1 can be found in a variety of architectures.

 strong d-cite {
 font-weight: normal;
 }

 figure.columns {
 display: grid;
 grid-gap: 0.5em;
 grid-template-columns: repeat(4, 1fr);
 overflow: hidden;
 }

 .columns .model {
 display: grid;
 /\* position: relative;
 grid-row-gap: 0.5em;
 grid-auto-flow: row;
 align-content: start;
 grid-template-rows: min-content; \*/
 }

 .model picture {
 position: relative;
 }

 .model img {
 box-sizing: border-box;
 position: relative;
 width: unset;
 object-fit: none;
 z-index: 1;
 outline: 6px solid white;
 border-radius: 8px;
 }

 .model figcaption>\* {
 display: block;
 }

 /\*
 HAAAACKY
 \*/
 .model:first-of-type picture::before {
 content: "";
 display: block;
 position: absolute;
 height: 1px;
 width: 350%;
 left: 0;
 top: 48px;
 background-color: #aaa;
 z-index: 0;
 }

 .columns>figcaption {
 grid-column: 1 / span 4;
 }

**InceptionV1**
Layer `mixed3a`
At ~33% CNN depth

![](diagrams/1.1-feature-vis/neuron-106.png "InceptionV1's layer mixed3a, Unit 106")

![](diagrams/1.1-feature-vis/neuron-110.png "InceptionV1's layer mixed3a, Unit 110")

![](diagrams/1.1-feature-vis/neuron-180.png "InceptionV1's layer mixed3a, Unit 180")

![](diagrams/1.1-feature-vis/neuron-132.png "InceptionV1's layer mixed3a, Unit 132")

![](diagrams/1.1-feature-vis/neuron-112.png "InceptionV1's layer mixed3a, Unit 112")

![](diagrams/1.1-feature-vis/neuron-108.png "InceptionV1's layer mixed3a, Unit 108")

**AlexNet**
Layer `Conv2D_2`
At ~29% CNN depth

![](diagrams/4.1-similar-directions/alexnet/Conv2D_2-183.png "AlexNet's layer Conv2D_2, Unit 183")

![](diagrams/4.1-similar-directions/alexnet/Conv2D_2-175.png "AlexNet's layer Conv2D_2, Unit 175")

![](diagrams/4.1-similar-directions/alexnet/Conv2D_2-215.png "AlexNet's layer Conv2D_2, Unit 215")

![](diagrams/4.1-similar-directions/alexnet/Conv2D_2-230.png "AlexNet's layer Conv2D_2, Unit 230")

![](diagrams/4.1-similar-directions/alexnet/Conv2D_2-161.png "AlexNet's layer Conv2D_2, Unit 161")

![](diagrams/4.1-similar-directions/alexnet/Conv2D_2-205.png "AlexNet's layer Conv2D_2, Unit 205")

**InceptionV4**
Layer `Mixed_5a`
At ~33% CNN depth

![](diagrams/4.1-similar-directions/inceptionv4/channel-82.png "InceptionV4's layer Mixed_5a, Unit 82")

![](diagrams/4.1-similar-directions/inceptionv4/channel-178.png "InceptionV4's layer Mixed_5a, Unit 178")

![](diagrams/4.1-similar-directions/inceptionv4/channel-77.png "InceptionV4's layer Mixed_5a, Unit 77")

![](diagrams/4.1-similar-directions/inceptionv4/channel-18.png "InceptionV4's layer Mixed_5a, Unit 18")

![](diagrams/4.1-similar-directions/inceptionv4/channel-52.png "InceptionV4's layer Mixed_5a, Unit 52")

![](diagrams/4.1-similar-directions/inceptionv4/channel-76.png "InceptionV4's layer Mixed_5a, Unit 76")

**ResNetV2-50**
Layer `B2_U1_conv2`
At ~29% CNN depth

![](diagrams/4.1-similar-directions/resnetv2/channel-118.png "ResNetV2-50's layer B2_U1_conv2, Unit 118")

![](diagrams/4.1-similar-directions/resnetv2/channel-41.png "ResNetV2-50's layer B2_U1_conv2, Unit 41")

![](diagrams/4.1-similar-directions/resnetv2/channel-58.png "ResNetV2-50's layer B2_U1_conv2, Unit 58")

![](diagrams/4.1-similar-directions/resnetv2/channel-50.png "ResNetV2-50's layer B2_U1_conv2, Unit 50")

![](diagrams/4.1-similar-directions/resnetv2/channel-45.png "ResNetV2-50's layer B2_U1_conv2, Unit 45")

![](diagrams/4.1-similar-directions/resnetv2/channel-53.png "ResNetV2-50's layer B2_U1_conv2, Unit 53")

[14](#figure-14). High-low frequency detectors that we’ve found in AlexNet, InceptionV4, and ResnetV2-50 (right), compared to their most similar counterpart from InceptionV1 (left). These are individual neurons, not linear combinations approximating the detectors in InceptionV1.

Notice that these detectors are found at very similar depths within the different networks, between 29% and 33% network depth!Network depth is here defined as the index of the layer divided by the total number of layers. While the particular orientations each network’s high-low frequency detectors respond to may vary slightly, each network has its own family of detectors that together cover the full 360º and comprise a rotationally [equivariant](https://distill.pub/2020/circuits/equivariance/) family.
 Architecture aside – what about networks trained on substantially different *datasets*? In the extreme case, one could imagine a synthetic dataset where high-low frequency detectors don’t arise. For most practical datasets, however, we expect to find them. For example, we even find some candidate high-low frequency detectors in AlexNet (Places): [down-up](https://microscope.openai.com/models/alexnet_caffe_places365/conv2_Conv2D_0/37), [left-right](https://microscope.openai.com/models/alexnet_caffe_places365/conv2_Conv2D_0/91), and [up-down](https://microscope.openai.com/models/alexnet_caffe_places365/conv2_Conv2D_0/104).

Even though these families are from three completely different networks, we also discover that their high-low frequency detectors are built up from high and low frequency components.

### HF-factor and LF-factor in Other Networks

 As we did with InceptionV1, we can again perform NMF on the weights of the high-low frequency detectors in each network in order to extract the strongest two factors.

 .upstream-nmf {
 display: grid;
 grid-row-gap: 2rem;
 margin-bottom: 3rem;
 }

 .upstream-nmf .row {
 display: grid;
 grid-template-columns: min-content 1fr 6fr;
 grid-column-gap: 1rem;
 grid-row-gap: .5rem;

 }

 .units,
 .weights {
 display: grid;
 grid-template-columns: repeat(6, 1fr);
 grid-gap: 0.5rem;
 grid-column-start: 3;
 }

 img.fv {
 max-width: 100%;
 border-radius: 8px;
 }

 div.units img.full {
 margin-left: 1px;
 }

 img.full {
 width: unset;
 object-fit: none;
 object-position: center;
 image-rendering: optimizeQuality;
 }

 img.weight {
 width: 100%;
 image-rendering: pixelated;
 align-self: center;
 border: 1px solid #ccc;
 }

 img.factor {
 /\* padding-right: 0.75rem; \*/
 }

 .annotated-image {
 display: grid;
 grid-auto-flow: column;
 align-items: center;
 }

 .annotated-image span {
 writing-mode: vertical-lr;
 }

 .layer-label {
 grid-row-start: span 2;
 border-right: 1px solid #aaa;
 text-align: end;
 }

 .layer-label label {
 display: inline-block;
 margin-right: .5em;
 writing-mode: vertical-lr;

 }

 .layer-label.hidden {
 border-color: transparent;
 }

 .layer-label.hidden label {
 visibility: hidden;
 }

####  AlexNet

layer

[![](diagrams/6.0-universality/AlexNet-0.png "Unit 55")](https://microscope.openai.com/models/alexnet/Conv2D_2_0/55)[![](diagrams/6.0-universality/AlexNet-1.png "Unit 47")](https://microscope.openai.com/models/alexnet/Conv2D_2_0/47)[![](diagrams/6.0-universality/AlexNet-2.png "Unit 87")](https://microscope.openai.com/models/alexnet/Conv2D_2_0/87)[![](diagrams/6.0-universality/AlexNet-3.png "Unit 33")](https://microscope.openai.com/models/alexnet/Conv2D_2_0/33)[![](diagrams/6.0-universality/AlexNet-4.png "Unit 77")](https://microscope.openai.com/models/alexnet/Conv2D_2_0/77)[![](diagrams/6.0-universality/AlexNet-5.png "Unit 102")](https://microscope.openai.com/models/alexnet/Conv2D_2_0/102)

Conv2D\_2 → conv1\_1

HF-factor
![](diagrams/6.0-universality/AlexNet-hi.png)

![](diagrams/6.0-universality/AlexNet-0-hi.png "Unit 55")
![](diagrams/6.0-universality/AlexNet-1-hi.png "Unit 47")
![](diagrams/6.0-universality/AlexNet-2-hi.png "Unit 87")
![](diagrams/6.0-universality/AlexNet-3-hi.png "Unit 33")
![](diagrams/6.0-universality/AlexNet-4-hi.png "Unit 77")
![](diagrams/6.0-universality/AlexNet-5-hi.png "Unit 102")

LF-factor
![](diagrams/6.0-universality/AlexNet-lo.png)

![](diagrams/6.0-universality/AlexNet-0-lo.png "Unit 55")
![](diagrams/6.0-universality/AlexNet-1-lo.png "Unit 47")
![](diagrams/6.0-universality/AlexNet-2-lo.png "Unit 87")
![](diagrams/6.0-universality/AlexNet-3-lo.png "Unit 33")
![](diagrams/6.0-universality/AlexNet-4-lo.png "Unit 77")
![](diagrams/6.0-universality/AlexNet-5-lo.png "Unit 102")

####  InceptionV3\_slim

layer

[![](diagrams/6.0-universality/InceptionV3_slim-0.png "Unit 82")](https://microscope.openai.com/models/inceptionv3_slim/InceptionV3_InceptionV3_Conv2d_4a_3x3_Relu_0/82)[![](diagrams/6.0-universality/InceptionV3_slim-1.png "Unit 83")](https://microscope.openai.com/models/inceptionv3_slim/InceptionV3_InceptionV3_Conv2d_4a_3x3_Relu_0/83)[![](diagrams/6.0-universality/InceptionV3_slim-2.png "Unit 137")](https://microscope.openai.com/models/inceptionv3_slim/InceptionV3_InceptionV3_Conv2d_4a_3x3_Relu_0/137)[![](diagrams/6.0-universality/InceptionV3_slim-3.png "Unit 139")](https://microscope.openai.com/models/inceptionv3_slim/InceptionV3_InceptionV3_Conv2d_4a_3x3_Relu_0/139)[![](diagrams/6.0-universality/InceptionV3_slim-4.png "Unit 155")](https://microscope.openai.com/models/inceptionv3_slim/InceptionV3_InceptionV3_Conv2d_4a_3x3_Relu_0/155)[![](diagrams/6.0-universality/InceptionV3_slim-5.png "Unit 159")](https://microscope.openai.com/models/inceptionv3_slim/InceptionV3_InceptionV3_Conv2d_4a_3x3_Relu_0/159)

Conv2d\_4a → Conv2d\_3b

HF-factor
![](diagrams/6.0-universality/InceptionV3_slim-hi.png)

![](diagrams/6.0-universality/InceptionV3_slim-0-hi.png "Unit 82")
![](diagrams/6.0-universality/InceptionV3_slim-1-hi.png "Unit 83")
![](diagrams/6.0-universality/InceptionV3_slim-2-hi.png "Unit 137")
![](diagrams/6.0-universality/InceptionV3_slim-3-hi.png "Unit 139")
![](diagrams/6.0-universality/InceptionV3_slim-4-hi.png "Unit 155")
![](diagrams/6.0-universality/InceptionV3_slim-5-hi.png "Unit 159")

LF-factor
![](diagrams/6.0-universality/InceptionV3_slim-lo.png)

![](diagrams/6.0-universality/InceptionV3_slim-0-lo.png "Unit 82")
![](diagrams/6.0-universality/InceptionV3_slim-1-lo.png "Unit 83")
![](diagrams/6.0-universality/InceptionV3_slim-2-lo.png "Unit 137")
![](diagrams/6.0-universality/InceptionV3_slim-3-lo.png "Unit 139")
![](diagrams/6.0-universality/InceptionV3_slim-4-lo.png "Unit 155")
![](diagrams/6.0-universality/InceptionV3_slim-5-lo.png "Unit 159")

####  ResnetV2\_50\_slim

layer

[![](diagrams/6.0-universality/ResnetV2_50_slim-0.png "Unit 118")](https://microscope.openai.com/models/resnetv2_50_slim/resnet_v2_50_block2_unit_1_bottleneck_v2_conv2_Relu_0/118)[![](diagrams/6.0-universality/ResnetV2_50_slim-1.png "Unit 41")](https://microscope.openai.com/models/resnetv2_50_slim/resnet_v2_50_block2_unit_1_bottleneck_v2_conv2_Relu_0/41)[![](diagrams/6.0-universality/ResnetV2_50_slim-2.png "Unit 53")](https://microscope.openai.com/models/resnetv2_50_slim/resnet_v2_50_block2_unit_1_bottleneck_v2_conv2_Relu_0/53)[![](diagrams/6.0-universality/ResnetV2_50_slim-3.png "Unit 44")](https://microscope.openai.com/models/resnetv2_50_slim/resnet_v2_50_block2_unit_1_bottleneck_v2_conv2_Relu_0/44)[![](diagrams/6.0-universality/ResnetV2_50_slim-4.png "Unit 25")](https://microscope.openai.com/models/resnetv2_50_slim/resnet_v2_50_block2_unit_1_bottleneck_v2_conv2_Relu_0/25)[![](diagrams/6.0-universality/ResnetV2_50_slim-5.png "Unit 50")](https://microscope.openai.com/models/resnetv2_50_slim/resnet_v2_50_block2_unit_1_bottleneck_v2_conv2_Relu_0/50)

B2\_U1\_conv2 → B2\_U1\_conv1

HF-factor
![](diagrams/6.0-universality/ResnetV2_50_slim-hi.png)

![](diagrams/6.0-universality/ResnetV2_50_slim-0-hi.png "Unit 118")
![](diagrams/6.0-universality/ResnetV2_50_slim-1-hi.png "Unit 41")
![](diagrams/6.0-universality/ResnetV2_50_slim-2-hi.png "Unit 53")
![](diagrams/6.0-universality/ResnetV2_50_slim-3-hi.png "Unit 44")
![](diagrams/6.0-universality/ResnetV2_50_slim-4-hi.png "Unit 25")
![](diagrams/6.0-universality/ResnetV2_50_slim-5-hi.png "Unit 50")

LF-factor
![](diagrams/6.0-universality/ResnetV2_50_slim-lo.png)

![](diagrams/6.0-universality/ResnetV2_50_slim-0-lo.png "Unit 118")
![](diagrams/6.0-universality/ResnetV2_50_slim-1-lo.png "Unit 41")
![](diagrams/6.0-universality/ResnetV2_50_slim-2-lo.png "Unit 53")
![](diagrams/6.0-universality/ResnetV2_50_slim-3-lo.png "Unit 44")
![](diagrams/6.0-universality/ResnetV2_50_slim-4-lo.png "Unit 25")
![](diagrams/6.0-universality/ResnetV2_50_slim-5-lo.png "Unit 50")

[15](#figure-15):
 NMF of high-low frequency detectors in

 AlexNet’s
 [Conv2D\_2](https://microscope.openai.com/models/alexnet/Conv2D_2_0?models.op.feature_vis.type=channel&models.op.technique=feature_vis) with respect to [conv1\_1](https://microscope.openai.com/models/alexnet/conv1_1_0?models.op.feature_vis.type=channel&models.op.technique=feature_vis),

 InceptionV3\_slim’s
 [Conv2d\_4a](https://microscope.openai.com/models/inceptionv3_slim/InceptionV3_InceptionV3_Conv2d_4a_3x3_Relu_0?models.op.feature_vis.type=channel&models.op.technique=feature_vis) with respect to [Conv2d\_3b](https://microscope.openai.com/models/inceptionv3_slim/InceptionV3_InceptionV3_Conv2d_3b_1x1_Relu_0?models.op.feature_vis.type=channel&models.op.technique=feature_vis),

 and
 ResnetV2\_50\_slim’s
 [B2\_U1\_conv2](https://microscope.openai.com/models/resnetv2_50_slim/resnet_v2_50_block2_unit_1_bottleneck_v2_conv2_Relu_0?models.op.feature_vis.type=channel&models.op.technique=feature_vis) with respect to [B2\_U1\_conv1](https://microscope.openai.com/models/resnetv2_50_slim/resnet_v2_50_block2_unit_2_bottleneck_v2_conv1_Relu_0?models.op.feature_vis.type=channel&models.op.technique=feature_vis),

 showing activations and inhibitions.

The feature visualizations of the two factors reveal one clear HF-factor and one clear LF-factor, just like what we found in InceptionV1. Furthermore, the weights on the two factors are again very close to symmetric.

Our earlier conclusions therefore also hold across these different networks: high-low frequency detectors are built up from the specific spatial arrangement of a high frequency component and a low frequency component.

Conclusion
----------

Although high-low frequency detectors represent a feature that we didn’t necessarily expect to find in a neural network, we find that we can still explore and understand them using the interpretability tools we’ve built up for exploring circuits: NMF, feature visualization, synthetic stimuli, and more.

We’ve also learned that high-low frequency detectors are built up from comprehensible lower-level parts, and we’ve shown how they contribute to later, higher-level features.

 Finally, we’ve seen that high-low frequency detectors are common across multiple network architectures.

Given the universality observations, we might wonder whether the existence of high-low frequency detectors isn’t so unnatural after all. We even find [approximate](https://microscope.openai.com/models/alexnet_caffe_places365/conv2_Conv2D_0/91) high-low frequency detectors in AlexNet Places, with its substantially different training data. Beyond neural networks, the aesthetic quality imparted by the blurriness of an out-of-focus region of an image is already known as to photographers as [*bokeh*](https://en.wikipedia.org/wiki/Bokeh). And in VR, visual blur can either provide an effective depth-of-field cue or, conversely, can induce nausea in the user when implemented in a dissonant way. Perhaps frequency detection might well be commonplace in both natural and artificial vision systems as yet another type of informational cue.

Nevertheless, whether their existence is natural or not, we find that high-low frequency detectors are possible to characterize and understand.

![](images/multiple-pages.svg)

 This article is part of the Circuits thread, a collection of short articles and commentary by an open scientific collaboration delving into the inner workings of neural networks.

[Naturally Occurring Equivariance in Neural Networks](/2020/circuits/equivariance/)
[Curve Circuits](/2020/circuits/curve-circuits/)

High-Low Frequency Detectors

Ludwig Schubert
Chelsea Voss
Nick Cammarata
Gabriel Goh
Chris Olah

Summary: A family of early-vision neurons reacting to directional transitions from high to low spatial frequency.

Discrete neural clusters encode orientation, curvature and corners in macaque V4

Going deeper with convolutions

Zoom In: An Introduction to Circuits

Feature Visualization

ImageNet: A large-scale hierarchical image database

Convergent learning: Do different neural networks learn the same representations?

ImageNet Classification with Deep Convolutional Neural Networks

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Deep Residual Learning for Image Recognition


![](images/multiple-pages.svg)

 This article is part of the [Circuits thread](/2020/circuits/), an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.

[Curve Detectors](/2020/circuits/curve-detectors/)
[High-Low Frequency Detectors](/2020/circuits/frequency-edges/)

### Contents

[Equivariant Features](#features)
* [Rotation](#rotation-features),
 [Scale](#scale-features),
 [Hue](#hue-features),
 [Hue-Rotation](#hue-rotation-features),
 [Reflection](#reflection-features),
 [Miscellaneous](#miscellaneous-features)

Equivariant Circuits
* [High-Low](#circuit-hilo),
 [Contrast→Center](#circuit-greenpurple),
 [BW-Color](#circuit-BW),
 [Line→Circle/Divergence](#circuit-circlestar),
 [Curve→Circle/Evolute](#circuit-circle-evolute),
 [Human-Animal](#circuit-person),
 [Invariant Dog Head](#circuit-dog),
 [Hue→Hue](#circuit-hue),
 [Curve→Curve](#circuit-curve),
 [Contrast→Line](#circuit-coloredge)

[Equivariant Architectures](#architectures)
[Conclusion](#conclusion)

Convolutional neural networks contain a hidden world of symmetries within themselves. This symmetry is a powerful tool in understanding the features and circuits inside neural networks. It also suggests that efforts to design neural networks with additional symmetries baked in (eg. ) may be on a promising track.

To see these symmetries, we need to look at the individual neurons inside convolutional neural networks and the [circuits](https://distill.pub/2020/circuits/zoom-in/#claim-2) that connect them.
 It turns out that many neurons are slightly transformed versions of the same basic feature.
 This includes rotated copies of the same feature, scaled copies, flipped copies, features detecting different colors, and much more.
 We sometimes call this phenomenon “[equivariance](https://en.wikipedia.org/wiki/Equivariant_map),” since it means that switching the neurons is equivalent to transforming the input.
 The standard definition of [equivariance](https://en.wikipedia.org/wiki/Equivariant_map) in group theory is that a function fff is equivariant if for all g∈Gg\in Gg∈G, it’s the case that f(g⋅x)=g⋅f(x)f(g\cdot x) = g\cdot f(x)f(g⋅x)=g⋅f(x). At first blush, this doesn’t seem very relevant to transformed versions of neurons.

 Before we talk about the examples introduced in this article, let’s talk about how this definition maps to the classic example of equivariance in neural networks: translation and convolutional neural network nets. In a conv net, translating the input image is equivalent to translating the neurons in the hidden layers (ignoring pooling, striding, etc). Formally, g∈Z2g\in Z^2g∈Z2 and fff maps images to hidden layer activations. Then ggg acts on the input image xxx by translating spatially, and acts on the activations by also spatially translating them.

 Now let’s consider the case of curve detectors (the first example in the Equivariant Features section), which have ten rotated copies. In this case, g∈Z10g\in Z\_{10}g∈Z10 and f(x)=(curve1(x),...)f(x) = (\mathrm{curve}\_1(x), …)f(x)=(curve1(x),...) maps a position at an image to a ten dimensional vector describing how much each curve detector fires. Then ggg acts on the input image xxx by rotating it around that position and ggg acts on the hidden layers by reorganizing the neurons so that their orientations correspond to the appropriate rotations. This satisfies, at least approximately, the original definition of equivariance.

 This transformed neuron form of equivariance is a special case of equivariance. There are many ways a neural network could be equivariant without having transformed versions of neurons. Conversely, we’ll also see a number of examples of equivariance that don’t map exactly to the group theory definition of equivariance: some have “holes” where a transformed neuron is missing, while others consist of a set of transformations that have a weaker structure than a group or don’t correspond to a simple action on the image. But this general structure remains.

Equivariance can be seen as a kind of ”[circuit
 motif](https://distill.pub/2020/circuits/zoom-in/#claim-2-motifs),” an abstract recurring pattern across circuits analogous to motifs in systems biology .
 It can also be seen as a kind of larger-scale “structural phenomenon” (similar to [weight banding](/2020/circuits/weight-banding/) and [branch
 specialization](https://distill.pub/2020/circuits/branch-specialization/)), since a given equivariance type is often widespread in some layers and rare in others.

 In this article, we’ll focus on examples of equivariance in InceptionV1  trained
 on ImageNet, but we’ve observed at least some equivariance in every model
 trained on natural images we’ve studied.

---

Equivariant Features
--------------------

**Rotational Equivariance:** One example of equivariance is rotated versions of the same feature. These are especially common in [early vision](https://distill.pub/2020/circuits/early-vision/), for example [curve detectors](https://distill.pub/2020/circuits/curve-detectors/),  [high-low frequency detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3a_high_low_frequency), and [line detectors](https://distill.pub/2020/circuits/early-vision/#group_conv2d2_line).

Rotational Equivariance

Some rotationally equivariant features wrap around at 180 degrees due to symmetry.

(There are even units which wrap around at 90 degrees, such as hatch texture detectors.)

Curve Detectors

Rotational Equivariance (mod 180)

High-Low Frequency Detectors
Edge Detectors
Line Detectors

One can test that these are genuinely rotated versions of the same feature by taking examples that cause one to fire, rotating them, and checking that the others fire as expected. The [article](https://distill.pub/2020/circuits/curve-detectors/) on curve detectors tests their equivariance through several experiments, including rotating stimuli that activate one neuron and seeing how the others respond.

[![](images/curve-rotate-small.png)](https://distill.pub/2020/circuits/curve-detectors/#joint-tuning-curves)
One way to verify that units like curve detectors are truly rotated versions of the same feature is to take stimuli that activate one and see how they fire as you rotate the stimuli. [Learn more.](https://distill.pub/2020/circuits/curve-detectors/#joint-tuning-curves)

**Scale Equivariance:** Rotated versions aren’t the only kind of variation we see. It’s also quite common to see the same feature at different scales, although usually the scaled features occur at different layers. For example, we see [circle detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_circles_loops) across a huge variety of scales, with the small ones in early layers and the large ones in later layers.

Scale Equivariance
Circle Detectors

**Hue Equivariance:** For color-detecting features, we often see variants detecting the same thing in different hues. For example, [color center-surround](https://distill.pub/2020/circuits/early-vision/#group_mixed3a_color_center_surround) units will detect one hue in the center, and the opposing hue on around it. Units can be found doing this up until the seventh or even eighth layer of InceptionV1.

Hue Equivariance
Color Center
-Surround

**Hue-Rotation Equivariance:** In early vision, we very often see [color contrast units](https://distill.pub/2020/circuits/early-vision/#group_conv2d2_color_contrast). These units detect one hue on one side, and the opposite hue on the other. As a result, they have variation in both hue and rotation. These variations are particularly interesting, because there’s an interaction between hue and rotation. But cycling hue by 180 degrees flips which hue is on which side, and is so is equivalent to rotating by 180 degrees.

In the following diagram, we show orientation rotating the whole 360 degrees, but hue only rotating 180. At the bottom of the chart, it wraps around to the top but shifts by 180 degrees.

Rotational Equivariance
Hue Equivariance
Color Contrast Detectors
(hue+180, orientation) = (hue, orientation+180)

**Reflection Equivariance:** As we move into the mid layers of the network, rotated variations become less prominent, but horizontally flipped pairs become quite prevalent.

Horizontal Flip
Dog snout detectors
S-curve detectors
Human beside animal
Horizontal Flip
Horizontal Flip

**Miscellaneous Equivariance:** Finally, we see variations of features transformed in other miscellaneous ways. For example, short vs long-snouted versions of the same dog head features, or human vs dog versions of the same feature. We even see units which are equivariant to camera perspective (found in a Places365 model). These aren’t necessarily something that we would classically think of as forms of equivariance, but do seem to essentially be the same thing.

Snout Length
Human vs Dog
Perspective

---

Equivariant Circuits
--------------------

The equivariant behavior we observe in neurons is really a reflection of a deeper symmetry that exists in the weights of neural networks and the [circuits](https://distill.pub/2020/circuits/zoom-in/#claim-2) they form.

We’ll start by focusing on rotationally equivariant features that are formed from rotationally invariant features. This “invariant→equivariant” case is probably the simplest form of equivariant circuit. Next, we’ll look at “equivariant→invariant” circuits, and then finally the more complex “equivariant→equivariant” circuits.

**High-Low Circuit:** In the following example, we see [high-low frequency detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3a_high_low_frequency) get built from a high-frequency factor and a low-frequency factor (both factors correspond to a combination of neurons in the previous layer). Each high-low frequency detector responds to a transition in frequency in a given direction, detecting high-frequency patterns on one side, and low frequency patterns on the other. Notice how the same weight pattern rotates, making rotated versions of the feature.

positive (excitation)
negative (inhibition)

 respond to a high-frequency neuron factor on one side and low frequency on the other. Notice how the weights rotate:
High-low frequency detectors

This makes them rotationally equivariant.

**Contrast→Center Circuit:** This same pattern can be used in reverse to turn rotationally equivariant features back into rotationally invariant features (an “equivariant→invariant” circuit). In the following example, we see several green-purple [color contrast detectors](https://distill.pub/2020/circuits/early-vision/#group_conv2d2_color_contrast) get combined to create green-purple and purple-green [center-surround detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3a_color_center_surround).

 Compare the weights in this circuit to the ones in the previous one. It’s literally the same weight pattern transposed.

Rotational equivariance can be turned into invariance with the transpose of an
invariant -> equivariant circuit.

Here, we see  (rotationally equivariant) combine to make  (rotationally invariant). Again, notice how the weights rotate, forming the same pattern we saw above with high-low frequency detectors, but with inputs and outputs swapped.
color contrast unitscolor center surround units

positive (excitation)
negative (inhibition)

Sometimes we see one of these immediately follow the other: equivariance be created, and then immediately partially used to create invariant units.

**BW-Color Circuit:** In the following example, a generic color factor and a black and white factor are used to create black and white vs color features. Later, the [black and white vs color features](https://distill.pub/2020/circuits/early-vision/#group_mixed3a_bw_vs_color) are combined to create units which detect black and white at the center, but color around, or vice versa.

But then one major use of the equivariant units is combining them into rotationally invariant center surround units.
First, rotationally equivariant “black and white vs color” units are formed from mostly invariant features.

positive (excitation)
negative (inhibition)

**Line→Circle/Divergence Circuit:** Another example of equivariant features being combined to create invariant features is very early line-like [complex Gabor detectors](https://distill.pub/2020/circuits/early-vision/#group_conv2d1_complex_gabor) being combined to create a small circle unit and diverging lines unit.

positive (excitation)

negative (inhibition)

A  is created by detecting early edges  to a normal line.
circle detectorperpendicular

A  is created by detecting early edges  to a normal line.
diverging line detectorparallel

**Curve→Circle/Evolute Circuit:** For a more complex example of rotational equivariance being combined to create invariant units, we can look at [curve detectors](https://distill.pub/2020/circuits/curve-detectors/) being combined to create [circle](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_circles_loops) and [evolute detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_evolute). This circuit is also an example of scale equivariance. The same general pattern which turns small curve detectors into a small circle detector turns large curve detectors into a large circle detector. The same pattern which turns medium curve detectors into a medium evolute detector turns large curves into a large evolute detector.

conv2d -> mixed3a
Small circle from curves

mixed3a -> mixed3b
Medium evolute from curves

mixed3b -> mixed4a
Large circle and evolute from curves

**Human-Animal Circuit:** So far, all of the examples we’ve seen of circuits have involved rotation. These human-animal and animal-human detectors are an example of horizontal flip equivariance instead:

positive (excitation)

negative (inhibition)

Human detectors excite the human side of each unit and inhibit the other.
Other units (mainly dog detectors) inhibit the human side of each unit and excite the other.

**Invariant Dog Head Circuit:** Conversely, this example (part of the broader [oriented dog head circuit](https://distill.pub/2020/circuits/zoom-in/#claim-2-dog)) shows left and right oriented dog heads get combined into a pose invariant dog head detector. Notice how the weights flip.

### “Equivariant→Equivariant” Circuits

The circuits we’ve looked at so far were either “invariant→equivariant” or “equivariant→invariant.” Either they had invariant input units, or invariant output units. Circuits of this form are quite simple: the weights rotate, or flip, or otherwise transform, but only in response to the transformation of a single feature. When we look at “equivariant→equivariant” circuits, things become a bit more complex. Both the input and output features transform, and we need to consider the relative relationship between the two units.

**Hue→Hue Circuit:** Let’s start with a circuit connecting two sets of hue-equivariant center-surround detectors. Each unit in the second layer is excited by the unit selecting for a similar hue in the previous layer.

Each unit is excited by the unit with the same hue in the previous layer.
The weights between all selected color center-surround units. Units are excited by units with the same, and tend to be inhibited by those with slightly different hues. (In early layers, very different hues inhibit; by later layers very different colors are already distinguished and inhibition focuses on similar colors.)

To understand the above, we need to focus on the relative relationships between each input and output neuron — in this case, how far the hues are apart on the color wheel. When they have the same hue, the relationship is excitatory. When they have close but different hues, it’s inhibitory. And when they are very different, the weight is close to zero. The units used to illustrate hue equivariance here were selected to have a straightforward circuit. Other units may have more complex relationships. For example, some units respond to a range of hues like yellow-red and have correspondingly more complex weights.

**Curve→Curve Circuit:** Let’s now consider a slightly more complex example, how early curve detectors connect to late curve detectors. We’ll focus on four [curve detectors](https://distill.pub/2020/circuits/curve-detectors/) that are 90 degrees rotated from each other.Again, the curve detectors presented were selected to make the circuit as simple and pedagogical as possible. They have clean weights and even spacing between them, which will make the pattern easier to see. A forthcoming article will discuss curve circuits in detail.

If we just look at the matrix of weights, it’s a bit hard to understand. But if we focus on how each curve detector connects to the earlier curve in the same and opposite orientations, it becomes easier to see the structure. Rather than each curve being built from the same neurons in the previous layer, they shift. Each curve is excited by curves in the same orientation and inhibited by those in the opposite. At the same time, the spatial structure of the weights also rotate.

Weights from each of the four curve detectors displayed to each of the other four. Notice how the diagonal is excitatory ( ), while the off-diagonal is inhibitory ( ). The weights shown to the left are a subset of these.
Each curve inhibits the curve in the opposite orientation in the next layer along its tangent. Notice how the weights rotate:
Each curve excites the curve in the same orientation in the next layer along its tangent. Notice how the weights rotate:

**Contrast→Line Circuit:** For a yet more complex example, let’s look at how [color contrast detectors](https://distill.pub/2020/circuits/early-vision/#group_conv2d2_color_contrast) connect to [line detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3a_lines). The general idea is line detectors should fire more strongly if there are different colors on each side of the line. Conversely, they should be inhibited by a change in color if it is perpendicular to the line.

Note that this is an “equivariant→equivariant” circuit with respect to rotation, but “equivariant→invariant” with respect to hue.

 roughly arranged by orientation and hue.
Color contrast detectors

One of the downstream roles of color contrast detectors is to make line detectors respond to changes in color across the line. In this circuit, we will see how this is done.

A (slightly tilted)  is excited by horizontal color contrasts of all hues, and inhibited by vertical ones.
horizontal line detector

A  is excited by vertical color contrasts of all hues, and inhibited by horizontal ones.
vertical line detector

---

Equivariant Architectures
-------------------------

 Equivariance has a rich history in deep learning.
 Many important neural network architectures have equivariance at their core, and there is a very active thread of research around more aggressively incorporating equivariance.
 But the focus is normally on designing equivariant architectures, rather than “natural equivariance” we’ve discussed so far.
 How should we think about the relationship between “natural” and “designed” equivariance?
 As we’ll see, there appears to be quite a deep connection.

 Historically, there has been some interesting back and forth between the two.
 Researchers have often observed that many features in the first layer of neural networks are transformed versions of one basic template.Features in the first layer of neural networks are much more often studied than in other layers. This is because they are easy to study: you can just visualize the weights to pixel values, or more generally to input features.
 This naturally occurring equivariance in the first layer has then sometimes been — and in other cases, easily could have been — inspiration for the design of new architectures.

For example, if you train a fully-connected neural network on a visual task, the first layer will learn variants of the same features over and over: Gabor filters at different positions, orientations, and scales. Convolutional neural networks changed this. By baking the existence of translated copies of each feature directly into the network architecture, they generally remove the need for the network to learn translated copies of each feature. This resulted in a massive increase in statistical efficiency, and became a cornerstone of modern deep learning approaches to computer vision. But if we look at the first layer of a well-trained convolutional neural network, we see that other transformed versions of the same feature remain:

![](images/slim_conv1.png)
The weights for the units in the first layer of the TF-Slim
 version of InceptionV1 .We show the first layer conv weights of the tf-slim version of InceptionV1 rather than the canonical one because its weights are cleaner. This is likely due to the inclusion of batch-norm in the slim variant, causing cleaner gradients. Units are sorted by the first principal component of the adjacency matrix between the first and second layers. Note how many features are similar except for rotation, scale, and hue.

Inspired by this, a 2011 paper subtitled “One Gabor to Rule Them All”  created a sparse coding model which had a single Gabor filter translated, rotated, and scaled. In more recent years, a number of papers have extended this equivariance to the hidden layers of neural networks, and to broader kinds of transformations . Just as convolutional neural networks enforce that the weights between two features be the same if they have the same relative position:

 W(x1, y1, a) → (x2, y2, b)  =  W(x1+Δx, y1+Δy, a) → (x2+Δx, y2+Δy, b)W\_{(x\_1,~y\_1,~a) ~\to~ (x\_2,~y\_2,~b)} ~~=~~ W\_{(x\_1+\Delta x,~y\_1 +\Delta y,~a) ~\to~ (x\_2+\Delta x,~y\_2+\Delta y,~b)}W(x1, y1, a) → (x2, y2, b)  =  W(x1+Δx, y1+Δy, a) → (x2+Δx, y2+Δy, b)

… these more sophisticated equivariant networks make the weights between two neurons equal if they have the same relative relationship under more general transformations:
 For our purposes, it suffices to know that these equivariant neural networks have the same weights when there is the same relative relationship between neurons. This footnote is for the benefit of readers who may wish to engage more deeply in the enforced equivariance literature, and can be safely skipped.

 Group theory is an area of mathematics that gives us tools for describing symmetries and sets of interacting transformations. To build equivariant neural networks, we often borrow an idea from group theory called a group convolution. Just as a regular convolution can describe weights that correctly respect translational equivariance, a group convolution can describe weights that respect a complex set of interacting transformations (the group it operates over). Although you could try to work out how to tie the weights to achieve this from first principles, it’s easy to make mistakes. (One of the authors participated in many conversations with researchers in 2012 where people made errors on whiteboards about how sets of rotated and translated features should interact, without using convolutions.) Group convolutions can take any group you describe and give you the correct weight tying.

 For an approachable introduction to group convolutions, we recommend [this article](https://colah.github.io/posts/2014-12-Groups-Convolution/).

 If you dig further, you may begin to see papers discussing something called a group representation instead of group convolutions. This is a more advanced topic in group theory. The core idea is analogous to the Fourier transform. Recall that the Fourier transform turns convolution into pointwise multiplication (this is sometimes used to accelerate convolution). Well, the Fourier transform has a version that can operate over functions on groups, and also maps convolution to pointwise multiplication. And when you apply the Fourier transform to a group, the resulting coefficients correspond to something called a group representation, which you can think of as being analogous to a frequency in the regular Fourier transform.

Wa → b  =  WT(a)→T(b)W\_{a~\to~ b} ~~=~~ W\_{T(a) \to T(b)}Wa → b  =  WT(a)→T(b)

This is, at least approximately, what we saw conv nets naturally doing when we look at equivariant circuits! The weights had symmetries that caused neurons with similar relationships to have similar weights, much like an equivariant architecture would force them to.

Given that we have neural network architectures which mimic the natural structures we observe, it seems natural to wonder what features and circuits such models learn. Do they learn the same equivariant features we see naturally form? Or do they do something entirely different?
 To answer these questions, we trained an equivariant model roughly inspired by InceptionV1 on ImageNet. We made half the neurons rotationally equivariant (with 16 rotations), and made the others rotationally invariant. Since we put no effort into tuning it, the model achieved abysmal test accuracy but still learns interesting features.
 Here are the full set of features learned by the equivariant model. Half are forced to be rotationally equivariant, while half are forced to be rotationally invariant.

![](images/equiv.png)

 Looking at mixed3b, we found that the equivariant model learned analogues of many large rotationally equivariant families from InceptionV1, such as [curve detectors](https://distill.pub/2020/circuits/curve-detectors/), [boundary detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_boundary), [divot detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_divots), and [oriented fur detectors](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_generic_oriented_fur):

[respond to curves in various orientations.
Curve detectors](https://distill.pub/2020/circuits/curve-detectors/)
Naturally ocurring rotationally equivariant features
Analogous features found in a model where some units are forced to have 16 rotated copies
[Oriented fur detectors
 detect fur parting in a particular way.](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_generic_oriented_fur)
[Boundary detectors
 use multiple cues to detect oriented boundaries of objects.](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_boundary)
[Divot detectors

look for sharp curves sticking out.](https://distill.pub/2020/circuits/early-vision/#group_mixed3b_divots)

 The existence of analogous features in equivariant models can be seen as a successful prediction of interpretability.
 As researchers engaged in more qualitative research, we should always be worried that we may be fooling ourselves.
 Successfully predicting which features will form in an equivariant neural network architecture is actually a pretty non-trivial prediction to make, and a nice confirmation that we’re correctly understanding things.

 Another exciting possibility is that this kind of feature and circuit analysis may be able to help inform equivariance research.
 For example, the kinds of equivariance that naturally form might be helpful in informing what types of equivariance we should design into different layers of a neural network.

---

Conclusion
----------

Equivariance has a remarkable ability to simplify our understanding of neural networks. When we see neural networks as families of features, interacting in structured ways, understanding small templates can actually turn into understanding how large numbers of neurons interact. Equivariance is a big help whenever we discover it.

We sometimes think of understanding neural networks as being like reverse engineering a regular computer program. In this analogy, equivariance is like finding the same inlined function repeated throughout the code. Once you realize that you’re seeing many copies of the same function, you only need to understand it once.

But natural equivariance does have some limitations. For starters, we have to find the equivariant families. This can actually take us quite a bit of work, poring through neurons. Further, they may not be exactly equivariant: one unit may be wired up slightly differently, or have a small exception, and so understanding it as equivariant could leave gaps in our understanding.

We’re excited about the potential of equivariant architectures to make the features and circuits of neural networks easier to understand. This seems especially promising in the context of early vision, where the vast majority of features seem to be equivariant to rotation, hue, scale, or a combination of those.

One of the biggest — and least discussed — advantages we have over neuroscientists in studying vision in artificial neural networks instead of biological neural networks is translational equivariance. By only having one neuron for each feature instead of tens of thousands of translated copies, convolutional neural networks massively reduce the complexity of studying artificial vision systems relative to biological ones. This has been a key ingredient in making it at all plausible that we can systematically understand InceptionV1.

Perhaps in the future, the right equivariant architecture will be able to shave another order of magnitude of complexity off of understanding early vision in neural networks. If so, understanding early vision might move from “possible with effort” to “easily achievable.”

![](images/multiple-pages.svg)

 This article is part of the Circuits thread, a collection of short articles and commentary by an open scientific collaboration delving into the inner workings of neural networks.

[Curve Detectors](/2020/circuits/curve-detectors/)
[High-Low Frequency Detectors](/2020/circuits/frequency-edges/)

Naturally Occurring Equivariance in Neural Networks

Chris Olah
Nick Cammarata
Ludwig Schubert
Gabriel Goh

Summary: Neural networks naturally learn many transformed copies of the same feature, connected by symmetric weights.

The statistical inefficiency of sparse coding for images (or, one Gabor to rule them all)

Group equivariant convolutional networks

Exploiting cyclic symmetry in convolutional neural networks

Steerable CNNs

Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds

3D G-CNNs for pulmonary nodule detection

An introduction to systems biology: design principles of biological circuits

Going deeper with convolutions

Imagenet: A large-scale hierarchical image database

Places: An image database for deep scene understanding

TF-Slim: A high level library to define complex models in TensorFlow