![](images/multiple-pages.svg)

 This article is part of the [Circuits thread](/2020/circuits/), an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.

[Circuits Thread](/2020/circuits/)
[An Overview of Early Vision in InceptionV1](/2020/circuits/early-vision/)

### Contents

[Three Speculative Claims](#three-speculative-claims)
[Claim 1: Features](#claim-1)
* [Example 1: Curve Detectors](#claim-1-curves)
* [Example 2: High-Low Frequency Detectors](#claim-1-hilo)
* [Example 3: Pose-Invariant Dog Head Detector](#claim-1-dog)
* [Polysemantic Neurons](#claim-1-polysemantic)

[Claim 2: Circuits](#claim-2)
* [Circuit 1: Curve Detectors](#claim-2-curves)
* [Circuit 2: Oriented Dog Head Detection](#claim-2-dog)
* [Circuit 3: Cars in Superposition](#claim-2-superposition)
* [Circuit Motifs](#claim-2-motifs)

[Claim 3: Universality](#claim-3)
[Interpretability as a Natural Science](#natural-science)
[Closing Thoughts](#closing)

 Many important transition points in the history of science have been moments when science “zoomed in.”
 At these points, we develop a visualization or tool that allows us to see the world in a new level of detail, and a new field of science develops to study the world through this lens.

 For example, microscopes let us see cells, leading to cellular biology. Science zoomed in. Several techniques including x-ray crystallography let us see DNA, leading to the molecular revolution. Science zoomed in. Atomic theory. Subatomic particles. Neuroscience. Science zoomed in.

 These transitions weren’t just a change in precision: they were qualitative changes in what the objects of scientific inquiry are.
 For example, cellular biology isn’t just more careful zoology.
 It’s a new kind of inquiry that dramatically shifts what we can understand.

 The famous examples of this phenomenon happened at a very large scale,
 but it can also be the more modest shift of a small research community realizing they can now study their topic in a finer grained level of detail.

![](./images/micrographia2.jpg)

 Hooke’s Micrographia revealed a rich microscopic world as seen
 through a microscope, including the initial discovery of cells.

Images from the National Library of Wales.

Just as the early microscope hinted at a new world of cells and microorganisms, visualizations of artificial neural networks have revealed tantalizing hints and glimpses of a rich inner world within our models (e.g. ).
 This has led us to wonder: Is it possible that deep learning is at a similar, albeit more modest, transition point?

 Most work on interpretability aims to give simple explanations of an entire neural network’s behavior.
 But what if we instead take an approach inspired by neuroscience or cellular biology — an approach of zooming in?
 What if we treated individual neurons, even individual weights, as being worthy of serious investigation?
 What if we were willing to spend thousands of hours tracing through every neuron and its connections?
 What kind of picture of neural networks would emerge?

 In contrast to the typical picture of neural networks as a black box, we’ve been surprised how approachable the network is on this scale.
 Not only do neurons seem understandable (even ones that initially seemed inscrutable), but the “circuits” of connections between them seem to be meaningful algorithms corresponding to facts about the world.
 You can watch a circle detector be assembled from curves.
 You can see a dog head be assembled from eyes, snout, fur and tongue.
 You can observe how a car is composed from wheels and windows.
 You can even find circuits implementing simple logic: cases where the network implements AND, OR or XOR over high-level visual features.

![](./images/deepdream.jpg)

 Over the last few years, we’ve seen many incredible visualizations and analyses hinting at a rich world of internal features in modern
 neural networks. Above, we see a [DeepDream](https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html) image, which sparked a great deal of excitement in this space.

 This introductory essay offers a high-level overview of our thinking and some of the working principles that we’ve found useful in this line of research.
 In future articles, we and our collaborators will publish detailed explorations of this inner world.

 But the truth is that we’ve only scratched the surface of understanding a single vision model.
 If these questions resonate with you, you are welcome to join us and our collaborators in the Circuits project, an open scientific collaboration hosted on the [Distill slack](http://slack.distill.pub/).

---

Three Speculative Claims
------------------------

One of the earliest articulations of something approaching modern cell theory was three claims by Theodor Schwann — who you may know for Schwann cells — in 1839:

![](./images/schwann-book.jpg)

#### Schwann’s Claims about Cells

Claim 1
 The cell is the unit of structure, physiology, and organization in living things.

Claim 2
 The cell retains a dual existence as a distinct entity and a building block in the construction of organisms.

Claim 3
 Cells form by free-cell formation, similar to the formation of crystals.

 This translation/summarization of Schwann’s claims can be found in many biology texts; we were unable to determine what the original source of the translation is. The image of Schwann's book is from the [Deutsches Textarchiv](http://www.deutschestextarchiv.de/book/show/schwann_mikroskopische_1839).

The first two of these claims are likely familiar, persisting in modern cellular theory. The third is likely not familiar, since it turned out to be horribly wrong.

We believe there’s a lot of value in articulating a strong version of something one may believe to be true, even if it might be false like Schwann’s third claim. In this spirit, we offer three claims about neural networks. They are intended both as empirical claims about the nature of neural networks, and also as normative claims about how it’s useful to understand them.

![](./images/atlas-book-crop.png)

#### Three Speculative Claims about Neural Networks

Claim 1: Features
 Features are the fundamental unit of neural networks.

 They correspond to directions.
 By “direction” we mean a linear combination of neurons in a layer.
 You can think of this as a direction vector in the vector space of activations of neurons in a given layer.
 Often, we find it most helpful to talk about individual neurons,
 but we’ll see that there are some cases where other combinations are a more useful way to analyze networks — especially when neurons are “polysemantic.”
 (See the [glossary](#glossary-direction) for a detailed definition.)

 These features can be rigorously studied and understood.

Claim 2: Circuits
 Features are connected by weights, forming circuits.
 A “circuit” is a computational subgraph of a neural network.
 It consists of a set of features, and the weighted edges that go between them in the original
 network.
 Often, we study quite small circuits — say with less than a dozen features — but they can also be much larger.
 (See the [glossary](#glossary-circuit) for a detailed definition.)

 These circuits can also be rigorously studied and understood.

Claim 3: Universality
 Analogous features and circuits form across models and tasks.

 Left: An [activation atlas](https://distill.pub/2019/activation-atlas/) visualizing part of the space neural network features can represent.

These claims are deliberately speculative.
 They also aren’t totally novel: claims along the lines of (1) and (3) have been suggested before, as we’ll discuss in more depth below.

 But we believe these claims are important to consider because, if true, they could form the basis of a new “zoomed in” field of
 interpretability. In the following sections, we’ll discuss each one individually and present some of the evidence that has led us to believe they might be true.

---

Claim 1: Features
-----------------

 .claim-quote {
 border-left: 1px solid #CCC;
 padding-left: 20px;
 color: #555;
 margin-bottom: 30px;
 }

 Features are the fundamental unit of neural networks.
 They correspond to directions. They can be rigorously studied and understood.

 We believe that neural networks consist of meaningful, understandable features. Early layers contain features like edge or curve detectors, while later layers have features like floppy ear detectors or wheel detectors.
 The community is divided on whether this is true.
 While many researchers treat the existence of meaningful neurons as an almost trivial fact — there’s even a small literature studying them  — many others are deeply skeptical and believe that past cases of neurons that seemed to track meaningful latent variables were mistaken .
 The community disagreement on meaningful features is hard to pin down, and only partially expressed in the literature. Foundational descriptions of deep learning often describe neural networks as detecting a hierarchy of meaningful features , and a number of papers have been written demonstrating seemingly meaningful features in different domains domains. At the same time, a more skeptical parallel literature has developed suggesting that neural networks primarily or only focus on texture, local structure, or imperceptible patterns , that meaningful features, when they exist, are less important than uninterpretable ones  and that seemingly interpretable neurons may be misunderstood . Although many of these papers express a highly nuanced view, that isn’t always how they’ve been understood. A number of media articles have been written embracing strong versions of these views, and we anecdotally find that the belief that neural networks don’t understand anything more than texture is quite common. Finally, people often have trouble articulating their exact views, because they don’t have clear language for articulating nuances between “a texture detector highly correlated with an object” and “an object detector.”
 Nevertheless, thousands of hours of studying individual neurons have led us to believe the typical case is that neurons (or in some cases, other directions in the vector space of neuron activations) are understandable.

 Of course, being understandable doesn’t mean being simple or easily understandable.
 Many neurons are initially mysterious and don’t follow our a priori guesses of what features might exist!
 However, our experience is that there’s usually a simple explanation behind these neurons, and that they’re actually doing something quite natural.
 For example, we were initially confused by high-low frequency detectors (discussed below) but in retrospect, they are simple and elegant.

 This introductory essay will only give an overview of a couple examples we think are illustrative, but it will be followed both by deep dives carefully characterizing individual features, and broad overviews sketching out all the features we understand to exist.
 We will take our examples from InceptionV1 for now, but believe these claims hold generally and will discuss other models in the final section on universality.

 Regardless of whether we’re correct or mistaken about meaningful features,
 we believe this is an important question for the community to resolve.
 We hope that introducing several specific carefully explored examples of seemingly understandable features will help advance the dialogue.

### Example 1: Curve Detectors

 Curve detecting neurons can be found in every non-trivial vision model we’ve carefully examined.
 These units are interesting because they straddle the boundary between features the community broadly agrees exist (e.g. edge detectors) and features for which there’s significant skepticism (e.g. high-level features such as ears, automotives, and faces).

We’ll focus on curve detectors in layer [`mixed3b`](/2020/circuits/early-vision/#mixed3b), an early layer of InceptionV1. These units responded to curved lines and boundaries with a radius of around 60 pixels. They are also slightly additionally excited by perpendicular lines along the boundary of the curve, and prefer the two sides of the curve to be different colors.

Curve detectors are found in families of units, with each member of the family detecting the same curve feature in a different orientation. Together, they jointly span the full range of orientations.

 It’s important to distinguish curve detectors from other units which may seem superficially similar.
 In particular, there are many units which use curves to detect a curved sub-component (e.g. circles, spirals, S-curves, hourglass shape, 3d curvature, …).
 There are also units which respond to curve related shapes like lines or sharp corners.
 We do not consider these units to be curve detectors.

![](./images/curves.png)

 But are these “curve detectors” really detecting curves?
 We will be dedicating an entire later [article](/2020/circuits/curve-detectors/) to exploring this in depth,
 but the summary is that we think the evidence is quite strong.

 We offer seven arguments, outlined below.
 It’s worth noting that none of these arguments are curve specific: they’re a useful, general toolkit for testing our understanding of other features as well.
 Several of these arguments — dataset examples, synthetic examples, and tuning curves — are classic methods from visual neuroscience (e.g. ).
 The last three arguments are based on circuits, which we’ll discuss in the next section.

![](images/arg-fv.png)

#### Argument 1: Feature Visualization

Optimizing the input to cause curve detectors to fire reliably produces curves. This establishes a causal link, since everything in the resulting image was added to cause the neuron to fire more.

You can learn more about feature visualization [here](https://distill.pub/2017/feature-visualization/).

![](images/arg-data.png)

#### Argument 2: Dataset Examples

The ImageNet images that cause these neurons to strongly fire are reliably curves in the expected orientation. The images that cause them to fire moderately are generally less perfect curves or curves off orientation.

![](images/arg-synthetic.png)

#### Argument 3: Synthetic Examples

Curve detectors respond as expected to a range of synthetic curves images created with varying orientations, curvatures, and backgrounds. They fire only near the expected orientation, and do not fire strongly for straight lines or sharp corners.

![](images/arg-tune.png)

#### Argument 4: Joint Tuning

If we take dataset examples that cause a neuron to fire and rotate them, they gradually stop firing and the curve detectors in the next orientation begins firing. This shows that they detect rotated versions of the same thing. Together, they tile the full 360 degrees of potential orientations.

![](images/arg-weights.png)

#### Argument 5: Feature implementation
 (circuit-based argument)

By looking at the circuit constructing the curve detectors, we can read a curve detection algorithm off of the weights. We also don’t see anything suggestive of a second alternative cause of firing, although there are many smaller weights we don’t understand the role of.

![](images/arg-use.png)

#### Argument 6: Feature use
 (circuit-based argument)

The downstream clients of curve detectors are features that naturally involve curves (e.g. circles, 3d curvature, spirals…). The curve detectors are used by these clients in the expected manner.

![](images/arg-hand.png)

#### Argument 7: Handwritten Circuits
 (circuit-based argument)

Based on our understanding of how curve detectors are implemented,
 we can do a cleanroom reimplementation,
 hand setting all weights to reimplement curve detection.
 These weights are an understandable curve detection algorithm, and significantly mimic the original curve detectors.

The above arguments don’t fully exclude the possibility of some rare secondary case where curve detectors fire for a different kind of stimulus. But they do seem to establish
 that (1) curves cause these neurons to fire,
 (2) each unit responds to curves at different angular orientations,
 and (3) if there are other stimuli that cause them to fire those stimuli are rare or cause weaker activations.
 More generally, these arguments seem to meet the evidentiary standards we understand to be used in neuroscience, which has established traditions and institutional knowledge of how to evaluate such claims.

 All of these arguments will be explored in detail in the later articles on [curve detectors](/2020/circuits/curve-detectors/) and curve detection circuits.

### Example 2: High-Low Frequency Detectors

 Curve detectors are an intuitive type of feature — the kind of feature one might guess exists in neural networks a priori.
 Given that they’re present, it’s not surprising we can understand them.
 But what about features that aren’t intuitive? Can we also understand those?
 We believe so.

 High-low frequency detectors are an example of a less intuitive type of feature. We find them in [early vision](/2020/circuits/early-vision/), and once you understand what they’re doing, they’re quite simple. They look for low-frequency patterns on one side of their receptive field, and high-frequency patterns on the other side. Like curve detectors, high-low frequency detectors are found in families of features that look for the same thing in different orientations.

![](./images/high-low.png)

Why are high-low frequency detectors useful to the network? They seem to be one of several heuristics for detecting the boundaries of objects, especially when the background is out of focus. In a later article, we’ll explore how they’re used in the construction of [sophisticated boundary detectors](/2020/circuits/early-vision/#mixed3b_discussion_boundary).

 (One hope some researchers have for interpretability is that understanding models will be able to teach us better abstractions for thinking about the world . High-low frequency detectors are, perhaps, an example of a small success in this: a natural, useful visual feature that we didn’t anticipate in advance.)

All seven of the techniques we used to interrogate curve neurons can also be used to study high-low frequency neurons with some tweaking — for instance, rendering synthetic high-low frequency examples. Again we believe these arguments collectively provide strong support for the idea that these really are a family of high-low frequency contrast detectors.

### Example 3: Pose-Invariant Dog Head Detector

Both curve detectors and high-low frequency detectors are low-level visual features, found in the early layers of InceptionV1. What about more complex, high-level features?

Let’s consider this unit which we believe to be a pose-invariant dog detector. As with any neuron, we can create a feature visualization and collect dataset examples. If you look at the feature visualization, the geometry is… not possible, but very informative about what it’s looking for and the dataset examples validate it.

![](./images/dog-pose.png)

 It’s worth noting that the combination of feature visualization and dataset examples alone are already quite a strong argument.
 Feature visualization establishes a causal link, while dataset examples test the neuron’s use in practice and whether there are a second type of stimuli that it reacts to.
 But we can bring all our other approaches to analyzing a neuron to bear again.
 For example, we can use a 3D model to generate synthetic dog head images from different angles.

 At the same time, some of the approaches we’ve emphasized so far become a lot of effort for these higher-level, more abstract features.
 Thankfully, our circuit-based arguments — which we’ll discuss more soon — will continue to be easy to apply, and give us really powerful tools for understanding and testing high-level features that don’t require a lot of effort.

### Polysemantic Neurons

This essay may be giving you an overly rosy picture: perhaps every neuron yields a nice, human-understandable concept if one seriously investigates it?

 Alas, this is not the case.
 Neural networks often contain “polysemantic neurons” that respond to multiple unrelated inputs.
 For example, InceptionV1 contains one neuron that responds to cat faces, fronts of cars, and cat legs.

![](./images/polysemantic.png)
4e:55 is a polysemantic neuron which responds to cat faces, fronts of cars, and cat legs. It was discussed in more depth in [Feature Visualization](https://distill.pub/2017/feature-visualization/) .

To be clear, this neuron isn’t responding to some commonality of cars and cat faces. Feature visualization shows us that it’s looking for the eyes and whiskers of a cat, for furry legs, and for shiny fronts of cars — not some subtle shared feature.

 We can still study such features, characterizing each different case they fire, and reason about their circuits to some extent.
 Despite this, polysemantic neurons are a major challenge for the circuits agenda, significantly limiting our ability to reason about neural networks.
 Why are polysemantic neurons so challenging? If one neuron with five different meanings connects to another neuron with five different meanings, that’s effectively 25 connections that can’t be considered individually.
 Our hope is that it may be possible to resolve polysemantic neurons,
 perhaps by “unfolding” a network to turn polysemantic neurons into pure features, or training networks to not exhibit polysemanticity in the first place.
 This is essentially the problem studied in the literature of disentangling representations, although at present that literature tends to focus on known features in the latent spaces of generative models.

 One natural question to ask is why do polysemantic neurons form?
 In the next section, we’ll see that they seem to result from a phenomenon we call “superposition” where a circuit spreads a feature across many neurons, presumably to pack more features into the limited number of neurons it has available.

---

Claim 2: Circuits
-----------------

 Features are connected by weights, forming circuits.
 These circuits can also be rigorously studied and understood.

 All neurons in our network are formed from linear combinations of neurons in the previous layer, followed by ReLU.
 If we can understand the features in both layers, shouldn’t we also be able to understand the connections between them?
 To explore this, we find it helpful to study circuits:
 sub-graphs of the network, consisting a set of tightly linked features and the weights between them.

The remarkable thing is how tractable and meaningful these circuits seem to be as objects of study. When we began looking, we expected to find something quite messy. Instead, we’ve found beautiful rich structures, often with [symmetry](/2020/circuits/equivariance/) to them. Once you understand what features they’re connecting together, the individual floating point number weights in your neural network become meaningful! *You can literally read meaningful algorithms off of the weights.*

Let’s consider some examples.

### Circuit 1: Curve Detectors

In the previous section, we discussed curve detectors, a family of units detecting curves in different angular orientations. In this section, we’ll explore how curve detectors are implemented from earlier features and connect to the rest of the model.

 Curve detectors are primarily implemented from earlier, less sophisticated curve detectors and line detectors. These curve detectors are used in the next layer to create 3D geometry and complex shape detectors. Of course, there’s a long tail of smaller connections to other features, but this seems to be the primary story.

For this introduction, we’ll focus on the interaction of the early curve detectors and our full curve detectors.

![](./images/curve-circuit.png)

Let’s focus even more and look at how a single early curve detector connects to a more sophisticated curve detector in the same orientation.

In this case, our model is implementing a 5x5 convolution, so the weights linking these two neurons are a 5x5 set of weights, which can be positive or negative.
 Many of the neurons discussed in this article, including curve detectors, live in branches of InceptionV1 that are structured as a 1x1 convolution that reduce the number of channels to a small bottleneck followed by a 3x3 or 5x5 convolution. The weights we present in this essay are the multiplied out version of the 1x1 and larger conv weights. We think it’s often useful to view this as a single low-rank weight matrix, but this technically does ignore one ReLU non-linearity.

 A positive weight means that if the earlier neuron fires in that position, it excites the late neuron. Conversely a negative weight would mean that it inhibits it.

What we see are strong positive weights, arranged in the shape of the curve detector. We can think of this as meaning that, at each point along the curve, our curve detector is looking for a “tangent curve” using the earlier curve detector.

![](./images/curve-weights-a.png)
The raw weights between the early curve detector and late curve detector in the same orientation are a curve of positive weights surrounded by small negative or zero weights.

![](./images/curve-weights-b.png)
This can be interpreted as looking for “tangent curves”
at each point along the curve.

This is true for every pair of early and full curve detectors in similar orientations. At every point along the curve, it detects the curve in a similar orientation. Similarly, curves in the opposite orientation are inhibitory at every point along the curve.

Curve detectors are excited by earlier detectors
 in **similar orientations**…

… and inhibited by earlier detectors in
 **opposing orientations**.

It’s worth reflecting here that we’re looking at neural network weights and they’re meaningful.

And the structure gets richer the closer you look. For example, if you look at an early curve detector and full curve detector in similar, but not exactly the same orientation you can often see it have stronger positive weights on the side of the curve it is more aligned with.

It’s also worth noting how the weights rotate with the orientation of the curve detector. The symmetry of the problem is reflected as a symmetry in the weights. We call circuits with exhibiting this phenomenon an “equivariant circuit”, and will discuss it in depth in a [later article](/2020/circuits/equivariance/).

### Circuit 2: Oriented Dog Head Detection

The curve detector circuit is a low-level circuit and only spans two layers. In this section, we’ll discuss a higher-level circuit spanning across four layers. This circuit will also teach us about how neural networks implement sophisticated invariances.

Remember that a huge part of what an ImageNet model has to do is tell apart different animals. In particular, it has to distinguish between a hundred different species of dogs! And so, unsurprisingly, it develops a large number of neurons dedicated to recognizing dog related features, including heads.

Within this “dog recognition” system, one circuit strikes us as particularly interesting: a collection of neurons that handle dog heads facing to the left and dog heads facing to the right. Over three layers, the network maintains two mirrored pathways, detecting analogous units facing to the left and to the right. At each step, these pathways try to inhibit each other, sharpening the contrast. Finally, it creates invariant neurons which respond to both pathways.

 @media (min-width: 1000px){
 .dog-circuit-figure {
 grid-column-start: page-start; grid-column-end: page-end; margin-left: -30px; padding-right: 50px;
 }
 }
 @media (max-width: 1000px){
 .dog-circuit-figure {
 grid-column-start: screen-start; grid-column-end: screen-end;
 padding-left: 20px;
 padding-right: 20px;
 }
 }

We call this pattern “unioning over cases”. The network separately detects two cases (left and right) and then takes a union over them to create invariant “multifaceted” units. Note that, because the two pathways inhibit each other, this circuit actually has some XOR like properties.

This circuit is striking because the network could have easily done something much less sophisticated.
 It could easily create invariant neurons by not caring very much about where the eyes, fur and snout went, and just looking for a jumble of them together.
 But instead, the network has learned to carve apart the left and right cases and handle them separately. We’re somewhat surprised that gradient descent could learn to do this!To be clear, there are also more direct pathways by which various constituents of heads influence these later head detectors, without going through the left and right pathways

But this summary of the circuit is only scratching the surface of what is going on. Every connection between neurons is a convolution, so we can also look at where an input neuron excites the the next one. And the models tends to be doing what you might have optimistically hoped. For example, consider these “head with neck” units. The head is only detected on the correct side:

 The union step is also interesting to look at the details of.
 The network doesn’t indiscriminately respond to the heads into the two orientations:
 the regions of excitation extend from the center in different directions depending on orientation, allowing snouts to converge in to the same point.

There’s a lot more to say about this circuit, so we plan to return to it in a future article and analyze it in depth, including testing our theory of the circuit by editing the weights.

### Circuit 3: Cars in Superposition

In `mixed4c`, a mid-late layer of InceptionV1, there is a car detecting neuron. Using features from the previous layers, it looks for wheels at the bottom of its convolutional window, and windows at the top.

But then the model does something surprising. Rather than create another pure car detector at the next layer, it spreads its car feature over a number of neurons that seem to primarily be doing something else — in particular, dog detectors.

This circuit suggests that polysemantic neurons are, in some sense, deliberate. That is, you could imagine a world where the process of detecting cars and dogs was deeply intertwined in the model for some reason, and as a result polysemantic neurons were difficult to avoid. But what we’re seeing here is that the model had a “pure neuron” and then mixed it up with other features.

We call this phenomenon superposition.

Why would it do such a thing? We believe superposition allows the model to use fewer neurons, conserving them for more important tasks. As long as cars and dogs don’t co-occur, the model can accurately retrieve the dog feature in a later layer, allowing it to store the feature without dedicating a neuron.Fundamentally, this is a property of the geometry of high-dimensional spaces, which only allow for n orthogonal vectors, but exponentially many almost orthogonal vectors.

### Circuit Motifs

As we’ve studied circuits throughout InceptionV1 and other models,
 we’ve seen the same abstract patterns over and over.
 [Equivariance](/2020/circuits/equivariance/), as we saw with the curve detectors.
 Unioning over cases, as we saw with the pose-invariant dog head detector.
 Superposition, as we saw with the car detector.

In biology, a circuit motif  is a recurring pattern in complex graphs like transcription networks or biological neural networks.
 Motifs are helpful because understanding one motif can give researchers leverage on all graphs where it occurs.

 We think it’s quite likely that studying motifs will be important in understanding the circuits of artificial neural networks. In the long run, it may be more important than the study of individual circuits.
 At the same time, we expect investigations of motifs to be well served by us first building up a solid foundation of well understood circuits first.

---

Claim 3: Universality
---------------------

Analogous features and circuits form across models and tasks.

 It’s a widely accepted fact that the first layer of vision models trained on natural images will learn Gabor filters.
 Once you accept that there are meaningful features in later layers, would it really be surprising for the same features to also form in layers beyond the first one?
 And once you believe there are analogous features in multiple layers, wouldn’t it be natural for them to connect in the same ways?

 Universality (or “convergent learning”) of features has been suggested before.
 Prior work has shown that different neural networks can develop highly correlated neurons
 and that they learn similar representations at hidden layers .
 This work seems highly suggestive, but there are alternative explanations to analogous features forming.
 For example, one could imagine two features — such as a fur texture detector and a sophisticated dog body detector — being highly correlated despite being importantly different features.
 Adopting the meaningful feature-skeptic perspective, it doesn’t seem definitive.

 Ideally, one would like to characterize several features and then rigorously demonstrate that those features — and not just correlated ones — are forming across many models.
 Then, to further establish that analogous circuits form, one would want to find analogous features over several layers of multiple models and show that the same weight structure forms between them in each model.

 Unfortunately, the only evidence we can offer today is anecdotal:
 we simply have not yet invested enough in the comparative study of features and circuits to give confident answers.
 With that said, we have observed that a couple low-level features seem to form across a variety of vision model architectures (including AlexNet, InceptionV1, InceptionV3, and residual networks) and in models trained on Places365 instead of ImageNet. We’ve also observed them repeatedly form in vanilla conv nets trained from scratch on ImageNet.

 .diagram-universality {
 display: grid;
 grid-column: text-start / page-end;
 grid-gap: 1rem;
 grid-template-columns: 1fr;
 justify-content: center;
 }

 @media (min-width: 1180px) {
 .diagram-universality {
 grid-column: page;
 grid-template-columns: 528px 448px;
 }
 .diagram-universality > figure:last-of-type .info {
 display: none;
 }
 .diagram-universality > figure:last-of-type li {
 grid-template-columns: 1fr;
 }
 .diagram-universality > figure:last-of-type > figcaption {
 padding-left: unset;
 }
 }

 .diagram-universality a {
 border-bottom: none;
 display: inline;
 }
 .diagram-universality a:hover {
 border-bottom: none;
 }
 .diagram-universality a:hover img {
 --filter: brightness(80%);
 }

 .diagram-universality h3 {
 margin-top: 0;
 }

 .diagram-universality h4 {
 margin: 4px 0;
 }

 .diagram-universality > figure {
 margin: 0;
 }

 .diagram-universality > figure > figcaption {
 padding-left: 9rem;
 }

 .diagram-universality ul {
 list-style: none;
 padding-left: unset;
 }

 .diagram-universality li {
 display: grid;
 grid-template-columns: 9rem 1fr;
 }

 .diagram-universality li .images {
 display: grid;
 grid-template-columns: repeat( auto-fill, 88px);
 grid-gap: 8px;
 }

 .diagram-universality img {
 width: 88px;
 height: 88px;
 background-color: gray;
 border-radius: 4px;
 object-fit: none;
 }

### Curve detectors

* #### AlexNet

Krizhevsky et al.

[![](images/universality/curves/AlexNet/unit-0.png)](https://microscope.openai.com/models/)
[![](images/universality/curves/AlexNet/unit-1.png)](https://microscope.openai.com/models/)
[![](images/universality/curves/AlexNet/unit-2.png)](https://microscope.openai.com/models/)
[![](images/universality/curves/AlexNet/unit-3.png)](https://microscope.openai.com/models/)
* #### InceptionV1

Szegedy et al.

[![](images/universality/curves/GoogLeNet/unit-0.png)](https://microscope.openai.com/models/inceptionv1/mixed3b_0/379)
[![](images/universality/curves/GoogLeNet/unit-1.png)](https://microscope.openai.com/models/inceptionv1/mixed3b_0/385)
[![](images/universality/curves/GoogLeNet/unit-2.png)](https://microscope.openai.com/models/inceptionv1/mixed3b_0/342)
[![](images/universality/curves/GoogLeNet/unit-3.png)](https://microscope.openai.com/models/inceptionv1/mixed3b_0/340)
* #### VGG19

Simonyan et al.

[![](images/universality/curves/VGG19/unit-0.png)](https://microscope.openai.com/models/)
[![](images/universality/curves/VGG19/unit-1.png)](https://microscope.openai.com/models/)
[![](images/universality/curves/VGG19/unit-2.png)](https://microscope.openai.com/models/)
[![](images/universality/curves/VGG19/unit-3.png)](https://microscope.openai.com/models/)
* #### ResNetV2-50

He et al.

[![](images/universality/curves/ResNetV2/unit-0.png)](https://microscope.openai.com/models/)
[![](images/universality/curves/ResNetV2/unit-1.png)](https://microscope.openai.com/models/)
[![](images/universality/curves/ResNetV2/unit-2.png)](https://microscope.openai.com/models/)
[![](images/universality/curves/ResNetV2/unit-3.png)](https://microscope.openai.com/models/)

### High-Low Frequency detectors

* #### AlexNet

Krizhevsky et al.

[![](images/universality/hilo/AlexNet/unit-2.png)](https://microscope.openai.com/models/)
[![](images/universality/hilo/AlexNet/unit-1.png)](https://microscope.openai.com/models/)
[![](images/universality/hilo/AlexNet/unit-3.png)](https://microscope.openai.com/models/)
[![](images/universality/hilo/AlexNet/unit-0.png)](https://microscope.openai.com/models/)
* #### InceptionV1

Szegedy et al.

[![](images/universality/hilo/GoogLeNet/unit-0.png)](https://microscope.openai.com/models/inceptionv1/mixed3a_0/136)
[![](images/universality/hilo/GoogLeNet/unit-1.png)](https://microscope.openai.com/models/inceptionv1/mixed3a_0/)
[![](images/universality/hilo/GoogLeNet/unit-2.png)](https://microscope.openai.com/models/inceptionv1/mixed3a_0/)
[![](images/universality/hilo/GoogLeNet/unit-3.png)](https://microscope.openai.com/models/inceptionv1/mixed3a_0/)
* #### VGG19

Simonyan et al.

[![](images/universality/hilo/VGG19/unit-0.png)](https://microscope.openai.com/models/)
[![](images/universality/hilo/VGG19/unit-1.png)](https://microscope.openai.com/models/)
[![](images/universality/hilo/VGG19/unit-2.png)](https://microscope.openai.com/models/)
[![](images/universality/hilo/VGG19/unit-3.png)](https://microscope.openai.com/models/)
* #### ResNetV2-50

He et al.

[![](images/universality/hilo/ResNetV2/unit-3.png)](https://microscope.openai.com/models/)
[![](images/universality/hilo/ResNetV2/unit-2.png)](https://microscope.openai.com/models/)
[![](images/universality/hilo/ResNetV2/unit-1.png)](https://microscope.openai.com/models/)
[![](images/universality/hilo/ResNetV2/unit-0.png)](https://microscope.openai.com/models/)

 These results have led us to suspect that the universality hypothesis is likely true, but further work will be needed to understand if the apparent universality of some low-level vision features is the exception or the rule.

 If it turns out that the universality hypothesis is broadly true in neural networks,
 it will be tempting to speculate: might biological neural networks also learn similar features?
 Researchers working at the intersection of neuroscience and deep learning have already shown that the units in artificial vision models can be useful for modeling biological neurons .
 And some of the features we’ve discovered in artificial neural networks, such as curve detectors, are also believed to exist in biological neural networks (e.g. ).
 This seems like significant cause for optimism.

 One particularly exciting possibility might be if artificial neural networks could predict features which were previously unknown but could then be found in biology.
 (Some neuroscientists we have spoken to have suggested that high-low frequency detectors might be a candidate for this.)
 If such a prediction could be made, it would be extremely strong evidence for the universality hypothesis.

 Focusing on the study of circuits, is universality really necessary?
 Unlike the first two claims, it wouldn’t be completely fatal to circuits research if this claim turned out to be false. But it does greatly inform what kind of research makes sense. We introduced circuits as a kind of “cellular biology of deep learning.” But imagine a world where every species had cells with a completely different set of organelles and proteins. Would it still make sense to study cells in general, or would we limit ourselves to the narrow study of a few kinds of particularly important species of cells? Similarly, imagine the study of anatomy in a world where every species of animal had a completely unrelated anatomy: would we seriously study anything other than humans and a couple domestic animals?

In the same way, the universality hypothesis determines what form of circuits research makes sense. If it was true in the strongest sense, one could imagine a kind of “periodic table of visual features” which we observe and catalogue across models. On the other hand, if it was mostly false, we would need to focus on a handful of models of particular societal importance and hope they stop changing every year. There might also be in between worlds, where some lessons transfer between models but others need to be learned from scratch.

---

Interpretability as a Natural Science
-------------------------------------

*The Structure of Scientific Revolutions* by Thomas Kuhn is a classic text on the history and sociology of science.
 In it, Kuhn distinguishes between “normal science” in which a scientific community has a paradigm, and “extraordinary science” in which a community lacks a paradigm, either because it never had one or because it was weakened by crisis.
 It’s worth noting that “extraordinary science” is not a desirable state: it’s a period where researchers struggle to be productive.

 Kuhn’s description of pre-paradigmatic fields feel eerily reminiscent of interpretability today.
 We were introduced to Kuhn’s work and this connection by conversations with Tom McGrath at DeepMind
 There isn’t consensus on what the objects of study are, what methods we should use to answer them, or how to evaluate research results.
 To quote a recent interview with Ian Goodfellow: “For interpretability, I don’t think we even have the right definitions.”

 One particularly challenging aspect of being in a pre-paradigmatic field is that there isn’t a shared sense of how to evaluate work in interpretability.
 There are two common proposals for dealing with this, drawing on the standards of adjacent fields. Some researchers, especially those with a deep learning background, want an “interpretability benchmark” which can evaluate how effective an interpretability method is. Other researchers with an HCI background may wish to evaluate interpretability methods through user studies.

 But interpretability could also borrow from a third paradigm: natural science.
 In this view, neural networks are an object of empirical investigation, perhaps similar to an organism in biology. Such work would try to make empirical claims about a given network, which could be held to the standard of falsifiability.

 Why don’t we see more of this kind of evaluation of work in interpretability and visualization?
 To be clear, we do see researchers who take more of this natural science approach, especially in earlier interpretability research. It just seems less common right now.
 Especially given that there’s so much adjacent ML work which does adopt this frame!
 One reason might be that it’s very difficult to make robustly true statements about the behavior of a neural network as a whole.
 They’re incredibly complicated objects.
 It’s also hard to formalize what the interesting empirical statements about them would, exactly, be.
 And so we often get standards of evaluations more targeted at whether an interpretability method is useful rather than whether we’re learning true statements.

 Circuits sidestep these challenges by focusing on tiny subgraphs of a neural network for which rigorous empirical investigation is tractable.
 They’re very much falsifiable: for example, if you understand a circuit, you should be able to predict what will change if you edit the weights.
 In fact, for small enough circuits, statements about their behavior become questions of mathematical reasoning.
 Of course, the cost of this rigor is that statements about circuits are much smaller in scope than overall model behavior.
 But it seems like, with sufficient effort, statements about model behavior could be broken down into statements about circuits.
 If so, perhaps circuits could act as a kind of epistemic foundation for interpretability.

---

Closing Thoughts
----------------

We take it for granted that the microscope is an important scientific instrument. It’s practically a symbol of science. But this wasn’t always the case, and microscopes didn’t initially take off as a scientific tool. In fact, they seem to have languished for around fifty years. The turning point was when Robert Hooke published Micrographia, a collection of drawings of things he’d seen using a microscope, including the first picture of a cell.

 Our impression is that there is some anxiety in the interpretability community that we aren’t taken very seriously.
 That this research is too qualitative.
 That it isn’t scientific.
 But the lesson of the microscope and cellular biology is that perhaps this is expected.
 The discovery of cells was a qualitative research result.
 That didn’t stop it from changing the world.

![](images/multiple-pages.svg)

 This article is part of the Circuits thread, a collection of short articles and commentary by an open scientific collaboration delving into the inner workings of neural networks.

[Circuits Thread](/2020/circuits/)
[An Overview of Early Vision in InceptionV1](/2020/circuits/early-vision/)

Zoom In: An Introduction to Circuits

Chris Olah
Nick Cammarata
Ludwig Schubert
Gabriel Goh
Michael Petrov
Shan Carter

Summary: By studying the connections between neurons, we can find meaningful algorithms in the weights of neural networks.

Micrographia: or Some Physiological Descriptions of Minute Bodies Made by Magnifying Glasses. With Observations and Inquiries Thereupon

Visualizing and understanding recurrent networks

Visualizing higher-layer features of a deep network

Feature Visualization

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Visualizing and understanding convolutional networks

Interpretable Explanations of Black Boxes by Meaningful Perturbation

PatternNet and PatternLRP--Improving the interpretability of neural networks

Visualizing and Measuring the Geometry of BERT

Activation atlas

Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations

Distributed representations of words and phrases and their compositionality

Learning to generate reviews and discovering sentiment

Object detectors emerge in deep scene cnns

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Measuring the tendency of CNNs to Learn Surface Statistical Regularities

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Approximating cnns with bag-of-local-features models works surprisingly well on imagenet

Adversarial examples are not bugs, they are features

On the importance of single directions for generalization

Deep learning

Going deeper with convolutions

Using Artificial Intelligence to Augment Human Intelligence

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

An introduction to systems biology: design principles of biological circuits

Convergent learning: Do different neural networks learn the same representations?

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Similarity of neural network representations revisited

ImageNet Classification with Deep Convolutional Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Deep Residual Learning for Image Recognition

Discrete neural clusters encode orientation, curvature and corners in macaque V4

The structure of scientific revolutions

Ian Goodfellow: Generative Adversarial Networks

The Building Blocks of Interpretability


### Contents

[Model](#model)
[Experiments](#experiment-1)
* [Learning to Grow](#experiment-1)
* [What persists, exists](#experiment-2)
* [Learning to regenerate](#experiment-3)
* [Rotating the perceptive field](#experiment-4)

[Related Work](#related-work)
[Discussion](#discussion)

![](images/multiple-pages.svg)

 This article is part of the
 [Differentiable Self-organizing Systems Thread](/2020/selforg/),
 an experimental format collecting invited short articles delving into
 differentiable self-organizing systems, interspersed with critical
 commentary from several experts in adjacent fields.

[Differentiable Self-organizing Systems Thread](/2020/selforg/)
[Self-classifying MNIST Digits](/2020/selforg/mnist/)

 Most multicellular organisms begin their life as a single egg cell - a
 single cell whose progeny reliably self-assemble into highly complex
 anatomies with many organs and tissues in precisely the same arrangement
 each time. The ability to build their own bodies is probably the most
 fundamental skill every living creature possesses. Morphogenesis (the
 process of an organism’s shape development) is one of the most striking
 examples of a phenomenon called *self-organisation*. Cells, the tiny
 building blocks of bodies, communicate with their neighbors to decide the
 shape of organs and body plans, where to grow each organ, how to
 interconnect them, and when to eventually stop. Understanding the interplay
 of the emergence of complex outcomes from simple rules and
 homeostatic
 Self-regulatory feedback loops trying maintain the body in a stable state
 or preserve its correct overall morphology under external
 perturbations
 feedback loops is an active area of research
 . What is clear
 is that evolution has learned to exploit the laws of physics and computation
 to implement the highly robust morphogenetic software that runs on
 genome-encoded cellular hardware.

 This process is extremely robust to perturbations. Even when the organism is
 fully developed, some species still have the capability to repair damage - a
 process known as regeneration. Some creatures, such as salamanders, can
 fully regenerate vital organs, limbs, eyes, or even parts of the brain!
 Morphogenesis is a surprisingly adaptive process. Sometimes even a very
 atypical development process can result in a viable organism - for example,
 when an early mammalian embryo is cut in two, each half will form a complete
 individual - monozygotic twins!

 The biggest puzzle in this field is the question of how the cell collective
 knows what to build and when to stop. The sciences of genomics and stem cell
 biology are only part of the puzzle, as they explain the distribution of
 specific components in each cell, and the establishment of different types
 of cells. While we know of many genes that are *required* for the
 process of regeneration, we still do not know the algorithm that is
 *sufficient* for cells to know how to build or remodel complex organs
 to a very specific anatomical end-goal. Thus, one major lynch-pin of future
 work in biomedicine is the discovery of the process by which large-scale
 anatomy is specified within cell collectives, and how we can rewrite this
 information to have rational control of growth and form. It is also becoming
 clear that the software of life possesses numerous modules or subroutines,
 such as “build an eye here”, which can be activated with simple signal
 triggers. Discovery of such subroutines and a
 mapping out of the developmental logic is a new field at the intersection of
 developmental biology and computer science. An important next step is to try
 to formulate computational models of this process, both to enrich the
 conceptual toolkit of biologists and to help translate the discoveries of
 biology into better robotics and computational technology.

 Imagine if we could design systems of the same plasticity and robustness as
 biological life: structures and machines that could grow and repair
 themselves. Such technology would transform the current efforts in
 regenerative medicine, where scientists and clinicians seek to discover the
 inputs or stimuli that could cause cells in the body to build structures on
 demand as needed. To help crack the puzzle of the morphogenetic code, and
 also exploit the insights of biology to create self-repairing systems in
 real life, we try to replicate some of the desired properties in an
 *in silico* experiment.

Model
-----

 Those in engineering disciplines and researchers often use many kinds of
 simulations incorporating local interaction, including systems of partial
 derivative equation (PDEs), particle systems, and various kinds of Cellular
 Automata (CA). We will focus on Cellular Automata models as a roadmap for
 the effort of identifying cell-level rules which give rise to complex,
 regenerative behavior of the collective. CAs typically consist of a grid of
 cells being iteratively updated, with the same set of rules being applied to
 each cell at every step. The new state of a cell depends only on the states
 of the few cells in its immediate neighborhood. Despite their apparent
 simplicity, CAs often demonstrate rich, interesting behaviours, and have a
 long history of being applied to modeling biological phenomena.

 Let’s try to develop a cellular automata update rule that, starting from a
 single cell, will produce a predefined multicellular pattern on a 2D grid.
 This is our analogous toy model of organism development. To design the CA,
 we must specify the possible cell states, and their update function. Typical
 CA models represent cell states with a set of discrete values, although
 variants using vectors of continuous values exist. The use of continuous
 values has the virtue of allowing the update rule to be a differentiable
 function of the cell’s neighbourhood’s states. The rules that guide
 individual cell behavior based on the local environment are analogous to the
 low-level hardware specification encoded by the genome of an organism.
 Running our model for a set amount of steps from a starting configuration
 will reveal the patterning behavior that is enabled by such hardware.

 So - what is so special about differentiable update rules? They will allow
 us to use the powerful language of loss functions to express our wishes, and
 the extensive existing machinery around gradient-based numerical
 optimization to fulfill them. The art of stacking together differentiable
 functions, and optimizing their parameters to perform various tasks has a
 long history. In recent years it has flourished under various names, such as
 (Deep) Neural Networks, Deep Learning or Differentiable Programming.

A single update step of the model.

### Cell State

 We will represent each cell state as a vector of 16 real values (see the
 figure above). The first three channels represent the cell color visible to
 us (RGB). The target pattern has color channel values in range [0.0,1.0][0.0, 1.0][0.0,1.0]
 and an α\alphaα equal to 1.0 for foreground pixels, and 0.0 for background.

 The alpha channel (α\alphaα) has a special meaning: it demarcates living
 cells, those belonging to the pattern being grown. In particular, cells
 having α>0.1\alpha > 0.1α>0.1 and their neighbors are considered “living”. Other
 cells are “dead” or empty and have their state vector values explicitly set
 to 0.0 at each time step. Thus cells with α>0.1\alpha > 0.1α>0.1 can be thought of
 as “mature”, while their neighbors with α≤0.1\alpha \leq 0.1α≤0.1 are “growing”, and
 can become mature if their alpha passes the 0.1 threshold.

![](figures/alive2.svg)

state⃗→0.00\vec{state} \rightarrow 0.00state⃗→0.00 when no neighbour with α>0.10\alpha > 0.10α>0.10

 Hidden channels don’t have a predefined meaning, and it’s up to the update
 rule to decide what to use them for. They can be interpreted as
 concentrations of some chemicals, electric potentials or some other
 signaling mechanism that are used by cells to orchestrate the growth. In
 terms of our biological analogy - all our cells share the same genome
 (update rule) and are only differentiated by the information encoded the
 chemical signalling they receive, emit, and store internally (their state
 vectors).

### Cellular Automaton rule

 Now it’s time to define the update rule. Our CA runs on a regular 2D grid of
 16-dimensional vectors, essentially a 3D array of shape [height, width, 16].
 We want to apply the same operation to each cell, and the result of this
 operation can only depend on the small (3x3) neighborhood of the cell. This
 is heavily reminiscent of the convolution operation, one of the cornerstones
 of signal processing and differential programming. Convolution is a linear
 operation, but it can be combined with other per-cell operations to produce
 a complex update rule, capable of learning the desired behaviour. Our cell
 update rule can be split into the following phases, applied in order:

**Perception.** This step defines what each cell perceives of
 the environment surrounding it. We implement this via a 3x3 convolution with
 a fixed kernel. One may argue that defining this kernel is superfluous -
 after all we could simply have the cell learn the requisite perception
 kernel coefficients. Our choice of fixed operations are motivated by the
 fact that real life cells often rely only on chemical gradients to guide the
 organism development. Thus, we are using classical Sobel filters to estimate
 the partial derivatives of cell state channels in the x⃗\vec{x}x⃗ and
 y⃗\vec{y}y⃗ directions, forming a 2D gradient vector in each direction, for
 each state channel. We concatenate those gradients with the cells own
 states, forming a 16∗2+16=4816\*2+16=4816∗2+16=48 dimensional *perception vector*, or
 rather *percepted vector,* for each cell.

def perceive(state\_grid):

sobel\_x = [[-1, 0, +1],

[-2, 0, +2],

[-1, 0, +1]]

sobel\_y = transpose(sobel\_x)

# Convolve sobel filters with states

# in x, y and channel dimension.

grad\_x = conv2d(sobel\_x, state\_grid)

grad\_y = conv2d(sobel\_y, state\_grid)

# Concatenate the cell’s state channels,

# the gradients of channels in x and

# the gradient of channels in y.

perception\_grid = concat(

state\_grid, grad\_x, grad\_y, axis=2)

return perception\_grid

**Update rule.** Each cell now applies a series of operations
 to the perception vector, consisting of typical differentiable programming
 building blocks, such as 1x1-convolutions and ReLU nonlinearities, which we
 call the cell’s “update rule”. Recall that the update rule is learned, but
 every cell runs the same update rule. The network parametrizing this update
 rule consists of approximately 8,000 parameters. Inspired by residual neural
 networks, the update rule outputs an incremental update to the cell’s state,
 which applied to the cell before the next time step. The update rule is
 designed to exhibit “do-nothing” initial behaviour - implemented by
 initializing the weights of the final convolutional layer in the update rule
 with zero. We also forego applying a ReLU to the output of the last layer of
 the update rule as the incremental updates to the cell state must
 necessarily be able to both add or subtract from the state.

def update(perception\_vector):

# The following pseudocode operates on

# a single cell’s perception vector.

# Our reference implementation uses 1D

# convolutions for performance reasons.

x = dense(perception\_vector, output\_len=128)

x = relu(x)

ds = dense(x, output\_len=16, weights\_init=0.0)

return ds

**Stochastic cell update.** Typical cellular automata update
 all cells simultaneously. This implies the existence of a global clock,
 synchronizing all cells. Relying on global synchronisation is not something
 one expects from a self-organising system. We relax this requirement by
 assuming that each cell performs an update independently, waiting for a
 random time interval between updates. To model this behaviour we apply a
 random per-cell mask to update vectors, setting all update values to zero
 with some predefined probability (we use 0.5 during training). This
 operation can be also seen as an application of per-cell dropout to update
 vectors.

def stochastic\_update(state\_grid, ds\_grid):

# Zero out a random fraction of the updates.

rand\_mask = cast(random(64, 64) < 0.5, float32)

ds\_grid = ds\_grid \* rand\_mask

return state\_grid + ds\_grid

**Living cell masking.** We want to model the growth process
 that starts with a single cell, and don’t want empty cells to participate in
 computations or carry any hidden state. We enforce this by explicitly
 setting all channels of empty cells to zeros. A cell is considered empty if
 there is no “mature” (alpha>0.1) cell in its 3x3 neightborhood.

def alive\_masking(state\_grid):

# Take the alpha channel as the measure of “life”.

alive = max\_pool(state\_grid[:, :, 3], (3,3)) > 0.1

state\_grid = state\_grid \* cast(alive, float32)

return state\_grid

Experiment 1: Learning to Grow
------------------------------

Training regime for learning a target pattern.

 In our first experiment, we simply train the CA to achieve a target image
 after a random number of updates. This approach is quite naive and will run
 into issues. But the challenges it surfaces will help us refine future
 attempts.

 We initialize the grid with zeros, except a single seed cell in the center,
 which will have all channels except RGB
 We set RGB channels of the seed to zero because we want it to be visible
 on the white background.
 set to one. Once the grid is initialized, we iteratively apply the update
 rule. We sample a random number of CA steps from the [64, 96]
 This should be a sufficient number of steps to grow the pattern of the
 size we work with (40x40), even considering the stochastic nature of our
 update rule.
 range for each training step, as we want the pattern to be stable across a
 number of iterations. At the last step we apply pixel-wise L2 loss between
 RGBA channels in the grid and the target pattern. This loss can be
 differentiably optimized
 We observed training instabilities, that were manifesting themselves as
 sudden jumps of the loss value in the later stages of the training. We
 managed to mitigate them by applying per-variable L2 normalization to
 parameter gradients. This may have the effect similar to the weight
 normalization . Other training
 parameters are available in the accompanying source code.
 with respect to the update rule parameters by backpropagation-through-time,
 the standard method of training recurrent neural networks.

 Once the optimisation converges, we can run simulations to see how our
 learned CAs grow patterns starting from the seed cell. Let’s see what
 happens when we run it for longer than the number of steps used during
 training. The animation below shows the behaviour of a few different models,
 trained to generate different emoji patterns.

 Your browser does not support the video tag.

 Many of the patterns exhibit instability for longer time periods.

[Reproduce in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/growing_ca.ipynb#scrollTo=4O4tzfe-GRJ7)

 We can see that different training runs can lead to models with drastically
 different long term behaviours. Some tend to die out, some don’t seem to
 know how to stop growing, but some happen to be almost stable! How can we
 steer the training towards producing persistent patterns all the time?

Experiment 2: What persists, exists
-----------------------------------

 One way of understanding why the previous experiment was unstable is to draw
 a parallel to dynamical systems. We can consider every cell to be a
 dynamical system, with each cell sharing the same dynamics, and all cells
 being locally coupled amongst themselves. When we train our cell update
 model we are adjusting these dynamics. Our goal is to find dynamics that
 satisfy a number of properties. Initially, we wanted the system to evolve
 from the seed pattern to the target pattern - a trajectory which we achieved
 in Experiment 1. Now, we want to avoid the instability we observed - which
 in our dynamical system metaphor consists of making the target pattern an
 attractor.

 One strategy to achieve this is letting the CA iterate for much longer time
 and periodically applying the loss against the target, training the system
 by backpropagation through these longer time intervals. Intuitively we claim
 that with longer time intervals and several applications of loss, the model
 is more likely to create an attractor for the target shape, as we
 iteratively mold the dynamics to return to the target pattern from wherever
 the system has decided to venture. However, longer time periods
 substantially increase the training time and more importantly, the memory
 requirements, given that the entire episode’s intermediate activations must
 be stored in memory for a backwards-pass to occur.

 Instead, we propose a “sample pool” based strategy to a similar effect. We
 define a pool of seed states to start the iterations from, initially filled
 with the single black pixel seed state. We then sample a batch from this
 pool which we use in our training step. To prevent the equivalent of
 “catastrophic forgetting” we replace one sample in this batch with the
 original, single-pixel seed state. After concluding the training step , we
 replace samples in the pool that were sampled for the batch with the output
 states from the training step over this batch. The animation below shows a
 random sample of the entries in the pool every 20 training steps.

def pool\_training():

# Set alpha and hidden channels to (1.0).

seed = zeros(64, 64, 16)

seed[64//2, 64//2, 3:] = 1.0

target = targets[‘lizard’]

pool = [seed] \* 1024

for i in range(training\_iterations):

idxs, batch = pool.sample(32)

# Sort by loss, descending.

batch = sort\_desc(batch, loss(batch))

# Replace the highest-loss sample with the seed.

batch[0] = seed

# Perform training.

outputs, loss = train(batch, target)

# Place outputs back in the pool.

pool[idxs] = outputs

 Your browser does not support the video tag.

 A random sample of the patterns in the pool during training, sampled
 every 20 training steps.

[Reproduce in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/growing_ca.ipynb#scrollTo=B4JAbAJf6Alw)

 Early on in the training process, the random dynamics in the system allow
 the model to end up in various incomplete and incorrect states. As these
 states are sampled from the pool, we refine the dynamics to be able to
 recover from such states. Finally, as the model becomes more robust at going
 from a seed state to the target state, the samples in the pool reflect this
 and are more likely to be very close to the target pattern, allowing the
 training to refine these almost completed patterns further.

 Essentially, we use the previous final states as new starting points to
 force our CA to learn how to persist or even improve an already formed
 pattern, in addition to being able to grow it from a seed. This makes it
 possible to add a periodical loss for significantly longer time intervals
 than otherwise possible, encouraging the generation of an attractor as the
 target shape in our coupled system. We also noticed that reseeding the
 highest loss sample in the batch, instead of a random one, makes training
 more stable at the initial stages, as it helps to clean up the low quality
 states from the pool.

 Here is what a typical training progress of a CA rule looks like. The cell
 rule learns to stabilize the pattern in parallel to refining its features.

 Your browser does not support the video tag.

 CA behaviour at training steps 100, 500, 1000, 4000.

[Reproduce in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/growing_ca.ipynb#scrollTo=nqvkfl9W4ODI)

Experiment 3: Learning to regenerate
------------------------------------

 In addition to being able to grow their own bodies, living creatures are
 great at maintaining them. Not only does worn out skin get replaced with new
 skin, but very heavy damage to complex vital organs can be regenerated in
 some species. Is there a chance that some of the models we trained above
 have regenerative capabilities?

 Your browser does not support the video tag.

 Patterns exhibit some regenerative properties upon being damaged, but
 not full re-growth.

[Reproduce in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/growing_ca.ipynb#scrollTo=S5JRLGxX1dnX)

 The animation above shows three different models trained using the same
 settings. We let each of the models develop a pattern over 100 steps, then
 damage the final state in five different ways: by removing different halves
 of the formed pattern, and by cutting out a square from the center. Once
 again, we see that these models show quite different out-of-training mode
 behaviour. For example “the lizard” develops quite strong regenerative
 capabilities, without being explicitly trained for it!

 Since we trained our coupled system of cells to generate an attractor
 towards a target shape from a single cell, it was likely that these systems,
 once damaged, would generalize towards non-self-destructive reactions.
 That’s because the systems were trained to grow, stabilize, and never
 entirely self-destruct. Some of these systems might naturally gravitate
 towards regenerative capabilities, but nothing stops them from developing
 different behaviors such as explosive mitoses (uncontrolled growth),
 unresponsiveness to damage (overstabilization), or even self destruction,
 especially for the more severe types of damage.

 If we want our model to show more consistent and accurate regenerative
 capabilities, we can try to increase the basin of attraction for our target
 pattern - increase the space of cell configurations that naturally gravitate
 towards our target shape. We will do this by damaging a few pool-sampled
 states before each training step. The system now has to be capable of
 regenerating from states damaged by randomly placed erasing circles. Our
 hope is that this will generalize to regenerational capabilities from
 various types of damage.

 Your browser does not support the video tag.

 Damaging samples in the pool encourages the learning of robust
 regenerative qualities. Row 1 are samples from the pool, Row 2 are their
 respective states after iterating the model.

[Reproduce in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/growing_ca.ipynb#scrollTo=QeXZKb5v2gxj)

 The animation above shows training progress, which includes sample damage.
 We sample 8 states from the pool. Then we replace the highest-loss sample
 (top-left-most in the above) with the seed state, and damage the three
 lowest-loss (top-right-most) states by setting a random circular region
 within the pattern to zeros. The bottom row shows states after iteration
 from the respective top-most starting state. As in Experiment 2, the
 resulting states get injected back into the pool.

 Your browser does not support the video tag.

 Patterns exposed to damage during training exhibit astounding
 regenerative capabilities.

[Reproduce in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/growing_ca.ipynb#scrollTo=TDzJM69u4_8p)

 As we can see from the animation above, models that were exposed to damage
 during training are much more robust, including to types of damage not
 experienced in the training process (for instance rectangular damage as
 above).

Experiment 4: Rotating the perceptive field
-------------------------------------------

 As previously described, we model the cell’s perception of its neighbouring
 cells by estimating the gradients of state channels in x⃗\vec{x}x⃗ and
 y⃗\vec{y}y⃗ using Sobel filters. A convenient analogy is that each agent has
 two sensors (chemosensory receptors, for instance) pointing in orthogonal
 directions that can sense the gradients in the concentration of certain
 chemicals along the axis of the sensor. What happens if we rotate those
 sensors? We can do this by rotating the Sobel kernels.

[KxKy]=[cosθ−sinθsinθcosθ]∗[SobelxSobely] \begin{bmatrix} K\_x \\ K\_y \end{bmatrix} = \begin{bmatrix} \cos \theta &
 -\sin \theta \\ \sin \theta & \cos \theta \end{bmatrix} \* \begin{bmatrix}
 Sobel\_x \\ Sobel\_y \end{bmatrix} [KxKy]=[cosθsinθ−sinθcosθ]∗[SobelxSobely]

 This simple modification of the perceptive field produces rotated versions
 of the pattern for an angle of choosing without retraining as seen below.

![](figures/rotation.png)

 Rotating the axis along which the perception step computes gradients
 brings about rotated versions of the pattern.

[Reproduce in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/growing_ca.ipynb#scrollTo=1CVR9MeYnjuY)

 In a perfect world, not quantized by individual cells in a pixel-lattice,
 this would not be too surprising, as, after all, one would expect the
 perceived gradients in x⃗\vec{x}x⃗ and y⃗\vec{y}y⃗ to be invariant to the chosen
 angle - a simple change of frame of reference. However, it is important to
 note that things are not as simple in a pixel based model. Rotating pixel
 based graphics involves computing a mapping that’s not necessarily bijective
 and classically involves interpolating between pixels to achieve the desired
 result. This is because a single pixel, when rotated, will now likely
 overlap several pixels. The successful growth of patterns as above suggests
 a certain robustness to the underlying conditions outside of those
 experienced during training.

Related Work
------------

### CA and PDEs

 There exists an extensive body of literature that describes the various
 flavours of cellular automata and PDE systems, and their applications to
 modelling physical, biological or even social systems. Although it would be
 impossible to present a just overview of this field in a few lines, we will
 describe some prominent examples that inspired this work. Alan Turing
 introduced his famous Turing patterns back in 1952
 , suggesting how
 reaction-diffusion systems can be a valid model for chemical behaviors
 during morphogenesis. A particularly inspiring reaction-diffusion model that
 stood the test of time is the Gray-Scott model
 , which shows an extreme variety of
 behaviors controlled by just a few variables.

 Ever since von Neumann introduced CAs
  as models for self-replication they
 have captivated researchers’ minds, who observed extremely complex
 behaviours emerging from very simple rules. Likewise, the a broader audience
 outside of academia were seduced by CA’s life-like behaviours thanks to
 Conway’s Game of Life . Perhaps
 motivated in part by the proof that something as simple as the Rule 110 is
 Turing complete, Wolfram’s “*A New Kind of Science”*
 asks for a paradigm shift centered
 around the extensive usage of elementary computer programs such as CA as
 tools for understanding the world.

 More recently, several researchers generalized Conway’s Game of life to work
 on more continuous domains. We were particularly inspired by Rafler’s
 SmoothLife  and Chan’s Lenia
 , the latter of
 which also discovers and classifies entire species of “lifeforms”.

 A number of researchers have used evolutionary algorithms to find CA rules
 that reproduce predefined simple patterns
 .
 For example, J. Miller  proposed an
 experiment similar to ours, using evolutionary algorithms to design a CA
 rule that could build and regenerate the French flag, starting from a seed
 cell.

### Neural Networks and Self-Organisation

 The close relation between Convolutional Neural Networks and Cellular
 Automata has already been observed by a number of researchers
 . The
 connection is so strong it allowed us to build Neural CA models using
 components readily available in popular ML frameworks. Thus, using a
 different jargon, our Neural CA could potentially be named “Recurrent
 Residual Convolutional Networks with ‘per-pixel’ Dropout”.

 The Neural GPU
  offers
 a computational architecture very similar to ours, but applied in the
 context of learning multiplication and a sorting algorithm.

 Looking more broadly, we think that the concept of self-organisation is
 finding its way into mainstream machine learning with popularisation of
 Graph Neural Network  models.
 Typically, GNNs run a repeated computation across vertices of a (possibly
 dynamic) graph. Vertices communicate locally through graph edges, and
 aggregate global information required to perform the task over multiple
 rounds of message exchanges, just as atoms can be thought of as
 communicating with each other to produce the emergent properties of a
 molecule , or even points of a point
 cloud talk to their neighbors to figure out their global shape
 .

 Self-organization also appeared in fascinating contemporary work using more
 traditional dynamic graph networks, where the authors evolved
 Self-Assembling Agents to solve a variety of virtual tasks
 .

### Swarm Robotics

 One of the most remarkable demonstrations of the power of self-organisation
 is when it is applied to swarm modeling. Back in 1987, Reynolds’ Boids
  simulated the flocking behaviour of birds with
 just a tiny set of handcrafted rules. Nowadays, we can embed tiny robots
 with programs and test their collective behavior on physical agents, as
 demonstrated by work such as Mergeable Nervous Systems
  and Kilobots
 . To the best of our knowledge, programs
 embedded into swarm robots are currently designed by humans. We hope our
 work can serve as an inspiration for the field and encourage the design of
 collective behaviors through differentiable modeling.

Discussion
----------

### Embryogenetic Modeling

 Your browser does not support the video tag.

 Regeneration-capable 2-headed planarian, the creature that inspired this
 work

[Reproduce in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/growing_ca.ipynb#scrollTo=fQ1u2MqFy7Ni)

 This article describes a toy embryogenesis and regeneration model. This is a
 major direction for future work, with many applications in biology and
 beyond. In addition to the implications for understanding the evolution and
 control of regeneration, and harnessing this understanding for biomedical
 repair, there is the field of bioengineering. As the field transitions from
 synthetic biology of single cell collectives to a true synthetic morphology
 of novel living machines , it
 will be essential to develop strategies for programming system-level
 capabilities, such as anatomical homeostasis (regenerative repair). It has
 long been known that regenerative organisms can restore a specific
 anatomical pattern; however, more recently it’s been found that the target
 morphology is not hard coded by the DNA, but is maintained by a
 physiological circuit that stores a setpoint for this anatomical homeostasis
 . Techniques are
 now available for re-writing this setpoint, resulting for example
  in 2-headed flatworms
 that, when cut into pieces in plain water (with no more manipulations)
 result in subsequent generations of 2-headed regenerated worms (as shown
 above). It is essential to begin to develop models of the computational
 processes that store the system-level target state for swarm behavior
 , so that efficient strategies can be developed for rationally editing this
 information structure, resulting in desired large-scale outcomes (thus
 defeating the inverse problem that holds back regenerative medicine and many
 other advances).

### Engineering and machine learning

 The models described in this article run on the powerful GPU of a modern
 computer or a smartphone. Yet, let’s speculate about what a “more physical”
 implementation of such a system could look like. We can imagine it as a grid
 of tiny independent computers, simulating individual cells. Each of those
 computers would require approximately 10Kb of ROM to store the “cell
 genome”: neural network weights and the control code, and about 256 bytes of
 RAM for the cell state and intermediate activations. The cells must be able
 to communicate their 16-value state vectors to neighbors. Each cell would
 also require an RGB-diode to display the color of the pixel it represents. A
 single cell update would require about 10k multiply-add operations and does
 not have to be synchronised across the grid. We propose that cells might
 wait for random time intervals between updates. The system described above
 is uniform and decentralised. Yet, our method provides a way to program it
 to reach the predefined global state, and recover this state in case of
 multi-element failures and restarts. We therefore conjecture this kind of
 modeling may be used for designing reliable, self-organising agents. On the
 more theoretical machine learning front, we show an instance of a
 decentralized model able to accomplish remarkably complex tasks. We believe
 this direction to be opposite to the more traditional global modeling used
 in the majority of contemporary work in the deep learning field, and we hope
 this work to be an inspiration to explore more decentralized learning
 modeling.

![](images/multiple-pages.svg)

 This article is part of the
 [Differentiable Self-organizing Systems Thread](/2020/selforg/),
 an experimental format collecting invited short articles delving into
 differentiable self-organizing systems, interspersed with critical
 commentary from several experts in adjacent fields.

[Differentiable Self-organizing Systems Thread](/2020/selforg/)
[Self-classifying MNIST Digits](/2020/selforg/mnist/)

Growing Neural Cellular Automata

Alexander Mordvintsev
Eyvind Niklasson
Michael Levin

Summary: Training an end-to-end differentiable, self-organising cellular automata model of morphogenesis, able to both grow and regenerate specific patterns.

Top-down models in biology: explanation and control of complex living systems above the molecular level

Re-membering the body: applications of computational neuroscience to the top-down control of regeneration of limbs and other complex organs

Transmembrane voltage potential controls embryonic eye patterning in Xenopus laevis

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

The chemical basis of morphogenesis

Complex Patterns in a Simple System

MATHEMATICAL GAMES

A New Kind of Science

Lenia: Biology of Artificial Life

Evolving a Self-Repairing, Self-Regulating, French Flag Organism

Learning Cellular Automaton Dynamics with Neural Networks

Cellular automata as convolutional neural networks

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Dynamic Graph CNN for Learning on Point Clouds

Learning to Control Self- Assembling Morphologies: A Study of Generalization via Modularity

Flocks, Herds and Schools: A Distributed Behavioral Model

Mergeable nervous systems for robots

Kilobot: A low cost scalable robot system for collective behaviors

What Bodies Think About: Bioelectric Computation Outside the Nervous System

A scalable pipeline for designing reconfigurable organisms

Perspective: The promise of multi-cellular engineered living systems

Physiological inputs regulate species-specific anatomy during embryogenesis and regeneration

Long-range neural and gap junction protein-mediated cues control polarity during planarian regeneration

Long-Term, Stochastic Editing of Regenerative Anatomy via Targeting Endogenous Bioelectric Gradients

Pattern Regeneration in Coupled Networks

Bioelectrical control of positional information in development and regeneration: A review of conceptual and computational advances

Modeling Cell Migration in a Simulated Bioelectrical Signaling Network for Anatomical Regeneration

Investigating the effects of noise on a cell-to-cell communication mechanism for structure regeneration

Social Intelligence

Inceptionism: Going deeper into neural networks