Zoom In: An Introduction to Circuits

Chris Olah
Nick Cammarata
Ludwig Schubert
Gabriel Goh
Michael Petrov
Shan Carter

Abstract: This article is part of the Circuits thread,  an experimental format collecting invited short articles and critical  commentary delving into the inner workings of neural networks.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

[Circuits Thread](https://distill.pub/2020/circuits/)

### Contents

[Three Speculative Claims](#three-speculative-claims)

[Claim 1: Features](#claim-1)

* [Example 1: Curve Detectors](#claim-1-curves)

* [Example 2: High-Low Frequency Detectors](#claim-1-hilo)

* [Example 3: Pose-Invariant Dog Head Detector](#claim-1-dog)

* [Polysemantic Neurons](#claim-1-polysemantic)

[Claim 2: Circuits](#claim-2)

* [Circuit 1: Curve Detectors](#claim-2-curves)

* [Circuit 2: Oriented Dog Head Detection](#claim-2-dog)

* [Circuit 3: Cars in Superposition](#claim-2-superposition)

* [Circuit Motifs](#claim-2-motifs)

[Claim 3: Universality](#claim-3)

[Interpretability as a Natural Science](#natural-science)

[Closing Thoughts](#closing)

 Many important transition points in the history of science have

been moments when science "zoomed in."

 At these points, we develop a visualization or tool that allows us

 to see the world in a new level of detail, and a new field of science

develops to study the world through this lens.

 For example, microscopes let us see cells, leading to cellular

biology. Science zoomed in. Several techniques including x-ray

crystallography let us see DNA, leading to the molecular revolution.

Science zoomed in. Atomic theory. Subatomic particles. Neuroscience.

Science zoomed in.

 These transitions weren't just a change in precision: they were

qualitative changes in what the objects of scientific inquiry are.

 For example, cellular biology isn't just more careful zoology.

 It's a new kind of inquiry that dramatically shifts what we can

understand.

 The famous examples of this phenomenon happened at a very large

scale,

 but it can also be the more modest shift of a small research

community realizing they can now study their topic in a finer grained

level of detail.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/micrographia2.jpg)

 Hooke's Micrographia revealed a rich microscopic world as seen

 through a microscope, including the initial discovery of cells.

Images from the National Library of Wales.

Just as the early microscope hinted at a new world of cells and

 microorganisms, visualizations of artificial neural networks have

revealed tantalizing hints and glimpses of a rich inner world within our

 models (e.g. ).

 What kind of picture of neural networks would emerge?

 In contrast to the typical picture of neural networks as a black

 box, we've been surprised how approachable the network is on this

scale.

 Not only do neurons seem understandable (even ones that

initially seemed inscrutable), but the "circuits" of connections between

 them seem to be meaningful algorithms corresponding to facts about the

world.

 You can watch a circle detector be assembled from curves.

 You can see a dog head be assembled from eyes, snout, fur and

tongue.

 You can observe how a car is composed from wheels and windows.

 You can even find circuits implementing simple logic: cases

where the network implements AND, OR or XOR over high-level visual

features.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/deepdream.jpg)

 This introductory essay offers a high-level overview of our

thinking and some of the working principles that we've found useful in

this line of research.

 In future articles, we and our collaborators will publish

detailed explorations of this inner world.

 But the truth is that we've only scratched the surface of

understanding a single vision model.

 If these questions resonate with you, you are welcome to join us

 and our collaborators in the Circuits project, an open scientific

collaboration hosted on the [Distill slack](http://slack.distill.pub/).

---

Three Speculative Claims

------------------------

One of the earliest articulations of something approaching

modern cell theory was three claims by Theodor Schwann — who you may

know for Schwann cells — in 1839:

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/schwann-book.jpg)

#### Schwann's Claims about Cells

Claim 1

Claim 2

Claim 3

 Cells form by free-cell formation, similar to the formation of crystals.

 This translation/summarization of Schwann's claims can be

found in many biology texts; we were unable to determine what the

The first two of these claims are likely familiar, persisting

in modern cellular theory. The third is likely not familiar, since it

turned out to be horribly wrong.

We believe there's a lot of value in articulating a strong

version of something one may believe to be true, even if it might be

false like Schwann's third claim. In this spirit, we offer three claims

about neural networks. They are intended both as empirical claims about

the nature of neural networks, and also as normative claims about how

it's useful to understand them.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/atlas-book-crop.png)

#### Three Speculative Claims about Neural Networks

Claim 1: Features

 Features are the fundamental unit of neural networks.

 They correspond to directions.

 By "direction" we mean a linear combination of neurons

in a layer.

 You can think of this as a direction vector in the

vector space of activations of neurons in a given layer.

 Often, we find it most helpful to talk about individual

neurons,

 but we'll see that there are some cases where other

combinations are a more useful way to analyze networks — especially when

 neurons are "polysemantic."

 (See the [glossary](#glossary-direction) for a detailed definition.)

 These features can be rigorously studied and understood.

Claim 2: Circuits

 Features are connected by weights, forming circuits.

 A "circuit" is a computational subgraph of a neural network.

 network.

 (See the [glossary](#glossary-circuit) for a detailed definition.)

 These circuits can also be rigorously studied and understood.

Claim 3: Universality

 Analogous features and circuits form across models and tasks.

These claims are deliberately speculative.

 They also aren't totally novel: claims along the lines of (1)

and (3) have been suggested before, as we'll discuss in more depth

below.

 But we believe these claims are important to consider because,

if true, they could form the basis of a new "zoomed in" field of

 interpretability. In the following sections, we'll discuss each

one individually and present some of the evidence that has led us to

believe they might be true.

---

Claim 1: Features

-----------------

 .claim-quote {

 border-left: 1px solid #CCC;

 padding-left: 20px;

 color: #555;

 margin-bottom: 30px;

 }

 Features are the fundamental unit of neural networks.

 They correspond to directions. They can be rigorously studied and understood.

 We believe that neural networks consist of meaningful,

understandable features. Early layers contain features like edge or

curve detectors, while later layers have features like floppy ear

detectors or wheel detectors.

 The community is divided on whether this is true.

 While many researchers treat the existence of meaningful neurons

 as an almost trivial fact — there's even a small literature studying

them  —

 many others are deeply skeptical and believe that past cases of neurons

 that seemed to track meaningful latent variables were mistaken .

 The community disagreement on

meaningful features is hard to pin down, and only partially expressed in

 the literature. Foundational descriptions of deep learning often

describe neural networks as detecting a hierarchy of meaningful features

 At the same time, a more skeptical parallel literature has developed

suggesting that neural networks primarily or only focus on texture,

 Although many of these papers express a highly nuanced view, that isn't

 always how they've been understood. A number of media articles have

been written embracing strong versions of these views, and we

anecdotally find that the belief that neural networks don't understand

anything more than texture is quite common. Finally, people often have

trouble articulating their exact views, because they don't have clear

language for articulating nuances between "a texture detector highly

correlated with an object" and "an object detector."

 Nevertheless, thousands of hours of studying individual neurons

have led us to believe the typical case is that neurons (or in some

cases, other directions in the vector space of neuron activations) are

understandable.

 Of course, being understandable doesn't mean being simple or

easily understandable.

 Many neurons are initially mysterious and don't follow our a

priori guesses of what features might exist!

 However, our experience is that there's usually a simple

explanation behind these neurons, and that they're actually doing

something quite natural.

 For example, we were initially confused by high-low frequency

detectors (discussed below) but in retrospect, they are simple and

elegant.

 This introductory essay will only give an overview of a couple

examples we think are illustrative, but it will be followed both by deep

 dives carefully characterizing individual features, and broad overviews

 sketching out all the features we understand to exist.

 Regardless of whether we're correct or mistaken about meaningful

 features,

 we believe this is an important question for the community to

resolve.

 We hope that introducing several specific carefully explored

examples of seemingly understandable features will help advance the

dialogue.

### Example 1: Curve Detectors

 Curve detecting neurons can be found in every non-trivial vision

 model we've carefully examined.

 These units are interesting because they straddle the boundary

between features the community broadly agrees exist (e.g. edge

detectors) and features for which there's significant skepticism (e.g.

high-level features such as ears, automotives, and faces).

 an early layer of InceptionV1. These units responded to curved lines

and boundaries with a radius of around 60 pixels. They are also slightly

 additionally excited by perpendicular lines along the boundary of the

curve, and prefer the two sides of the curve to be different colors.

Curve detectors are found in families of units, with each

member of the family detecting the same curve feature in a different

orientation. Together, they jointly span the full range of orientations.

 It's important to distinguish curve detectors from other units

which may seem superficially similar.

 In particular, there are many units which use curves to detect a

 curved sub-component (e.g. circles, spirals, S-curves, hourglass shape,

 3d curvature, …).

 There are also units which respond to curve related shapes like

lines or sharp corners.

 We do not consider these units to be curve detectors.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/curves.png)

 But are these "curve detectors" really detecting curves?

 but the summary is that we think the evidence is quite strong.

 We offer seven arguments, outlined below.

 It's worth noting that none of these arguments are curve

specific: they're a useful, general toolkit for testing our

understanding of other features as well.

 Several of these arguments — dataset examples, synthetic

examples, and tuning curves — are classic methods from visual

neuroscience (e.g. ).

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/arg-fv.png)

#### Argument 1: Feature Visualization

Optimizing the input to cause curve detectors to fire

reliably produces curves. This establishes a causal link, since

everything in the resulting image was added to cause the neuron to fire

more.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/arg-data.png)

#### Argument 2: Dataset Examples

The ImageNet images that cause these neurons to

strongly fire are reliably curves in the expected orientation. The

images that cause them to fire moderately are generally less perfect

curves or curves off orientation.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/arg-synthetic.png)

#### Argument 3: Synthetic Examples

Curve detectors respond as expected to a range of

synthetic curves images created with varying orientations, curvatures,

and backgrounds. They fire only near the expected orientation, and do

not fire strongly for straight lines or sharp corners.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/arg-tune.png)

#### Argument 4: Joint Tuning

If we take dataset examples that cause a neuron to fire

 and rotate them, they gradually stop firing and the curve detectors in

the next orientation begins firing. This shows that they detect rotated

versions of the same thing. Together, they tile the full 360 degrees of

potential orientations.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/arg-weights.png)

#### Argument 5: Feature implementation

 (circuit-based argument)

By looking at the circuit constructing the curve

detectors, we can read a curve detection algorithm off of the weights.

We also don't see anything suggestive of a second alternative cause of

firing, although there are many smaller weights we don't understand the

role of.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/arg-use.png)

#### Argument 6: Feature use

 (circuit-based argument)

The downstream clients of curve detectors are features

that naturally involve curves (e.g. circles, 3d curvature, spirals…).

The curve detectors are used by these clients in the expected manner.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/arg-hand.png)

#### Argument 7: Handwritten Circuits

 (circuit-based argument)

Based on our understanding of how curve detectors are

implemented,

 we can do a cleanroom reimplementation,

 hand setting all weights to reimplement curve detection.

 These weights are an understandable curve detection

algorithm, and significantly mimic the original curve detectors.

The above arguments don't fully exclude the possibility of some

 rare secondary case where curve detectors fire for a different kind of

 stimulus. But they do seem to establish

 that (1) curves cause these neurons to fire,

 (2) each unit responds to curves at different angular

orientations,

 and (3) if there are other stimuli that cause them to fire those

 stimuli are rare or cause weaker activations.

 More generally, these arguments seem to meet the evidentiary

standards we understand to be used in neuroscience, which has

established traditions and institutional knowledge of how to evaluate

such claims.

### Example 2: High-Low Frequency Detectors

 Curve detectors are an intuitive type of feature — the kind of

feature one might guess exists in neural networks a priori.

 Given that they're present, it's not surprising we can

understand them.

 But what about features that aren't intuitive? Can we also

understand those?

 We believe so.

 and once you understand what they're doing, they're quite simple. They

look for low-frequency patterns on one side of their receptive field,

and high-frequency patterns on the other side. Like curve detectors,

high-low frequency detectors are found in families of features that look

 for the same thing in different orientations.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/high-low.png)

Why are high-low frequency detectors useful to the network?

They seem to be one of several heuristics for detecting the boundaries

of objects, especially when the background is out of focus. In a later

 (One hope some researchers have for interpretability is that

understanding models will be able to teach us better abstractions for

thinking about the world .

 High-low frequency detectors are, perhaps, an example of a small

success in this: a natural, useful visual feature that we didn't

anticipate in advance.)

All seven of the techniques we used to interrogate curve

neurons can also be used to study high-low frequency neurons with some

tweaking — for instance, rendering synthetic high-low frequency

examples. Again we believe these arguments collectively provide strong

support for the idea that these really are a family of high-low

frequency contrast detectors.

### Example 3: Pose-Invariant Dog Head Detector

Both curve detectors and high-low frequency detectors are

low-level visual features, found in the early layers of InceptionV1.

What about more complex, high-level features?

Let's consider this unit which we believe to be a

pose-invariant dog detector. As with any neuron, we can create a feature

 visualization and collect dataset examples. If you look at the feature

visualization, the geometry is… not possible, but very informative about

 what it's looking for and the dataset examples validate it.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/dog-pose.png)

 It's worth noting that the combination of feature visualization

and dataset examples alone are already quite a strong argument.

 Feature visualization establishes a causal link, while dataset

examples test the neuron's use in practice and whether there are a

second type of stimuli that it reacts to.

 But we can bring all our other approaches to analyzing a neuron

to bear again.

 For example, we can use a 3D model to generate synthetic dog head

images from different angles.

 At the same time, some of the approaches we've emphasized so far

 become a lot of effort for these higher-level, more abstract features.

 Thankfully, our circuit-based arguments — which we'll discuss

more soon — will continue to be easy to apply, and give us really

powerful tools for understanding and testing high-level features that

don't require a lot of effort.

### Polysemantic Neurons

This essay may be giving you an overly rosy picture: perhaps

every neuron yields a nice, human-understandable concept if one

seriously investigates it?

 Alas, this is not the case.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/polysemantic.png)

To be clear, this neuron isn't responding to some commonality

of cars and cat faces. Feature visualization shows us that it's looking

for the eyes and whiskers of a cat, for furry legs, and for shiny fronts

 of cars — not some subtle shared feature.

 We can still study such features, characterizing each different

case they fire, and reason about their circuits to some extent.

 Despite this, polysemantic neurons are a major challenge for the

 circuits agenda, significantly limiting our ability to reason about

neural networks.

 Why are polysemantic neurons so challenging? If one neuron

with five different meanings connects to another neuron with five

different meanings, that's effectively 25 connections that can't be

considered individually.

 Our hope is that it may be possible to resolve polysemantic

neurons,

 perhaps by "unfolding" a network to turn polysemantic neurons

into pure features, or training networks to not exhibit polysemanticity

in the first place.

 This is essentially the problem studied in the literature of

disentangling representations, although at present that literature tends

 to focus on known features in the latent spaces of generative models.

 One natural question to ask is why do polysemantic neurons form?

 In the next section, we'll see that they seem to result from a

phenomenon we call "superposition" where a circuit spreads a feature

across many neurons, presumably to pack more features into the limited

number of neurons it has available.

---

Claim 2: Circuits

-----------------

 Features are connected by weights, forming circuits.

 These circuits can also be rigorously studied and understood.

 To explore this, we find it helpful to study circuits:

The remarkable thing is how tractable and meaningful these

circuits seem to be as objects of study. When we began looking, we

expected to find something quite messy. Instead, we've found beautiful

 to them. Once you understand what features they're connecting together,

 the individual floating point number weights in your neural network

Let's consider some examples.

### Circuit 1: Curve Detectors

In the previous section, we discussed curve detectors, a family

 of units detecting curves in different angular orientations. In this

section, we'll explore how curve detectors are implemented from earlier

features and connect to the rest of the model.

 Curve detectors are primarily implemented from earlier, less

sophisticated curve detectors and line detectors. These curve detectors

are used in the next layer to create 3D geometry and complex shape

detectors. Of course, there's a long tail of smaller connections to

other features, but this seems to be the primary story.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/curve-circuit.png)

Let's focus even more and look at how a single early curve

detector connects to a more sophisticated curve detector in the same

orientation.

In this case, our model is implementing a 5x5 convolution,

so the weights linking these two neurons are a 5x5 set of weights, which

 can be positive or negative.

 Many of the neurons discussed in this article, including curve

detectors, live in branches of InceptionV1 that are structured as a 1x1

convolution that reduce the number of channels to a small bottleneck

followed by a 3x3 or 5x5 convolution. The weights we present in this

essay are the multiplied out version of the 1x1 and larger conv weights.

 We think it's often useful to view this as a single low-rank weight

matrix, but this technically does ignore one ReLU non-linearity.

 A positive weight means that if the earlier neuron fires in that

 position, it excites the late neuron. Conversely a negative weight

would mean that it inhibits it.

What we see are strong positive weights, arranged in the

shape of the curve detector. We can think of this as meaning that, at

each point along the curve, our curve detector is looking for a "tangent

 curve" using the earlier curve detector.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/curve-weights-a.png)

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/curve-weights-b.png)

This can be interpreted as looking for "tangent curves"

at each point along the curve.

This is true for every pair of early and full curve detectors

in similar orientations. At every point along the curve, it detects the

curve in a similar orientation. Similarly, curves in the opposite

orientation are inhibitory at every point along the curve.

Curve detectors are excited by earlier detectors

 in **similar orientations**…

… and inhibited by earlier detectors in

 **opposing orientations**.

And the structure gets richer the closer you look. For

example, if you look at an early curve detector and full curve detector

in similar, but not exactly the same orientation you can often see it

have stronger positive weights on the side of the curve it is more

aligned with.

It's also worth noting how the weights rotate with the

orientation of the curve detector. The symmetry of the problem is

reflected as a symmetry in the weights. We call circuits with exhibiting

 this phenomenon an "equivariant circuit", and will discuss it in depth

in a [later article](https://distill.pub/2020/circuits/equivariance/).

### Circuit 2: Oriented Dog Head Detection

The curve detector circuit is a low-level circuit and only

spans two layers. In this section, we'll discuss a higher-level circuit

spanning across four layers. This circuit will also teach us about how

neural networks implement sophisticated invariances.

Remember that a huge part of what an ImageNet model has to

do is tell apart different animals. In particular, it has to distinguish

 between a hundred different species of dogs! And so, unsurprisingly, it

 develops a large number of neurons dedicated to recognizing dog related

 features, including heads.

Within this "dog recognition" system, one circuit strikes

us as particularly interesting: a collection of neurons that handle dog

heads facing to the left and dog heads facing to the right. Over three

layers, the network maintains two mirrored pathways, detecting analogous

 units facing to the left and to the right. At each step, these pathways

 try to inhibit each other, sharpening the contrast. Finally, it creates

 invariant neurons which respond to both pathways.

 @media (min-width: 1000px){

 .dog-circuit-figure {

 }

 }

 @media (max-width: 1000px){

 .dog-circuit-figure {

 grid-column-start: screen-start; grid-column-end: screen-end;

 padding-left: 20px;

 padding-right: 20px;

 }

 }

We call this pattern "unioning over cases". The network

separately detects two cases (left and right) and then takes a union

This circuit is striking because the network could have

easily done something much less sophisticated.

 It could easily create invariant neurons by not caring very much

 about where the eyes, fur and snout went, and just looking for a jumble

 of them together.

 But instead, the network has learned to carve apart the left and

 right cases and handle them separately. We're somewhat surprised that

gradient descent could learn to do this!To

 be clear, there are also more direct pathways by which various

constituents of heads influence these later head detectors, without

going through the left and right pathways

But this summary of the circuit is only scratching the surface

of what is going on. Every connection between neurons is a convolution,

so we can also look at where an input neuron excites the the next one.

And the models tends to be doing what you might have optimistically

hoped. For example, consider these "head with neck" units. The head is

only detected on the correct side:

 The union step is also interesting to look at the details of.

 The network doesn't indiscriminately respond to the heads into

the two orientations:

 the regions of excitation extend from the center in different

directions depending on orientation, allowing snouts to converge in to

the same point.

There's a lot more to say about this circuit, so we plan to

return to it in a future article and analyze it in depth, including

testing our theory of the circuit by editing the weights.

### Circuit 3: Cars in Superposition

In `mixed4c`, a mid-late layer of InceptionV1, there

 is a car detecting neuron. Using features from the previous layers, it

looks for wheels at the bottom of its convolutional window, and windows

at the top.

But then the model does something surprising. Rather than

create another pure car detector at the next layer, it spreads its car

feature over a number of neurons that seem to primarily be doing

something else — in particular, dog detectors.

This circuit suggests that polysemantic neurons are, in some

sense, deliberate. That is, you could imagine a world where the process

of detecting cars and dogs was deeply intertwined in the model for some

reason, and as a result polysemantic neurons were difficult to avoid.

But what we're seeing here is that the model had a "pure neuron" and

then mixed it up with other features.

We call this phenomenon superposition.

Why would it do such a thing? We believe superposition

allows the model to use fewer neurons, conserving them for more

important tasks. As long as cars and dogs don't co-occur, the model can

accurately retrieve the dog feature in a later layer, allowing it to

store the feature without dedicating a neuron.Fundamentally,

 this is a property of the geometry of high-dimensional spaces, which

only allow for n orthogonal vectors, but exponentially many almost

orthogonal vectors.

### Circuit Motifs

As we've studied circuits throughout InceptionV1 and other models,

 we've seen the same abstract patterns over and over.

 Unioning over cases, as we saw with the pose-invariant dog head detector.

 Superposition, as we saw with the car detector.

 We think it's quite likely that studying motifs will be

important in understanding the circuits of artificial neural networks.

In the long run, it may be more important than the study of individual

circuits.

 At the same time, we expect investigations of motifs to be well

served by us first building up a solid foundation of well understood

circuits first.

---

Claim 3: Universality

---------------------

Analogous features and circuits form across models and tasks.

 It's a widely accepted fact that the first layer of vision

models trained on natural images will learn Gabor filters.

 Once you accept that there are meaningful features in later

layers, would it really be surprising for the same features to also form

 in layers beyond the first one?

 And once you believe there are analogous features in multiple

layers, wouldn't it be natural for them to connect in the same ways?

 Universality (or "convergent learning") of features has been suggested before.

 and that they learn similar representations at hidden layers .

 This work seems highly suggestive, but there are alternative

explanations to analogous features forming.

 For example, one could imagine two features — such as a fur

texture detector and a sophisticated dog body detector — being highly

correlated despite being importantly different features.

 Adopting the meaningful feature-skeptic perspective, it doesn't

seem definitive.

 Ideally, one would like to characterize several features and

then rigorously demonstrate that those features — and not just

correlated ones — are forming across many models.

 Then, to further establish that analogous circuits form, one

would want to find analogous features over several layers of multiple

models and show that the same weight structure forms between them in

each model.

 Unfortunately, the only evidence we can offer today is

anecdotal:

 we simply have not yet invested enough in the comparative study

of features and circuits to give confident answers.

 With that said, we have observed that a couple low-level

features seem to form across a variety of vision model architectures

(including AlexNet, InceptionV1, InceptionV3, and residual networks) and

 in models trained on Places365 instead of ImageNet. We've also observed

 them repeatedly form in vanilla conv nets trained from scratch on

ImageNet.

 .diagram-universality {

 display: grid;

 grid-column: text-start / page-end;

 grid-gap: 1rem;

 grid-template-columns: 1fr;

 justify-content: center;

 }

 @media (min-width: 1180px) {

 .diagram-universality {

 grid-column: page;

 grid-template-columns: 528px 448px;

 }

 .diagram-universality > figure:last-of-type .info {

 display: none;

 }

 .diagram-universality > figure:last-of-type li {

 grid-template-columns: 1fr;

 }

 .diagram-universality > figure:last-of-type > figcaption {

 padding-left: unset;

 }

 }

 .diagram-universality a {

 border-bottom: none;

 display: inline;

 }

 .diagram-universality a:hover {

 border-bottom: none;

 }

 .diagram-universality a:hover img {

 --filter: brightness(80%);

 }

 .diagram-universality h3 {

 margin-top: 0;

 }

 .diagram-universality h4 {

 margin: 4px 0;

 }

 .diagram-universality > figure {

 margin: 0;

 }

 .diagram-universality > figure > figcaption {

 padding-left: 9rem;

 }

 .diagram-universality ul {

 list-style: none;

 padding-left: unset;

 }

 .diagram-universality li {

 display: grid;

 grid-template-columns: 9rem 1fr;

 }

 .diagram-universality li .images {

 display: grid;

 grid-template-columns: repeat( auto-fill, 88px);

 grid-gap: 8px;

 }

 .diagram-universality img {

 width: 88px;

 height: 88px;

 background-color: gray;

 border-radius: 4px;

 object-fit: none;

 }

### Curve detectors

* #### AlexNet

Krizhevsky et al.

* #### InceptionV1

Szegedy et al.

* #### VGG19

Simonyan et al.

* #### ResNetV2-50

He et al.

### High-Low Frequency detectors

* #### AlexNet

Krizhevsky et al.

* #### InceptionV1

Szegedy et al.

* #### VGG19

Simonyan et al.

* #### ResNetV2-50

He et al.

 These results have led us to suspect that the universality

hypothesis is likely true, but further work will be needed to understand

 if the apparent universality of some low-level vision features is the

exception or the rule.

 If it turns out that the universality hypothesis is broadly true

 in neural networks,

 it will be tempting to speculate: might biological neural

networks also learn similar features?

 Researchers working at the intersection of neuroscience and deep

 learning have already shown that the units in artificial vision models

can be useful for modeling biological neurons .

 And some of the features we've discovered in artificial neural

networks, such as curve detectors, are also believed to exist in

biological neural networks (e.g. ).

 This seems like significant cause for optimism.

 One particularly exciting possibility might be if artificial

neural networks could predict features which were previously unknown but

 could then be found in biology.

 (Some neuroscientists we have spoken to have suggested that

high-low frequency detectors might be a candidate for this.)

 If such a prediction could be made, it would be extremely strong

 evidence for the universality hypothesis.

 Focusing on the study of circuits, is universality really

necessary?

 Unlike the first two claims, it wouldn't be completely fatal

to circuits research if this claim turned out to be false. But it does

greatly inform what kind of research makes sense. We introduced circuits

 as a kind of "cellular biology of deep learning." But imagine a world

where every species had cells with a completely different set of

organelles and proteins. Would it still make sense to study cells in

general, or would we limit ourselves to the narrow study of a few kinds

of particularly important species of cells? Similarly, imagine the study

 of anatomy in a world where every species of animal had a completely

unrelated anatomy: would we seriously study anything other than humans

and a couple domestic animals?

In the same way, the universality hypothesis determines

what form of circuits research makes sense. If it was true in the

strongest sense, one could imagine a kind of "periodic table of visual

features" which we observe and catalogue across models. On the other

hand, if it was mostly false, we would need to focus on a handful of

models of particular societal importance and hope they stop changing

every year. There might also be in between worlds, where some lessons

transfer between models but others need to be learned from scratch.

---

Interpretability as a Natural Science

-------------------------------------

*The Structure of Scientific Revolutions* by Thomas Kuhn

 is a classic text on the history and sociology of science.

 In it, Kuhn distinguishes between "normal science" in which a

scientific community has a paradigm, and "extraordinary science" in

which a community lacks a paradigm, either because it never had one or

because it was weakened by crisis.

 It's worth noting that "extraordinary science" is not a

desirable state: it's a period where researchers struggle to be

productive.

 There isn't consensus on what the objects of study are, what

methods we should use to answer them, or how to evaluate research

results.

 To quote a recent interview with Ian Goodfellow: "For

interpretability, I don't think we even have the right definitions."

 One particularly challenging aspect of being in a

pre-paradigmatic field is that there isn't a shared sense of how to

evaluate work in interpretability.

 There are two common proposals for dealing with this, drawing on

 the standards of adjacent fields. Some researchers, especially those

with a deep learning background, want an "interpretability benchmark"

which can evaluate how effective an interpretability method is. Other

researchers with an HCI background may wish to evaluate interpretability

 methods through user studies.

 But interpretability could also borrow from a third paradigm:

natural science.

 In this view, neural networks are an object of empirical

investigation, perhaps similar to an organism in biology. Such work

would try to make empirical claims about a given network, which could be

 held to the standard of falsifiability.

 To be clear, we do see researchers who take more of this

natural science approach, especially in earlier interpretability

research. It just seems less common right now.

 Especially given that there's so much adjacent ML work which

does adopt this frame!

 One reason might be that it's very difficult to make robustly

true statements about the behavior of a neural network as a whole.

 They're incredibly complicated objects.

 It's also hard to formalize what the interesting empirical

statements about them would, exactly, be.

 And so we often get standards of evaluations more targeted at

whether an interpretability method is useful rather than whether we're

learning true statements.

 Circuits sidestep these challenges by focusing on tiny subgraphs

 of a neural network for which rigorous empirical investigation is

tractable.

 They're very much falsifiable: for example, if you understand a

circuit, you should be able to predict what will change if you edit the

weights.

 In fact, for small enough circuits, statements about their

behavior become questions of mathematical reasoning.

 Of course, the cost of this rigor is that statements about

circuits are much smaller in scope than overall model behavior.

 But it seems like, with sufficient effort, statements about

model behavior could be broken down into statements about circuits.

 If so, perhaps circuits could act as a kind of epistemic

foundation for interpretability.

---

Closing Thoughts

----------------

We take it for granted that the microscope is an important

scientific instrument. It's practically a symbol of science. But this

wasn't always the case, and microscopes didn't initially take off as a

scientific tool. In fact, they seem to have languished for around fifty

 That this research is too qualitative.

 That it isn't scientific.

 The discovery of cells was a qualitative research result.

 That didn't stop it from changing the world.

![](Zoom%20In%20An%20Introduction%20to%20Circuits_files/multiple-pages.svg)

 This article is part of the Circuits thread, a collection

of short articles and commentary by an open scientific collaboration

delving into the inner workings of neural networks.

[Circuits Thread](https://distill.pub/2020/circuits/)

Micrographia:
 or Some Physiological Descriptions of Minute Bodies Made by Magnifying
Glasses. With Observations and Inquiries Thereupon

Visualizing and understanding recurrent networks

Visualizing higher-layer features of a deep network

Feature Visualization

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Visualizing and understanding convolutional networks

Interpretable Explanations of Black Boxes by Meaningful Perturbation

PatternNet and PatternLRP--Improving the interpretability of neural networks

Visualizing and Measuring the Geometry of BERT

Activation atlas

Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations

Distributed representations of words and phrases and their compositionality

Learning to generate reviews and discovering sentiment

Object detectors emerge in deep scene cnns

Network Dissection: Quantifying Interpretability of Deep Visual Representations

On Interpretability and Feature Representations: An Analysis of the Sentiment Neuron

Measuring the tendency of CNNs to Learn Surface Statistical Regularities

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Approximating cnns with bag-of-local-features models works surprisingly well on imagenet

Adversarial examples are not bugs, they are features

On the importance of single directions for generalization

Deep learning

Going deeper with convolutions

Receptive fields, binocular interaction and functional architecture in the cat's visual cortex

Using Artificial Intelligence to Augment Human Intelligence

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

An introduction to systems biology: design principles of biological circuits

Convergent learning: Do different neural networks learn the same representations?

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Similarity of neural network representations revisited

ImageNet Classification with Deep Convolutional Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Deep Residual Learning for Image Recognition

Performance-optimized hierarchical models predict neural responses in higher visual cortex

Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream

Seeing it all: Convolutional network layers map the function of the human visual system

Discrete neural clusters encode orientation, curvature and corners in macaque V4

Shape representation in area V4: position-specific tuning for boundary conformation

The structure of scientific revolutions

Ian Goodfellow: Generative Adversarial Networks

The Building Blocks of Interpretability


Differentiable Image Parameterizations

Alexander Mordvintsev
Nicola Pezzotti
Ludwig Schubert
Chris Olah

Abstract: Neural networks trained to classify images have a remarkable — and surprising! — capacity to generate images.     Techniques such as DeepDream , style transfer, and feature visualization  leverage this capacity as a powerful tool for exploring the inner  workings of neural networks, and to fuel a small artistic movement based  on neural art.

 Techniques such as DeepDream , style transfer, and feature visualization

 leverage this capacity as a powerful tool for exploring the inner

workings of neural networks, and to fuel a small artistic movement based

 on neural art.

 All these techniques work in roughly the same way.

 Neural networks used in computer vision have a rich internal

representation of the images they look at.

 We can use this representation to describe the properties we want an

 image to have (e.g. style), and then optimize the input image to have

those properties.

 This kind of optimization is possible because the networks are

differentiable with respect to their inputs: we can slightly tweak the

image to better fit the desired properties, and then iteratively apply

such tweaks in gradient descent.

 Typically, we parameterize the input image as the RGB values of each

 pixel, but that isn't the only way.

 As long as the mapping from parameters to images is differentiable,

we can still optimize alternative parameterizations with gradient

descent.

[1](#figure-differentiable-parameterizations):

 As long as an

image parameterization

is differentiable, we can backpropagate

(    )

through it.

MappingParametersimage/RGB spaceLossFunction

 Differentiable image parameterizations invite us to ask "what kind

of image generation process can we backpropagate through?"

 The answer is quite a lot, and some of the more exotic possibilities

 can create a wide range of interesting effects, including 3D neural

art, images with transparency, and aligned interpolation.

 Previous work using specific unusual image parameterizations

 has shown exciting results — we think that zooming out and looking at

this area as a whole suggests there's even more potential.

### Why Does Parameterization Matter?

 It may seem surprising that changing the parameterization of an

optimization problem can significantly change the result, despite the

objective function that is actually being optimized remaining the same.

 We see four reasons why the choice of parameterization can have a

significant effect:

**(1) - Improved Optimization** -

Transforming the input to make an optimization problem easier — a

technique called "preconditioning" — is a staple of optimization.

 Preconditioning is most often presented as a transformation of the gradient

 (usually multiplying it by a positive definite "preconditioner" matrix).

**(2) - Basins of Attraction** -

When we optimize the input to a neural network, there are often many

different solutions, corresponding to different local minima.

The probability of our optimization process falling into any particular

local minima is controlled by its basin of attraction (i.e., the region

of the optimization landscape under the influence of the minimum).

Changing the parameterization of an optimization problem is known to

change the sizes of different basins of attraction, influencing the

likely result.

**(3) - Additional Constraints**  -

Some parameterizations cover only a subset of possible inputs, rather

than the entire space.

An optimizer working in such a parameterization will still find

solutions that minimize or maximize the objective function, but they'll

be subject to the constraints of the parameterization.

By picking the right set of constraints, one can impose a variety of

constraints, ranging from simple constraints (e.g., the boundary of the

image must be black), to rich, subtle constraints.

**(4) - Implicitly Optimizing other Objects** -

 A parameterization may internally use a different kind of object than

the one it outputs and we optimize for.

 For example, while the natural input to a vision network is an RGB

image, we can parameterize that image as a rendering of a 3D object and,

 by backpropagating through the rendering process, optimize that

instead.

 Because the 3D object has more degrees of freedom than the image, we

In the rest of the article we give concrete examples where such

approaches are beneficial and lead to surprising and interesting visual

results.

---

[1](#section-aligned-interpolation)

[Aligned Feature Visualization Interpolation](#section-aligned-interpolation)

-----------------------------------------------------------------------------

 Feature visualization is most often used to visualize individual neurons,

 When we want to really understand the interaction between two neurons,

 we can go a step further and create multiple visualizations,

 Despite this, there is a small challenge: feature visualization is stochastic.

 If we make them naively, the resulting visualizations will be *unaligned*:

 visual landmarks such as eyes appear in different locations in each image.

 because they're swamped by the much larger differences in layout.

[2](#figure-aligned-interpolation-comparison)

#### Unaligned Interpolation

Visual landmarks, such as eyes, change position from one frame to the next.

pause

#### Aligned Interpolation

Frames are easier to compare because visual landmarks stay in place.

pause

 There are a number of possible approaches one could try

 For example, one could explicitly penalize differences between

adjacent frames. Our final result and our colab notebook use this

technique in combination with a shared parameterization.

[3](#figure-aligned-interpolation-examples)

We start with independently parameterized frames.

Each frame is then combined with a single, shared parameterization…

…to create a visually aligned neuron interpolation.

 By partially sharing a parameterization between frames, we encourage

 the resulting visualizations to naturally align.

 Intuitively, the shared parameterization provides a common reference

 for the displacement of visual landmarks, while the unique one gives to

 each frame its own visual appeal based on its interpolation weights.

 Let's consider randomly initializing xxx and yyy, and then optimizing them.

 Normally, the optimization problems are independent, so xxx and yyy

 are equally likely to come to unaligned solutions (where they have

different signs) as aligned ones.

 But if we add a shared parameterization, the problems become

coupled and the basin of attraction where they're aligned becomes

bigger.

![](Differentiable%20Image%20Parameterizations_files/basin-alignment.png)

 This is an initial example of how differentiable parameterizations

in general can be a useful additional tool in visualizing neural

networks.

---

[2](#section-styletransfer)

[Style Transfer with non-VGG architectures](#section-styletransfer)

-------------------------------------------------------------------

 Neural style transfer has a mystery:

 This isn't because no one is interested in doing style transfer on

other architectures, but because attempts to do it on other

architectures consistently work poorly.

 Several hypotheses have been proposed to explain why VGG works so

much better than other models.

 One suggested explanation is that VGG's large size causes it to

capture information that other models discard.

 This extra information, the hypothesis goes, isn't helpful for

classification, but it does cause the model to work better for style

transfer.

 An alternate hypothesis is that other models downsample more

aggressively than VGG, losing spatial information.

 We suspect that there may be another factor: most modern vision

 We find the same approach also improves style transfer, allowing us

to use a model that did not otherwise produce visually appealing style

transfer results:

#### Content Image

#### Style Image

#### Final image Optimization

Pixel Space

Decorrelated Space

[4](#figure-style-transfer-examples):

 Move the slider under "final image optimization" to compare

optimization in pixel space with optimization in a decorrelated space.

Both images were created with the same objective and differ only in

their parameterization.

 Let's consider this change in a bit more detail. Style transfer

involves three images: a content image, a style image, and the image we

optimize.

 All three of these feed into the CNN, and the style transfer

objective is based on the differences

in how these images activate the CNN.

 The only change we make is how we parameterize the optimized image.

Instead of parameterizing it in terms of pixels (which are highly

correlated with their neighbors), we use a scaled Fourier transform.

[5](#figure-style-transfer-diagram)

---

[3](#section-xy2rgb)

[Compositional Pattern Producing Networks](#section-xy2rgb)

-----------------------------------------------------------

 So far, we've explored image parameterizations that are relatively

close to how we normally think of images, using pixels or Fourier

components.

 CPPNs are neural networks that map (x,y)(x,y)(x,y) positions to image colors:

(x,y) →CPPN (r,g,b)(x,y) ~\xrightarrow{\tiny CPPN}~ (r,g,b)(x,y) CPPN (r,g,b)

 By applying the CPPN to a grid of positions, one can make arbitrary

resolution images.

 The parameters of the CPPN network — the weights and

biases — determine what image is produced.

 Depending on the architecture chosen for the CPPN, pixels in the

resulting image are constraint to share, up to a certain degree, the

color of their neighbors.

 Often this is done by evolution ;

 here we explore the possibility to backpropagate some objective

function, such a feature visualization objective.

 This is easily done since the CPPN network is differentiable as the

convolutional neural network and the objective function can be

propagated also through the CPPN to update its parameters accordingly.

 That is to say, CPPNs are a differentiable image

parameterization — a general tool for parameterizing images in any

neural art or visualization task.

WeightsChannelCNNCPPNimage/RGB

[6](#figure-xy2rgb-diagram):

 CPPNs are a differentiable image parameterization. We can use them

 for neural art or visualization tasks by backpropagating past the

image, through the CPPN to its parameters.

 Using CPPNs as image parameterization can add an interesting

 is an artistic medium where images are created by manipulating colorful

 light beams with prisms and mirrors. Notable examples of this technique

 are the [work of Stephen Knapp](http://www.lightpaintings.com/).

Note

 that light-painting metaphor here is rather fragile: for example light

composition is an additive process, while CPPNs can have

negative-weighted connections between layers.

[7](#figure-xy2rgb-examples):

 The visual quality of the generated images is heavily influenced by

the architecture of the chosen CPPN.

 Not only the shape of the network, i.e., the number of layers and

filters, plays a role, but also the chosen activation functions and

normalization. For example, deeper networks produce more fine grained

details compared to shallow ones.

 We encourage readers to experiment in generating different images by

changing the architecture of the CPPN. This can be easily done by

changing the code in the supplementary notebook.

 The evolution of the patterns generated by the CPPN are artistic

artifacts themselves.

 To maintain the metaphor of light-paintings, the optimization process

correspond to an iterative adjustments of the beam directions and

shapes.

 Because the iterative changes have a more global effect compared to,

for example, a pixel parameterization, at the beginning of the

optimization only major patterns are visible.

 By iteratively adjusting the weights, our imaginary beams are

positioned in such a way that fine details emerge.

![](Differentiable%20Image%20Parameterizations_files/pointer.svg)

[8](#figure-xy2rgb-training):

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 By playing with this metaphor, we can also create a new kind of

animation that morph one of the above images into a different one.

 Intuitively, we start from one of the light-paintings and we move the

beams to create a different one.

 This result is in fact achieved by interpolating the weights of the

CPPN representations of the two patterns. A number of intermediate

frames are then generated by generating an image given the interpolated

CPPN representation.

 As before, changes in the parameter have a global effect and create

visually appealing intermediate frames.

![](Differentiable%20Image%20Parameterizations_files/pointer.svg)

[9](#figure-xy2rgb-interpolation):

 Interpolating CPPN weights between two learned points.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 In this section we presented a parameterization that goes beyond a

standard image representation.

 Neural networks, a CPPN in this case, can be used to parameterize an

image that is optimized for a given objective function.

 More specifically, we combined a feature-visualization objective

function with a CPPN parameterization to create infinite-resolution

images of distinctive visual style.

---

[4](#section-rgba)

[Generation of Semi-Transparent Patterns](#section-rgba)

--------------------------------------------------------

 of images instead of a single image, and then sampling one or a few

images from that family at each optimization step.

 This is important because many of the objects we'll explore

optimizing have more degrees of freedom than the images going into the

network.

 To be concrete, let's consider the case of semi-transparent images.

These images have, in addition to the RGB channels, an alpha channel

that encodes each pixel's opacity (in the range [0,1][0,1][0,1]).

 In order to feed such images into a neural network trained on RGB

images, we need to somehow collapse the alpha channel. One way to

 where IaI\_aIa is the alpha channel of the image III.

 If we used a static background BGBGBG,

 such as black, the transparency would merely indicate pixel positions

in which that background contributes directly to the optimization

objective.

 In fact, this is equivalent to optimizing an RGB image and making it

 transparent in areas where its color matches with the background!

 Intuitively, we'd like transparent areas to correspond to something

like "the content of this area could be anything."

 Building on this intuition, we use a different random background at

every optimization step.

 We have tried both sampling from real images, and using different

types of noise.

 As long as they were sufficiently randomized, the different

distributions did not meaningfully influence the resulting optimization.

 Thus, for simplicity, we use a smooth 2D gaussian noise.

ChannelCNNrandom noise

[10](#figure-rgba-diagram):

 By default, optimizing our semi-transparent image will make the

image fully opaque, so the network can always get its optimal input.

 To avoid this, we need to change our objective with an objective

that encourages some transparency.

 We find it effective to replace the original objective with:

 with reducing the mean transparency.

 If the image becomes very transparent, it will focus on the original

 objective. If it becomes too opaque, it will temporarily stop caring

about the original objective and focus on decreasing the average

opacity.

bright

dark

checkerboard

Alpha mask

masked

opaque

[11](#figure-rgba-examples):

 It turns out that the generation of semi-transparent images is

useful in feature visualization.

 Feature visualization aims to understand what neurons in a vision

model are looking for, by creating images that maximally activate them.

 Unfortunately, there is no way for these visualizations to

distinguish which areas of an image strongly influence a neuron's

activation and those which only marginally do so.

 This issue does not occur when

optimizing for the activation of entire channels, because in that case

every pixel has multiple neurons that are close to centered on it. As a

consequence, the entire input image gets filled with copies of what

those neurons care about strongly.

 Ideally, we would like a way for our visualizations to make this

distinction in importance — one natural way to represent that a part of

the image doesn't matter is for it to be transparent.

 Thus, if we optimize an image with an alpha channel and encourage

the overall image to be transparent, parts of the image that are

unimportant according to the feature visualization objective should

become transparent.

[12](#figure-rgba-interpretability-examples)

figure 12

Introducing transparency helps separate those areas.

---

[5](#section-featureviz-3d)

[Efficient Texture Optimization through 3D Rendering](#section-featureviz-3d)

-----------------------------------------------------------------------------

 We use a 3D rendering process to turn them into 2D RGB images that

can be fed into the network, and backpropagate through the rendering

process to optimize the texture of the 3D object.

 Our technique is similar to the approach that Athalye et al.

 used for the creation of real-world adversarial examples, as we rely on

 the backpropagation of the objective function to randomly sampled views

 of the 3D model.

 We differ from existing approaches for artistic texture generation,

 as we do not modify the geometry of the object during back-propagation.

 By disentangling the generation of the texture from the position of

their vertices, we can create very detailed texture for complex objects.

Before we can describe our approach, we first need to understand how a

3D object is stored and rendered on screen. The object's geometry is

 coordinate in the texture image. The model is then rendered, i.e. drawn

 on screen, by coloring every triangle with the region of the image that

 is delimited by the (u,v)(u,v)(u,v) coordinates of its vertices.

 A simple naive way to create the 3D object texture would be to

optimize an image the normal way and then use it as a texture to paint

on the object.

 However, this approach generates a texture that does not consider

the underlying UV-mapping and, therefore, will create a variety of

visual artifacts in the rendered object.

 First, **seams** are visible on the rendered texture, because the

 optimization is not aware of the underlying UV-mapping and, therefore,

does not optimize the texture consistently along the split patches of

the texture.

 Second, the generated patterns are **randomly oriented** on

different parts of the object (see, e.g., the vertical and wiggly

patterns) because they are not consistently oriented in the underlying

UV-mapping.

#### Input Controls

 seams

 or

 wrongly oriented patterns.

Unfold Texture

#### Naïve Optimization

#### Render-based Optimization

[13](#figure-featureviz-3d-explanation):

 3D model of the famous Stanford Bunny.

 You can interact with the model by rotating and zooming. Moreover, you

can unfold the object to its two-dimensional texture representation.

This unfolding reveals the UV mapping used to store the texture in the

texture image. Note how the render-based optimized texture is divided in

 several patches that allows for a complete and undistorted coverage of

the object.

 We take a different approach.

 The following diagram presents an overview of the proposed pipeline:

[14](#figure-featureviz-3d-diagram):

 We optimize a texture by backpropagating through the rendering

process. This is possible because we know how pixels in the rendered

image correspond to pixels in the texture.

We start the process by randomly initializing the texture with a Fourier

 parameterization.

At every training iteration we sample a random camera position, which is

 oriented towards the center of the bounding box of the object, and we

render the textured object as an image.

We then backpropagate the gradient of the desired objective function,

i.e., the feature of interest in the neural network, to the rendered

image.

However, an update of the rendered image does not correspond to an

update to the texture that we aim at optimizing. Hence, we need to

further propagate the changes to the object's texture.

The propagation is easily implemented by applying a reverse UV-mapping,

as for each pixel on screen we know its coordinate in the texture.

By modifying the texture, during the following optimization iterations,

the rendered image will incorporate the changes applied in the previous

iterations.

Unfold Texture

[15](#figure-featureviz-3d-examples):

The resulting textures are consistently optimized along the cuts, hence

removing the seams and enforcing an uniform orientation for the rendered

 object.

Morever, since the function optimization is disentangled by the geometry

 of the object, the resolution of the texture can be arbitrary high.

In the next section we will se how this framework can be reused for

performing an artistic style transfer to the object's texture.

---

[6](#section-style-transfer-3d)

[Style Transfer for Textures through 3D Rendering](#section-style-transfer-3d)

------------------------------------------------------------------------------

Now that we have established a framework for efficient backpropagation

into the UV-mapped texture, we can use it to adapt existing style

transfer techniques for 3D objects.

Similarly to the 2D case, we aim at redrawing the original object's

texture with the style of a user-provided image.

The following diagram presents an overview of the approach:

[16](#figure-style-transfer-3d-diagram)

The algorithm works in similar way to the one presented in the previous

section, starting from a randomly initialized texture.

At each iteration, we sample a random view point oriented toward the

center of the bounding box of the object and we render two images of it:

#### Content Model

#### Style Image

#### Model with optimized texture

Unfold Texture

[17](#figure-style-transfer-3d-examples):

 Style Transfer onto various 3D models. Note that visual landmarks in

 the content texture, such as eyes, show up correctly in the generated

texture.

Because every view is optimized independently, the optimization is

forced to try to add all the style's elements at every iteration.

For example, if we use as style image the Van Gogh's "Starry Night"

painting, stars will be added in every single view.

We found we obtain more pleasing results, such as those presented above,

 by introducing a sort of "memory" of the style of

previous views. To this end, we maintain moving averages of

style-representing Gram matrices

over the recently sampled viewpoints. On each optimization iteration we

compute the style loss against those averaged matrices,

instead of the ones computed for that particular view.

 to the current Gram matrices.

 An alternative approach, such as the one employed by ,

 would require sampling multiple viewpoints of the scene at each step,

 increasing memory requirements. In contrast, our substitution trick can be also

 used to apply style transfer to high resolution (>10M pixels) images on a

 single consumer-grade GPU.

Take as an example the model created by imposing Van Gogh's starry night

 as style image.

The resulting texture contains the rythmic and vigorous brush strokes

that characterize Van Gogh's work.

However, despite the style image's primarily cold tones, the resulting

fur has a warm orange undertone as it is preserved from the original

texture.

Even more interesting is how the eyes of the bunny are preserved when

different styles are transfered.

For example, when the style is obtained from the Van Gogh's painting,

![](Differentiable%20Image%20Parameterizations_files/printed_bunny_extended.jpg)

[18](#figure-style-transfer-3d-picture):

 Textured models produced with the presented method can be easily used

with popular 3D modeling software or game engines. To show this, we 3D

---

[Conclusions](#conclusions)

---------------------------

 For the creative artist or researcher, there's a large space of ways

 to parameterize images for optimization.

 This opens up not only dramatically different image results, but

also animations and 3D objects!

 We think the possibilities explored in this article only scratch the

 surface.

 For example, one could explore extending the optimization of 3D

object textures to optimizing the material or reflectance — or even go

the direction of Kato et al. and optimize the mesh vertex positions.

 This article focused on *differentiable* image

parameterizations, because they are easy to optimize and cover a wide

range of possible applications.

 But it's certainly possible to optimize image parameterizations that

 aren't differentiable, or are only partly differentiable, using

reinforcement learning or evolutionary strategies .

Inceptionism: Going deeper into neural networks

A Neural Algorithm of Artistic Style

Feature Visualization

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Synthesizing robust adversarial examples

The loss surfaces of multilayer networks

Very deep convolutional networks for large-scale image recognition

Deconvolution and checkerboard artifacts

Deep Image Prior

Compositional pattern producing networks: A novel abstraction of development

Neural Network Generative Art in Javascript

Image Regression

Generating Large Images from Latent Vectors

Artificial Evolution for Computer Graphics

Neural 3D Mesh Renderer

The Stanford Bunny

Natural evolution strategies.

Evolution strategies as a scalable alternative to reinforcement learning

Tensorflow: Large-scale machine learning on heterogeneous distributed systems


Research Debt

 .mountain-illustration {

 margin-top: 20px;

 margin-bottom: 10px;

 width: 180%;

 margin-left: -40%;

 }

 @media (min-width: 768px) {

 .mountain-illustration {

 max-width: 1200px;

 margin-left: auto;

 width: 100%;

 }

 }

 dt-article > h1:first-of-type {

 margin-top: 0;

 }

Achieving a research-level understanding of most topics is like

climbing a mountain. Aspiring researchers must struggle to understand

vast bodies of work that came before them, to learn techniques, and to

gain intuition. Upon reaching the top, the new researcher begins doing

novel work, throwing new stones onto the top of the mountain and making

it a little taller for whoever comes next.

Mathematics is a striking example of this. For centuries,

countless minds have climbed the mountain range of mathematics and laid

new boulders at the top. Over time, different peaks formed, built on top

 of particularly beautiful results. Now the peaks of mathematics are so

numerous and steep that no person can climb them all. Even with a

lifetime of dedicated effort, a mathematician may only enjoy some of

their vistas.

People expect the climb to be hard. It reflects the tremendous

progress and cumulative effort that's gone into mathematics. The climb

is seen as an intellectual pilgrimage, the labor a rite of passage. But

the climb could be massively easier. It's entirely possible to build

The climb isn't progress: the climb is a mountain of debt.

The Debt

--------

Programmers talk about technical debt: there are ways to write

software that are faster in the short run but problematic in the long

run. Managers talk about institutional debt: institutions can grow

quickly at the cost of bad practices creeping in. Both are easy to

accumulate but hard to get rid of.

Research can also have debt. It comes in several forms:

* **Poor Exposition** – Often, there is no good explanation of

important ideas and one has to struggle to understand them. This problem

 is so pervasive that we take it for granted and don't appreciate how

much better things could be.

* **Undigested Ideas** – Most ideas start off rough and hard to

understand. They become radically easier as we polish them, developing

the right analogies, language, and ways of thinking.

* **Bad abstractions and notation** – Abstractions and notation

are the user interface of research, shaping how we think and

communicate. Unfortunately, we often get stuck with the first formalisms

 to develop even when they're bad. For example, an object with extra

electrons is negative, and pi is wrong.

* **Noise** – Being a researcher is like standing in the middle

of a construction site. Countless papers scream for your attention and

there's no easy way to filter or summarize them.Because

 most work is explained poorly, it takes a lot of energy to understand

each piece of work. For many papers, one wants a simple one sentence

explanation of it, but needs to fight with it to get that sentence.

Because the simplest way to get the attention of interested parties is

to get everyone's attention, we get flooded with work. Because we

The insidious thing about research debt is that it's normal.

Everyone takes it for granted, and doesn't realize that things could be

different. For example, it's normal to give very mediocre explanations

of research, and people perceive that to be the ceiling of explanation

quality. On the rare occasions that truly excellent explanations come

along, people see them as one-off miracles rather than a sign that we

could systematically be doing better.

Interpretive Labor

------------------

There's a tradeoff between the energy put into explaining an idea,

and the energy needed to understand it. On one extreme, the explainer

can painstakingly craft a beautiful explanation, leading their audience

to understanding without even realizing it could have been difficult. On

 the other extreme, the explainer can do the absolute minimum and

Many explanations are not one-to-one. People give lectures,

write books, or communicate online. In these one-to-many cases, each

member of the audience pays the cost of understanding, even though the

 are coefficients for the trade off between energy on the explanation

side and energy on the understanding side. That is a is the energy spent

 on explaining and b is the corresponding effort needed to understand.

 example, Christopher's average blog post is read by over 100,000

people; if he can save each reader just one second, he's saved humanity

30 hours.

![](Research%20Debt_files/publish-one-many-crop.jpg)

In research, we often have a group of researchers all trying to

understand each other. Just like before, the cost of explaining stays

constant as the group grows, but the cost of understanding increases

with each new member. At some size, the effort to understand everyone

else becomes too much. As a defense mechanism, people specialize,

focusing on a narrower area of interest. The maintainable size of the

field is controlled by how its members trade off the energy between

communicating and understanding.

Research debt is the accumulation of missing interpretive

labor. It's extremely natural for young ideas to go through a stage of

debt, like early prototypes in engineering. The problem is that we often

 stop at that point. Young ideas aren't ending points for us to put in a

 paper and abandon. When we let things stop there the debt piles up. It

becomes harder to understand and build on each other's work and the

field fragments.

Clear Thinking

--------------

It's worth being clear that research debt isn't just about ideas

not being explained well. It's a lack of digesting ideas – or, at least,

 a lack of the public version of ideas being digested.Often,

 some individuals have a much more developed version of an idea than is

publicly shared. There are a lot of reasons for not sharing it (in

Developing good abstractions, notations, visualizations, and so

 forth, is improving the user interfaces for ideas. This helps both with

 understanding ideas for the first time and with thinking clearly about

them. Conversely, if we can't explain an idea well, that's often a sign

that we don't understand it as well as we could.

It shouldn't be that surprising that these two largely go hand

in hand. Part of thinking is having a conversation with ourselves.

Research Distillation

---------------------

Research distillation is the opposite of research debt. It can be

incredibly satisfying, combining deep scientific understanding, empathy,

 and design to do justice to our research and lay bare beautiful

insights.

Distillation is also hard. It's tempting to think of explaining

 an idea as just putting a layer of polish on it, but good explanations

often involve transforming the idea. This kind of refinement of an idea

can take just as much effort and deep understanding as the initial

discovery.

This leaves us with no easy way out. We can't solve research

debt by having one person write a textbook: their energy is spread too

thin to polish every idea from scratch. We can't outsource distillation

to less skilled non-experts: refining and explaining ideas requires

creativity and deep understanding, just as much as novel research.

Research distillation doesn't have to be you, but it does have to be us.

Where are the Distillers?

-------------------------

Like the theoretician, the experimentalist or the research

engineer, the research distiller is an integral role for a healthy

research community. Right now, almost no one is filling it.

Why do researchers not work on distillation? One possibility is

 perverse incentives, like wanting your work to look difficult. Those

certainly exist, but we don't think they're the main factor.

 There are a lot of perverse incentives that push against explaining

things well, sharing data, and so forth. This is especially true when

the work you are doing isn't that interesting or isn't reproducible and

you want to obscure that. Or if you have a lot of competitors and don't

want them to catch up.

However, our experience is that most good

researchers don't seem that motivated by these kind of factors. Instead,

 the main issue is that it isn't worthwhile for them to divert energy

from pursuing results to distill things. Perhaps things are different in

 other fields, or I'm not cynical enough.

Lots of people want to work on research distillation.

Unfortunately, it's very difficult to do so, because we don't support

them.There is a strange kind of informal

support for people working on research distillation. Christopher has

personally benefitted a great deal from this. But it's unreliable and

not widely advertised, which makes it hard to build a career on.

An aspiring research distiller lacks many things that are easy

 to take for granted: a career path, places to learn, examples and role

models. Underlying this is a deeper issue: their work isn't seen as a

real research contribution. We need to fix this.

An Ecosystem for Distillation

-----------------------------

If you are excited to distill ideas, seek clarity, and build

beautiful explanations, we are letting you down. You have something

precious to contribute and we aren't supporting you the way we should.

* Distill Infrastructure – Tools for making beautiful interactive essays.

This is just a start: there's a lot more that needs to be done. A

complete ecosystem for this kind of work needs several other components,

 including places where one can learn these skills and reliable sources

of employment doing this kind of work. We're optimistic that will come

with time.

---

Further Reading

---------------

![](Research%20Debt_files/headstand.jpg)

* **Visual Mathematics:**

 is particularly striking, but there are several lovely examples of new

* **Explorable Explanations:**

 There's a loose community exploring how the interactive medium

enabled by computers can be used to communicate and think in ways

previously impossible. These ideas start, as many ideas in computing do,

 with work done by Douglas Engelbart and Alan Kay.

There are also explorations of how we can augment our ability

 to think in this new medium, bringing previously inaccessible ideas

* **Research Distribution:**

* **Open-Notebook Science:**

 This seems really important. Traditionally, if one doesn't turn

research into a paper, it's basically as though you didn't do it. This

creates a strong incentive for all research to be dressed up as an

important paper, increasing noise.

* **Discussion of Debt and Distillation:**

 A number of mathematicians have discussed what we call research

 I'd draw particular attention to Thurston's account of accidentally

killing a field by drowning it in research debt in section 6 of .

The Tau Manifesto

Interpretive Labor

The utopia of rules: On technology, stupidity, and the secret joys of bureaucracy

The Mythical Man-Month

Visual Complex Analysis

Visual Group Theory

A visual explanation of Jensen's inequality

Visual Differential Geometry

A Research Center for Augmenting Human Intellect

Personal Dynamic Media

Explorable explanations

Visualizing Algorithms

How to Fold a Julia Fractal

Back to the Future of Handwriting Recognition

Media for thinking the unthinkable

Learnable programming

Thought as a Technology

Toward an Exploratory Medium for Mathematics

On Proof and Progress in Mathematics

A beginner’s guide to forcing

Recoltes et Semailles

Statement on conceptual contributions in theory