Sequence Modeling with CTC

Abstract: Consider speech recognition. We have a dataset of audio clips and corresponding transcripts. Unfortunately, we don’t know how the characters in the transcript align to the audio. This makes training a speech recognizer harder than it might at first seem.

Introduction

------------

Consider speech recognition. We have a dataset of audio clips and

corresponding transcripts. Unfortunately, we don't know how the characters in

the transcript align to the audio. This makes training a speech recognizer

harder than it might at first seem.

Without this alignment, the simple approaches aren't available to us. We

could devise a rule like "one character corresponds to ten inputs". But

people's rates of speech vary, so this type of rule can always be broken.

Another alternative is to hand-align each character to its location in the

audio. From a modeling standpoint this works well — we'd know the ground truth

for each input time-step. However, for any reasonably sized dataset this is

prohibitively time consuming.

This problem doesn't just turn up in speech recognition. We see it in many

other places. Handwriting recognition from images or sequences of pen strokes

is one example. Action labelling in videos is another.

![](Sequence%20Modeling%20with%20CTC_files/handwriting_recognition.svg)

**Handwriting recognition:** The input can be

 (x,y)(x,y)(x,y) coordinates of a pen stroke or

 pixels in an image.

![](Sequence%20Modeling%20with%20CTC_files/speech_recognition.svg)

**Speech recognition:** The input can be a spectrogram or some

 other frequency based feature extractor.

Connectionist Temporal Classification (CTC) is a way to get around not

knowing the alignment between the input and the output. As we'll see, it's

especially well suited to applications like speech and handwriting

recognition.

---

To be a bit more formal, let's consider mapping input sequences

We want to find an accurate mapping from XXX's to YYY's.

There are challenges which get in the way of us

using simpler supervised learning algorithms. In particular:

* Both XXX and YYY

 can vary in length.

* The ratio of the lengths of XXX and YYY

 can vary.

* We don't have an accurate alignment (correspondence of the elements) of

 XXX and Y.Y.Y.

The CTC algorithm overcomes these challenges. For a given XXX

it gives us an output distribution over all possible YYY's. We

can use this distribution either to *infer* a likely output or to assess

the *probability* of a given output.

Not all ways of computing the loss function and performing inference are

tractable. We'll require that CTC do both of these efficiently.

**Loss Function:** For a given input, we'd like to train our

model to maximize the probability it assigns to the right answer. To do this,

we'll need to efficiently compute the conditional probability

p(Y∣X).p(Y \mid X).p(Y∣X). The function p(Y∣X)p(Y \mid X)p(Y∣X) should

also be differentiable, so we can use gradient descent.

**Inference:** Naturally, after we've trained the model, we

want to use it to infer a likely YYY given an X.X.X.

This means solving

Y∗=argmaxYp(Y∣X).

Y∗=Yargmaxp(Y∣X).

Ideally Y∗Y^\*Y∗ can be found efficiently. With CTC we'll settle

for an approximate solution that's not too expensive to find.

The Algorithm

-------------

The CTC algorithm can assign a probability for any YYY

given an X.X.X. The key to computing this probability is how CTC

thinks about alignments between inputs and outputs. We'll start by looking at

these alignments and then show how to use them to compute the loss function and

perform inference.

### Alignment

The CTC algorithm is *alignment-free* — it doesn't require an

alignment between the input and the output. However, to get the probability of

an output given an input, CTC works by summing over the probability of all

possible alignments between the two. We need to understand what these

alignments are in order to understand how the loss function is ultimately

calculated.

To motivate the specific form of the CTC alignments, first consider a naive

approach. Let's use an example. Assume the input has length six and Y=Y

=Y= [c, a, t]. One way to align XXX and YYY

is to assign an output character to each input step and collapse repeats.

![](Sequence%20Modeling%20with%20CTC_files/naive_alignment.svg)

This approach has two problems.

* Often, it doesn't make sense to force every input step to align to

 some output. In speech recognition, for example, the input can have stretches

 of silence with no corresponding output.

* We have no way to produce outputs with multiple characters in a row.

 Consider the alignment [h, h, e, l, l, l, o]. Collapsing repeats will

 produce "helo" instead of "hello".

To get around these problems, CTC introduces a new token to the set of

allowed outputs. This new token is sometimes called the *blank* token. We'll

refer to it here as ϵ.\epsilon.ϵ. The

ϵ\epsilonϵ token doesn't correspond to anything and is simply

removed from the output.

The alignments allowed by CTC are the same length as the input. We allow any

alignment which maps to YYY after merging repeats and removing

ϵ\epsilonϵ tokens:

![](Sequence%20Modeling%20with%20CTC_files/ctc_alignment_steps.svg)

If YYY has two of the same character in a row, then a valid

alignment must have an ϵ\epsilonϵ between them. With this rule

in place, we can differentiate between alignments which collapse to "hello" and

those which collapse to "helo".

Let's go back to the output [c, a, t] with an input of length six. Here are

a few more examples of valid and invalid alignments.

![](Sequence%20Modeling%20with%20CTC_files/valid_invalid_alignments.svg)

The CTC alignments have a few notable properties. First, the allowed

alignments between XXX and YYY are monotonic.

If we advance to the next input, we can keep the corresponding output the

same or advance to the next one. A second property is that the alignment of

XXX to YYY is many-to-one. One or more input

elements can align to a single output element but not vice-versa. This implies

a third property: the length of YYY cannot be greater than the

length of X.X.X.

### Loss Function

The CTC alignments give us a natural way to go from probabilities at each

time-step to the probability of an output sequence.

![](Sequence%20Modeling%20with%20CTC_files/full_collapse_from_audio.svg)

To be precise, the CTC objective

for a single (X,Y)(X, Y)(X,Y) pair is:

p(Y∣X)=p(Y \mid X) \;\; =p(Y∣X)=

∑A∈AX,Y\sum\_{A \in \mathcal{A}\_{X,Y}}A∈AX,Y∑

∏t=1Tpt(at∣X)\prod\_{t=1}^T \; p\_t(a\_t \mid X)t=1∏Tpt(at∣X)

 The CTC conditional **probability**

**marginalizes** over the set of valid alignments

 computing the **probability** for a single alignment step-by-step.

Models trained with CTC typically use a recurrent neural network (RNN) to

estimate the per time-step probabilities, pt(at∣X).p\_t(a\_t \mid X).pt(at∣X).

An RNN usually works well since it accounts for context in the input, but we're

free to use any learning algorithm which produces a distribution over output

classes given a fixed-size slice of the input.

If we aren't careful, the CTC loss can be very expensive to compute. We

could try the straightforward approach and compute the score for each alignment

summing them all up as we go. The problem is there can be a massive number of

alignments.

 For a YYY of length UUU without any repeat

 characters and an XXX of length TTT the size

 of the set is (T+UT−U).{T + U \choose T - U}.(T−UT+U). For T=100T=100T=100 and

 U=50U=50U=50 this number is almost 1040.10^{40}.1040.

For most problems this would be too slow.

Thankfully, we can compute the loss much faster with a dynamic programming

algorithm. The key insight is that if two alignments have reached the same

output at the same step, then we can merge them.

![](Sequence%20Modeling%20with%20CTC_files/all_alignments.svg)

 Summing over all alignments can be very expensive.

![](Sequence%20Modeling%20with%20CTC_files/merged_alignments.svg)

 Dynamic programming merges alignments, so it's much faster.

Since we can have an ϵ\epsilonϵ before or after any token in

YYY, it's easier to describe the algorithm

using a sequence which includes them. We'll work with the sequence

Z=[ϵ, y1, ϵ, y2, …, ϵ, yU, ϵ]

Z=[ϵ, y1, ϵ, y2, …, ϵ, yU, ϵ]

which is YYY with an ϵ\epsilonϵ at

the beginning, end, and between every character.

Let's let αlphaα be the score of the merged alignments at a

given node. More precisely, αs,tlpha\_{s, t}αs,t is the CTC score of

the subsequence Z1:sZ\_{1:s}Z1:s after ttt input steps.

As we'll see, we'll compute the final CTC score, P(Y∣X)P(Y \mid X)P(Y∣X),

from the αlphaα's at the last time-step. As long as we know

the values of αlphaα at the previous time-step, we can compute

αs,t.lpha\_{s, t}.αs,t. There are two cases.

**Case 1:**

![](Sequence%20Modeling%20with%20CTC_files/cost_no_skip.svg)

In this case, we can't jump over zs−1z\_{s-1}zs−1, the previous

token in Z.Z.Z. The first reason is that the previous token can

be an element of YYY, and we can't skip elements of

Y.Y.Y. Since every element of YYY in

ZZZ is followed by an ϵ\epsilonϵ, we can

identify this when zs=ϵ. z\_{s} = \epsilon.zs=ϵ. The second reason is

that we must have an ϵ\epsilonϵ between repeat characters in

Y.Y.Y. We can identify this when

zs=zs−2.z\_s = z\_{s-2}.zs=zs−2.

To ensure we don't skip zs−1z\_{s-1}zs−1, we can either be there

at the previous time-step or have already passed through at some earlier

time-step. As a result there are two positions we can transition from.

αs,t=lpha\_{s, t} \; =αs,t=

 The CTC probability of the two valid subsequences after

 t−1t-1t−1 input steps.

pt(zs∣X)p\_t(z\_{s} \mid X)pt(zs∣X)

 The probability of the current character at input step t.t.t.

![](Sequence%20Modeling%20with%20CTC_files/cost_regular.svg)

**Case 2:**

In the second case, we're allowed to skip the previous token in

Z.Z.Z. We have this case whenever zs−1z\_{s-1}zs−1 is

an ϵ\epsilonϵ between unique characters. As a result there are

three positions we could have come from at the previous step.

αs,t=lpha\_{s, t} \; =αs,t=

 The CTC probability of the three valid subsequences after

 t−1t-1t−1 input steps.

pt(zs∣X)p\_t(z\_{s} \mid X)pt(zs∣X)

 The probability of the current character at input step t.t.t.

Below is an example of the computation performed by the dynamic programming

algorithm. Every valid alignment has a path in this graph.

 output

Y=Y =Y= [a, b]

 input, XXX

![](Sequence%20Modeling%20with%20CTC_files/ctc_cost.svg)

 Node (s,t)(s, t)(s,t) in the diagram represents

 αs,tlpha\_{s, t}αs,t – the CTC score of

 the subsequence Z1:sZ\_{1:s}Z1:s after

 ttt input steps.

There are two valid starting nodes and two valid final nodes since the

ϵ\epsilonϵ at the beginning and end of the sequence is

optional. The complete probability is the sum of the two final nodes.

Now that we can efficiently compute the loss function, the next step is to

compute a gradient and train the model. The CTC loss function is differentiable

with respect to the per time-step output probabilities since it's just sums and

products of them. Given this, we can analytically compute the gradient of the

loss function with respect to the (unnormalized) output probabilities and from

there run backpropagation as usual.

For a training set D\mathcal{D}D, the model's parameters

are tuned to minimize the negative log-likelihood

∑(X,Y)∈D−logp(Y∣X)

\sum\_{(X, Y) \in \mathcal{D}} -\log\; p(Y \mid X)

(X,Y)∈D∑−logp(Y∣X)

instead of maximizing the likelihood directly.

### Inference

After we've trained the model, we'd like to use it to find a likely output

for a given input. More precisely, we need to solve:

Y∗=argmaxYp(Y∣X)

Y∗=Yargmaxp(Y∣X)

One heuristic is to take the most likely output at each time-step. This

gives us the alignment with the highest probability:

A∗=argmaxA∏t=1Tpt(at∣X)

A∗=Aargmaxt=1∏Tpt(at∣X)

We can then collapse repeats and remove ϵ\epsilonϵ tokens to

get Y.Y.Y.

For many applications this heuristic works well, especially when most of the

probability mass is alloted to a single alignment. However, this approach can

sometimes miss easy to find outputs with much higher probability. The problem

is, it doesn't take into account the fact that a single output can have many

alignments.

Here's an example. Assume the alignments [a, a, ϵ\epsilonϵ]

and [a, a, a] individually have lower probability than [b, b, b]. But

the sum of their probabilities is actually greater than that of [b, b, b]. The

naive heuristic will incorrectly propose Y=Y =Y= [b] as

the most likely hypothesis. It should have chosen Y=Y =Y= [a].

To fix this, the algorithm needs to account for the fact that [a, a, a] and [a,

a, ϵ\epsilonϵ] collapse to the same output.

We can use a modified beam search to solve this. Given limited

computation, the modified beam search won't necessarily find the

most likely Y.Y.Y. It does, at least, have

the nice property that we can trade-off more computation

(a larger beam-size) for an asymptotically better solution.

A regular beam search computes a new set of hypotheses at each input step.

The new set of hypotheses is generated from the previous set by extending each

hypothesis with all possible output characters and keeping only the top

candidates.

![](Sequence%20Modeling%20with%20CTC_files/beam_search.svg)

 A standard beam search algorithm with an alphabet of

 {ϵ,a,b}\{\epsilon, a, b\}{ϵ,a,b} and a beam size

 of three.

We can modify the vanilla beam search to handle multiple alignments mapping to

the same output. In this case instead of keeping a list of alignments in the

beam, we store the output prefixes after collapsing repeats and removing

ϵ\epsilonϵ characters. At each step of the search we accumulate

scores for a given prefix based on all the alignments which map to it.

 The CTC beam search algorithm with an output alphabet

 {ϵ,a,b}\{\epsilon, a, b\}{ϵ,a,b}

 and a beam size of three.

A proposed extension can map to two output prefixes if the character is a

repeat. This is shown at T=3T=3T=3 in the figure above

where 'a' is proposed as an extension to the prefix [a]. Both [a] and [a, a] are

valid outputs for this proposed extension.

When we extend [a] to produce [a,a], we only want include the part of the

previous score for alignments which end in ϵ.\epsilon.ϵ. Remember, the

ϵ\epsilonϵ is required between repeat characters. Similarly,

when we don't extend the prefix and produce [a], we should only include the part

of the previous score for alignments which don't end in ϵ.\epsilon.ϵ.

Given this, we have to keep track of two probabilities for each prefix

in the beam. The probability of all alignments which end in

ϵ\epsilonϵ and the probability of all alignments which don't

end in ϵ.\epsilon.ϵ. When we rank the hypotheses at

each step before pruning the beam, we'll use their combined scores.

The implementation of this algorithm doesn't require much code, but it is

dense and tricky to get right. Checkout this

[gist](https://gist.github.com/awni/56369a90d03953e370f3964c826ed4b0)

for an example implementation in Python.

In some problems, such as speech recognition, incorporating a language model

over the outputs significantly improves accuracy. We can include the language

model as a factor in the inference problem.

Y∗=argmaxYY^\* \enspace = \enspace {\mathop{	ext{argmax}}\limits\_{Y}}

 Y∗=Yargmax

p(Y∣X)⋅p(Y \mid X) \quad \cdotp(Y∣X)⋅

 The CTC conditional probability.

p(Y)α⋅p(Y)^lpha \quad \cdotp(Y)α⋅

 The language model probability.

L(Y)βL(Y)^etaL(Y)β

 The "word" insertion bonus.

The function L(Y)L(Y)L(Y) computes the length of

YYY in terms of the language model tokens and acts as a word

insertion bonus. With a word-based language model L(Y)L(Y)L(Y)

counts the number of words in Y.Y.Y. If we use a character-based

language model then L(Y)L(Y)L(Y) counts the number of characters

in Y.Y.Y. The language model scores are only included when a

prefix is extended by a character (or word) and not at every step of the

algorithm. This causes the search to favor shorter prefixes, as measured by

L(Y)L(Y)L(Y), since they don't include as many language model

updates. The word insertion bonus helps with this. The parameters

αlphaα and βetaβ are usually set by

cross-validation.

The language model scores and word insertion term can be included in the

beam search. Whenever we propose to extend a prefix by a character, we can

include the language model score for the new character given the prefix so

far.

Properties of CTC

-----------------

We mentioned a few important properties of CTC so far. Here we'll go

into more depth on what these properties are and what trade-offs they offer.

### Conditional Independence

One of the most commonly cited shortcomings of CTC is the conditional

independence assumption it makes.

![](Sequence%20Modeling%20with%20CTC_files/conditional_independence.svg)

 Graphical model for CTC.

The model assumes that every output is conditionally independent of

the other outputs given the input. This is a bad assumption for many

sequence to sequence problems.

Say we had an audio clip of someone saying "triple A".

 Another valid transcription could

be "AAA". If the first letter of the predicted transcription is 'A', then

the next letter should be 'A' with high probability and 'r' with low

probability. The conditional independence assumption does not allow for this.

![](Sequence%20Modeling%20with%20CTC_files/triple_a.svg)

 If we predict an 'A' as the first letter then the suffix 'AA' should get much

 more probability than 'riple A'. If we predict 't' first, the opposite

 should be true.

In fact speech recognizers using CTC don't learn a language model over the

output nearly as well as models which are conditionally dependent.

 However, a separate language model can

be included and usually gives a good boost to accuracy.

The conditional independence assumption made by CTC isn't always a bad

thing. Baking in strong beliefs over output interactions makes the model less

adaptable to new or altered domains. For example, we might want to use a speech

recognizer trained on phone conversations between friends to transcribe

customer support calls. The language in the two domains can be quite different

even if the acoustic model is similar. With a CTC acoustic model, we can easily

swap in a new language model as we change domains.

### Alignment Properties

The CTC algorithm is *alignment-free*. The objective function

marginalizes over all alignments. While CTC does make strong assumptions about

the form of alignments between XXX and YYY, the

model is agnostic as to how probability is distributed amongst them. In some

problems CTC ends up allocating most of the probability to a single alignment.

However, this isn't guaranteed.

We could force the model to choose a single

alignment by replacing the sum with a max in the objective function,

p(Y∣X)=maxA∈AX,Y∏t=1Tp(at∣X).

p(Y∣X)=A∈AX,Ymaxt=1∏Tp(at∣X).

As mentioned before, CTC only allows *monotonic* alignments. In

problems such as speech recognition this may be a valid assumption. For other

problems like machine translation where a future word in a target sentence

can align to an earlier part of the source sentence, this assumption is a

deal-breaker.

Another important property of CTC alignments is that they are

*many-to-one*. Multiple inputs can align to at most one output. In some

cases this may not be desirable. We might want to enforce a strict one-to-one

correspondence between elements of XXX and

Y.Y.Y. Alternatively, we may want to allow multiple output

elements to align to a single input element. For example, the characters

"th" might align to a single input step of audio. A character based CTC model

would not allow that.

The many-to-one property implies that the output can't have more time-steps

than the input.

 If YYY has rrr consecutive

 repeats, then the length of YYY must be less than

 the length of XXX by 2r−1.2r - 1.2r−1.

This is usually not a problem for speech and handwriting recognition since the

input is much longer than the output. However, for other problems where

YYY is often longer than XXX, CTC just won't

work.

CTC in Context

--------------

In this section we'll discuss how CTC relates to other commonly used

algorithms for sequence modeling.

### HMMs

At a first glance, a Hidden Markov Model (HMM) seems quite different from

CTC. But, the two algorithms are actually quite similar. Understanding the

relationship between them will help us understand what advantages CTC has over

HMM sequence models and give us insight into how CTC could be changed for

various use cases.

Let's use the same notation as before,

XXX is the input sequence and YYY

is the output sequence with lengths TTT and

UUU respectively. We're interested in learning

p(Y∣X).p(Y \mid X).p(Y∣X). One way to simplify the problem is to apply

Bayes' Rule:

p(Y∣X)∝p(X∣Y)p(Y).

p(Y \mid X) \; \propto \; p(X \mid Y) \; p(Y).

p(Y∣X)∝p(X∣Y)p(Y).

The p(Y)p(Y)p(Y) term can be any language model, so let's focus on

p(X∣Y).p(X \mid Y).p(X∣Y). Like before we'll let

A\mathcal{A}A be a set of allowed alignments between

XXX and Y.Y.Y. Members of

A\mathcal{A}A have length T.T.T.

Let's otherwise leave A\mathcal{A}A unspecified for now. We'll

come back to it later. We can marginalize over alignments to get

p(X∣Y)=∑A∈Ap(X,A∣Y).

p(X \mid Y)\; = \; \sum\_{A \in \mathcal{A}} \; p(X, A \mid Y).

p(X∣Y)=A∈A∑p(X,A∣Y).

To simplify notation, let's remove the conditioning on YYY, it

will be present in every p(⋅).p(\cdot).p(⋅). With two assumptions we can

write down the standard HMM.

p(X)=p(X) \quad =p(X)=

 The probability of the input

∑A∈A∏t=1T\sum\_{A \in \mathcal{A}} \; \prod\_{t=1}^T∑A∈A∏t=1T

 Marginalizes over alignments

p(xt∣at)⋅p(x\_t \mid a\_t) \quad \cdotp(xt∣at)⋅

 The emission probability

p(at∣at−1)p(a\_t \mid a\_{t-1})p(at∣at−1)

 The transition probability

The first assumption is the usual Markov property. The state

ata\_tat is conditionally independent of all historic states given

the previous state at−1.a\_{t-1}.at−1. The second is that the observation

xtx\_txt is conditionally independent of everything given the

current state at.a\_t.at.

![](Sequence%20Modeling%20with%20CTC_files/hmm.svg)

 The graphical model for an HMM.

Now we can take just a few steps to transform the HMM into CTC and see how

the two models relate. First, let's assume that the transition probabilities

p(at∣at−1)p(a\_t \mid a\_{t-1})p(at∣at−1) are uniform. This gives

p(X)∝∑A∈A∏t=1Tp(xt∣at).

p(X)∝A∈A∑t=1∏Tp(xt∣at).

There are only two differences from this equation and the CTC loss function.

The first is that we are learning a model of XXX given

YYY as opposed to YYY given X.X.X.

The second is how the set A\mathcal{A}A is produced. Let's deal

with each in turn.

To do this, we apply Bayes' rule and rewrite the model as

p(X)∝∑A∈A∏t=1Tp(at∣xt)p(xt)p(at)

p(X)∝A∈A∑t=1∏Tp(at)p(at∣xt)p(xt)

∝∑A∈A∏t=1Tp(at∣xt)p(at).

∝A∈A∑t=1∏Tp(at)p(at∣xt).

If we assume a uniform prior over the states aaa and condition on all of

XXX instead of a single element at a time, we arrive at

p(X)∝∑A∈A∏t=1Tp(at∣X).

p(X)∝A∈A∑t=1∏Tp(at∣X).

The above equation is essentially the CTC loss function, assuming the set

A\mathcal{A}A is the same. In fact, the HMM framework does not specify what

A\mathcal{A}A should consist of. This part of the model can be designed on a

per-problem basis. In many cases the model doesn't condition on YYY and the

set A\mathcal{A}A consists of all possible length TTT sequences from the

output alphabet. In this case, the HMM can be drawn as an *ergodic* state

transition diagram in which every state connects to every other state. The

figure below shows this model with the alphabet or set of unique hidden states

as {a,b,c}.\{a, b, c\}.{a,b,c}.

In our case the transitions allowed by the model are strongly related to

Y.Y.Y. We want the HMM to reflect this. One possible model could

be a simple linear state transition diagram. The figure below shows this with

the same alphabet as before and Y=Y =Y= [a, b]. Another commonly

used model is the *Bakis* or left-right HMM. In this model any

transition which proceeds from the left to the right is allowed.

![](Sequence%20Modeling%20with%20CTC_files/ergodic_hmm.svg)

**Ergodic HMM:** Any node can be either a starting or

 final state.

![](Sequence%20Modeling%20with%20CTC_files/linear_hmm.svg)

**Linear HMM:** Start on the left, end on the right.

![](Sequence%20Modeling%20with%20CTC_files/ctc_hmm.svg)

**CTC HMM:** The first two nodes are the starting

 states and the last two nodes are the final states.

In CTC we augment the alphabet with ϵ\epsilonϵ and the HMM model allows a

subset of the left-right transitions. The CTC HMM has two start

states and two accepting states.

One possible source of confusion is that the HMM model differs for any unique

Y.Y.Y. This is in fact standard in applications such as speech recognition. The

state diagram changes based on the output Y.Y.Y. However, the functions which

estimate the observation and transition probabilities are shared.

Let's discuss how CTC improves on the original HMM model. First, we can think

of the CTC state diagram as a special case HMM which works well for many

problems of interest. Incorporating the blank as a hidden state in the HMM

allows us to use the alphabet of YYY as the other hidden states. This model

also gives a set of allowed alignments which may be a good prior for some

problems.

Perhaps most importantly, CTC is discriminative. It models p(Y∣X)p(Y \mid

 X)p(Y∣X) directly, an idea that's been important in the past with other

discriminative improvements to HMMs.

Discriminative training let's us apply powerful learning algorithms like the

RNN directly towards solving the problem we care about.

### Encoder-Decoder Models

The encoder-decoder is perhaps the most commonly used framework for sequence

modeling with neural networks. These models have an encoder

and a decoder. The encoder maps the input sequence XXX into a

hidden representation. The decoder consumes the hidden representation and

produces a distribution over the outputs. We can write this as

H=encode(X)p(Y∣X)=decode(H).

egin{aligned}

H\enspace &= \enspace	extsf{encode}(X) \[.5em]

p(Y \mid X)\enspace &= \enspace 	extsf{decode}(H).

\end{aligned}

Hp(Y∣X)=encode(X)=decode(H).

The encode(⋅)	extsf{encode}(\cdot)encode(⋅) and

decode(⋅)	extsf{decode}(\cdot)decode(⋅) functions are typically RNNs. The

decoder can optionally be equipped with an attention mechanism. The hidden

state sequence HHH has the same number of time-steps as the

input, T.T.T. Sometimes the encoder subsamples the input. If the

encoder subsamples the input by a factor sss then

HHH will have T/sT/sT/s time-steps.

We can interpret CTC in the encoder-decoder framework. This is helpful to

understand the developments in encoder-decoder models that are applicable to

CTC and to develop a common language for the properties of these

models.

**Encoder:** The encoder of a CTC model can be just about any

encoder we find in commonly used encoder-decoder models. For example the

encoder could be a multi-layer bidirectional RNN or a convolutional network.

There is a constraint on the CTC encoder that doesn't apply to the others. The

input length cannot be sub-sampled so much that T/sT/sT/s

is less than the length of the output.

**Decoder:** We can view the decoder of a CTC model as a simple

linear transformation followed by a softmax normalization. This layer should

project all TTT steps of the encoder output

HHH into the dimensionality of the output alphabet.

We mentioned earlier that CTC makes a conditional independence assumption over

the characters in the output sequence. This is one of the big advantages that

other encoder-decoder models have over CTC — they can model the

dependence over the outputs. However in practice, CTC is still more commonly

used in tasks like speech recognition as we can partially overcome the

conditional independence assumption by including an external language model.

Practitioner's Guide

--------------------

So far we've mostly developed a conceptual understanding of CTC. Here we'll go

through a few implementation tips for practitioners.

**Software:** Even with a solid understanding of CTC, the

implementation is difficult. The algorithm has several edge cases and a fast

implementation should be written in a lower-level programming language.

Open-source software tools make it much easier to get started:

* Baidu Research has open-sourced

 [warp-ctc](https://github.com/baidu-research/warp-ctc). The

 package is written in C++ and CUDA. The CTC loss function runs on either

 the CPU or the GPU. Bindings are available for Torch, TensorFlow and

 [PyTorch](https://github.com/awni/warp-ctc).

* TensorFlow has built in

 [CTC loss](https://www.tensorflow.org/api_docs/python/tf/nn/ctc_loss)

 functions for the CPU.

* Nvidia also provides a GPU implementation of CTC in

 [cuDNN](https://developer.nvidia.com/cudnn) versions 7 and up.

**Numerical Stability:** Computing the CTC loss naively is

numerically unstable. One method to avoid this is to normalize the

αlphaα's at each time-step. The original publication

has more detail on this including the adjustments to the gradient.

 In practice this works well enough

for medium length sequences but can still underflow for long sequences.

A better solution is to compute the loss function in log-space with the

log-sum-exp trick.

When computing the sum of two probabilities in log space use the identity

log(ea+eb)=max{a,b}+log(1+e−∣a−b∣)

\log(e^a + e^b) = \max\{a, b\} + \log(1 + e^{-|a-b|})

log(ea+eb)=max{a,b}+log(1+e−∣a−b∣)

Most programming languages have a stable function to compute

log(1+x)\log(1 + x)log(1+x) when

xxx is close to zero.

Inference should also be done in log-space using the log-sum-exp trick.

**Beam Search:** There are a couple of good tips to know about

when implementing and using the CTC beam search.

The correctness of the beam search can be tested as follows.

1. Run the beam search algorithm on an arbitrary input.

2. Save the inferred output Y¯ar{Y}Y¯ and the corresponding

 score c¯.ar{c}.c¯.

3. Compute the actual CTC score ccc for

 Y¯.ar{Y}.Y¯.

4. Check that c¯≈car{c} pprox cc¯≈c with the former being no

 greater than the latter. As the beam size increases the inferred output

 Y¯ar{Y}Y¯ may change, but the two numbers should grow

 closer.

A common question when using a beam search decoder is the size of the beam

to use. There is a trade-off between accuracy and runtime. We can check if the

beam size is in a good range. To do this first compute the CTC score for the

inferred output ci.c\_i.ci. Then compute the CTC score for the ground

truth output cg.c\_g.cg. If the two outputs are not the same, we

should have cg<ci.c\_g \lt c\_i.cg<ci. If ci<<cgc\_i << c\_gci<<cg then

the ground truth output actually has a higher probability under the model and

the beam search failed to find it. In this case a large increase to the beam

size may be warranted.

Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition

Exploring Neural Transducers for End-to-End Speech Recognition

Large scale discriminative training of hidden Markov models for speech recognition

Connectionist Temporal Classification : Labelling Unsegmented Sequence Data with Recurrent Neural Networks

Phone recognition on the TIMIT database

Supervised Sequence Labelling with Recurrent Neural Networks

Towards End-To-End Speech Recognition with Recurrent Neural Networks

Deep Speech: Scaling up end-to-end speech recognition

A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks

LipNet: End-to-End Sentence-level Lipreading

Connectionist Temporal Modeling for Weakly Supervised Action Labeling

An application of recurrent neural networks to discriminative keyword spotting

An End-to-End Architecture for Keyword Spotting and Voice Activity Detection

Sequence Transduction with Recurrent Neural Networks

Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling

Sequence Modeling via Segmentations

Segmental Recurrent Neural Networks

Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition

Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Sequence to Sequence Learning with Neural Networks

Attention and Augmented Recurrent Neural Networks


Curve Circuits

Nick Cammarata
Gabriel Goh
Shan Carter
Chelsea Voss
Ludwig Schubert
Chris Olah

Abstract: This article is part of the Circuits thread,  an experimental format collecting invited short articles and critical  commentary delving into the inner workings of neural networks.

### Contents

* [3a Line Neuron Family](#line-families)

* [Cosmetic Neurons](#cosmetic-neurons)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

 takes the opposite approach: building up from individual neurons and

individual weights to easily verifiable explanations of tiny slices of

neural networks. But it faces a different question: can we ever hope to

understand a full neural network this way?

 We find that although curve detection involves more than 50,000

parameters, those parameters actually implement a simple algorithm that

can be read off the weights and described in just a few English

sentences. Based on this understanding, we re-implement curve detectors

 Rotational equivariance reduces the complexity by a factor of 10-20x

while scale equivariance reduces it by an additional 2-3x, for a total

reduction of ~50xA small portion involving

 the interactions of color-contrast detectors with line detectors is

further reduced by a factor of ~8x due to hue equivariance, but doesn't

significantly reduce the complexity of the overall circuit..

 We find this an exciting example of how motifs can dramatically

simplify circuits. In this circuit the main motif is equivariance, but

in others different motifs may be more important.

While the curve

circuit in InceptionV1 is quaint next to the 175b weights in GPT-3, it's

 We think the surprising simplicity of the curve circuit is a glimmer of

 hope that the Circuits approach may scale to letting us

reverse-engineer big neural networks into small verifiable explanations.

---

The Curve Detector Algorithm

----------------------------

 Decomposed feature visualization renders a grid of feature

visualization that shows the expanded weights of an upstream layer to a

downstream neuron of interest (in this case 3b:379)Specifically,

 we calculate the gradient from each layer to the 3b curve family using a

 linear approximation of InceptionV1 where ReLUs are replaced with the

identity. Then we use feature visualization with the objective of

maximizing the dot product between each image and each spatial position

of the gradient.. Intuitively, this is like decomposing the

 curve into simpler shapes, with simpler shapes the further back we go,

from early curves, to lines, and eventually Gabor filters. Each shape is

 oriented along the tangent of the curve in 3b:379.

conv2d0Gabor

 Filters line the tangent with secondary influence from the Color

 tangent is lined primarily by Complex Gabor neurons, which resemble

Gabor Filters but are invariant to the exact positions of which side is

 tangent of 3a is primarily lined by the Early Curve family as well as a

 second Line family. We'll explore this layer in depth throughout this

 use decomposed feature visualization to show how the first four layers

of InceptionV1 incrementally build towards curve detectors in 3b. To

increase legibility, we render each feature visualization with alpha

transparency and grayscale. Since curve detectors are invariant to

color, the vibrant colors feature visualization produces don't give us

 To give us a sense of the strength of each position we set the opacity

relative to its magnitude. We see the highest magnitude weights follow

the tangent of 3b:379.We can take the

highest magnitude position from each layer above and see which neuron

 doing this decomposition, it really matters which metric you choose to

measure how neurons contribute to the vector, such as average or total

weight, since there is a long tail of neurons with low magnitude that

collectively make up a large fraction of the total weight. Training

models with weight sparsity would reduce this long tail..

 decompose the highest magnitude spatial position in each layer into

neuron families to see which families contribute most to curve

 each family we can also visualize how they connect to each other,

showing us a birds-eye view of the curve detection algorithm. Again, we

see the layers progressively building more complex shapes, with shapes

that closely resemble curve detectors like early curves and well-defined

 lines being built in 3a, only one layer before curve detectors.

![](Curve%20Circuits_files/figure_003.png)We

 can visualize how neuron families connect to see a high-level view of

the curve detection algorithm of InceptionV1. In addition to giving us a

 sense of how these families are built, it lets us see how information

flows to 3b curves. For instance, we see that conv2d2 lines contribute

to 3b curves through both 3a lines and 3a early curves. While it may

seem like this throws away lots of information, in a sense this is a

natural way to understand a circuit. It's much easier to think about a

line family in conv2d2 than 33 individual line neurons that vary in

orientation.Now that we know which neuron families

 are most important for curve detection, we can invest in understanding

the weights connecting them. Luckily this is sometimes easy, since

 Many families in this diagram implement the rotational equivariance

motif, meaning that each neuron in a family is approximately a rotated

version of another neuron in that family. We can see this in the weights

 connecting 3a early curves and 3b curves.

 weights connecting 3a early curves with 3b:379 implement rotational

equivariance. It turns out there are curves of curves literally

inscribed in the weights of neural networks. We'll discuss equivariance

later in this paper, and likely again in the Circuits thread in a

dedicated paper.When neuron families implement

rotational equivariance we learn a lot looking at the strongest positive

 and negative neuron connections, because the others are just rotated

versions of them. In the weights we see a general pattern, as each layer

 builds a contrast detector by tiling a simpler contrast detector along

its tangent. When the two shapes are aligned the weight is most

positive, and most negative when they are perpendicular.

![](Curve%20Circuits_files/figure_002.png)Some

 of the strongest positive and negative weights between neuron families

in the curve circuit. We see two patterns. First, the most positive

weights tend to be where shapes are aligned, and the most negative are

where shapes are in opposing orientations. Secondly, we see the same

transitions can occur multiple times across different layers, such as

the transition from lines to curves.So far we've

looked at the neuron families that are most important to curve

detection, and also at the circuit motifs connecting them. This

high-level "circuit schematic" is useful for seeing a complex algorithm

at a glance, and it tells us the main components we'd need to build if

we wanted to reimplement the circuit from scratch.

The circuit

schematic also makes it easy to describe a few sentence English story of

 how curve detection works. Gabor filters turn into proto-lines which

build lines and early curves. Finally, lines and early curves are

composed into curves. In each case, each shape family (eg conv2d2 line)

has positive weight across the tangent of the shape family it builds

(eg. 3a early curve). Each shape family implements the rotational

equivariance motif, containing multiple rotated copies of approximately

the same neuron.

In the next few sections we'll zoom in from this

high level description to a weight-level analysis of how each of the 3a

families we've looked at so far contribute to curve detectors.To

 read more about the idea and benefits of fluidly navigating different

---

The Components of Curve Detectors

---------------------------------

### 3a Early Curve Family

Early

 curves in 3a contribute more to the construction of curve detectors

than any other neuron family. At every position curve neurons are

excited by early curves in a similar orientation and inhibited by ones

in the opposing orientation, closely following the general pattern we

saw earlier.

 every position curve neurons are excited by early curves in a similar

orientation and inhibited by ones in the opposing orientation. If you

look closely you can see the weights shift slightly over the course of

the curve to track the change in local curvature.

 are a couple other subtle things you can find if you look closely.

Firstly, you can see that the weights shift slightly over the course of

the curve, to track the change in local curvature. Secondly, curve

detectors in the same orientation are slightly inhibitory when they're

located off the curve. This makes sense, as seeing an early curve

located anywhere but the tangent indicates a different shape than a pure

 curve.The weight matrix connecting early curves

and curves shows a striking set of positive weights lining the diagonal

where the two shapes have similar orientations. This band of positive

weights is surrounded by negative weights where the early curve and

curve orientations differ. The transition is smooth — if a curve

strongly wants to see an early curve it wants to see its neighbor a bit

too.

 do we see that curve detectors which are rotated 180 degrees from each

other are inhibitory, but not ones rotated 90 degrees? Recall from our

previous article that early curve detectors respond slightly to curves

that are 180 degrees rotated from their prefered orientation, which we

called an echo. This makes sense: a curve in the opposite orientation

runs tangent to the curve we're trying to detect in the middle of the

receptive field, causing similar edge detectors to fire.

These

negative weights are the reason curve neurons in 3b have no echoes,

which we can validate by using circuit editing to remove them and

confirming that echoes appear.

 curve family of 3b uses negative weights to avoid the problem of

echoes. By removing these negative weights with circuit editing we see

echoes reappear.Using negative weights in this way

 follows a more general trend: our experience is that negative weights

are used for things that are in some way similar enough that they could

### 3a Line Families

 Since they have similar roles in the context of curve detectors we'll

discuss them together, while pointing out their subtle differences.

Like

 early curves, lines align to the tangent of curve detectors, with more

positive weights when the neurons have the same orientation. However,

this pattern is more nuanced and discontinuous with lines than early

curves. A line with a similar orientation to a curve will excite it, but

 a line that's rotated a little may inhibit it. This makes it hard to

see a general pattern by looking directly at the weight matrix.

Instead,

 we can study which line orientations excite curve detectors using

synthetic stimuli. We can take a similar approach to decomposed feature

visualization to see which lines different spatial positions respond to.

 This shows us that each 3b curve neuron is excited by edges along its

tangent line with a tolerance of between about 10° to 45°.

 tolerance isn't always symmetric, which we can see in 3b:406 below. On

its left side it is most excited by lines about 10° upwards. If the line

 is oriented above 10° it is still excited, but if it is less than 10°

it switches to being inhibited.

 view tells us how each curve detector responds to different

orientations of lines in general. We can connect this back to individual

 line neurons by studying which orientations those line neurons respond

#### Lines

There

 are 11 simple line neurons in 3a that mostly fire to one orientation,

although some activate more weakly to an "echo" 90 degrees away.

Five

 neurons in 3a respond to lines that are perpendicular to the

orientation where they are longest. These neurons mostly detect fur, but

 they also contribute to 3b curves.

Finally,

 there are five line neurons with curiously sharp transitions. These

lines want an orientation facing a particular direction, and tolerate

error in that direction, but definitely don't want it going the other

way, even if slightly. In curve detection, this is useful for handling

imperfections, like bumps.

 find cliff-like line neurons an interesting example of non-linear

behavior. We usually think of neurons as measuring distance from some

ideal. For instance, we may expect car neurons to prefer Or

 we could imagine a more sophisticated metric, such as style. Since

ImageNet contains classes for sports cars and non-sports cars, there may

 be neurons that measure the "sportiness" of different cars. We think

that studying neurons that correspond to cars is an interesting research

 direction, since it's easy to accumulate datasets of cars with labelled

 properties such as year, country of origin, and price.

cars of a certain size, and have weaker activations to cars that are

bigger or smaller than its ideal. But this line family provides a

counter-example, accepting error on one side while not firing at all on

the other.

#### Lines in conv2d2

The

 different types of line neurons we looked at above each have different

behaviors, which is part of why the weight matrix between 3a lines and

3b curves is indecipherable. However, if we go back one more layer and

look at how conv2d2 lines connect to 3b curves, we see structure.

 think this points to an interesting property of both curves and lines

in InceptionV1. The line family in conv2d2 are roughly "pure" line

detectors, detecting lines in an elegant pattern. The next layer (3a)

builds lines too, but they're more applied and nuanced, with

seemingly-awkward behavior like cliff-lines where the network finds them

 useful. Similarly, the 3b curve detectors are surprisingly elegant

detectors of curves behaviorally, and mostly follow clean patterns in

their construction. In contrast, the curves in the next layer (4a) are

 and seemingly specialized for detecting real-world objects like the

tops of cups and pans. Perhaps this points to a yet unnamed motif of

pure shape detectors directly followed by applied ones.

### Cosmetic Neurons

In

 Curve Detectors we saw how curve neurons seem to be robust to several

cosmetic properties, with similar behavior across textures like metal

and wood in a variety of lighting. How do they do it?

![](Curve%20Circuits_files/datasetExamples.png)We

 believe this reflects a more widespread phenomenon in early vision. As

progressively sophisticated shapes are built in each layer, new shapes

incorporate cosmetic neuron families like colors and texture. For

instance, 3b curve neurons are built primarily from the line and early

curve neuron families, but they also incorporate a family of 65 texture

neurons. This means they both inherit the cosmetic robustness of the

line and early curve neurons, as well as strengthen it by including more

 textures.

While we won't do a detailed weight-level analysis of

how cosmetic robustness propagates through the shapes of early vision in

 this article, which is a broader topic than curve detection, we think

this is an exciting direction for future research.

---

An Artificial Artificial Neural Network

---------------------------------------

How

 do we know this story about the mechanics of curve detectors is true?

One way is to use it to reimplement curve detectors from scratch. We

manually set the weights of a blank neural network to implement the

 This was initially a few hour process for one person (Chris Olah), and

they did not look at the original neural network's weights when

constructing it, which would go against the spirit of the exercise.

Later, before publishing they tweaked the weights and in particular

added negative weights to remove echoes in the activations.

To

compare our artificial curve detectors against InceptionV1's naturally

 available to us. We'll choose three: feature visualization, dataset

examples, and synthetic stimuli. From there we'll run two additional

comparisons by leveraging circuits and model editing.

First we'll

look at feature visualization and responses to synthetic curve stimuli

together. We see the feature visualizations indeed render curves, except

 they are grayscale since we never include cosmetic features such as

color-contrast detectors in our artificial curves. We see their response

 to curve stimuli approximates the natural curve detectors across a

range of radii and all orientations. One difference is our artificial

 can compare the curve detectors in a neural network we hand-crafted

with the curve detectors in InceptionV1 by measuring how they activate

to synthetic curve stimuli. We see that across a range of radii and

orientations, our artificial curve neurons approximate the natural ones.We

 can also get a qualitative sense for the differences by looking at a

saliency map of dataset examples that cause artificial curve detectors

to fire stronglySince our artificial curve

 detectors don't respond to color, there are likely many false-negatives

 where natural curve detectors correctly catch that our artificial ones

don't. Our goal in building artificial curve detectors isn't perfection,

 but to show that we can use the circuit patterns described in this

article to roughly approximate the curve detection algorithm in

InceptionV1..

 we can compare the weights for the circuits connecting neuron families

in the two models, alongside feature visualizations for each of those

families. We see they follow approximately the same circuit structure.

 comparison of the most important family in each layer across our

artificial artificial neural network with InceptionV1 trained on

ImageNet.We can also zoom into specific weight

matrices that we've already studied in this article. We see the raw

weights between early curves and curves as well as lines to curves look

approximately like the ones in InceptionV1, but cleaner since we set

them programmatically.

 weights of the artificial artificial neural network closely resemble

the equivariant weight in the naturally trained network, except their

pattern is cleaner because the weights are programmetically generated.

equivariance.Finally, a preliminary experiment

suggests that adding artificial curve detectors helps recover some of

the loss in classification accuracy across the dataset of removing them

from the model entirelySpecifically, when

we remove InceptionV1's 3b curve detectors entirely, the model's top-5

accuracy across 2000 validation set images drops from 88.6% to 86.3% (46

 fewer correct classifications). When we replace 3b curve detectors with

 our artificial curve detectors more than half of this drop is

recovered, with an accuracy of 87.6% (reducing the gap from 46 incorrect

 classifications to 20). We suspect the remaining gap is due to three

factors. First, our artificial neurons are grayscale, since we are not

implementing color contrast neurons or texture neurons. Secondly, the

receptive fields of artificial neurons may not be exactly centered the

same as natural curve detectors. Thirdly, we may not have scaled the

activations of curve detectors optimally.

Additionally, there are

 two caveats worth mentioning about our experimental setup. First, our

ImageNet evaluation likely doesn't mimic the exact conditions the model

was trained under (eg. data preprocessing), since the original model was

 trained at Google using a precursor Tensorflow. Secondly, the reason we

 ran it on less than the full validation set was operational, not a

result of cherry-picking. We initially ran it on a small set in a

prototype experiment to validate our hypothesis. We planned to run it on

 the full set before our publication date, but due OpenAI infrastructure

 changes our setup broke and we were unable to reimplement it in time.

For this reason we emphasize that the experiment is preliminary,

although we suspect it's likely to work on the full validation set as

well..

Overall, we believe these five experiments

show our artificial curve detectors are roughly analogous to the

naturally trained ones. Since they are nearly a direct translation from

the neuron families and circuit motifs we've described in this article

into Python code for setting weights, we think this is strong evidence

these patterns accurately reflect the underlying circuits that construct

 curve detectors.

---

Downstream

----------

While

this article focused mostly on how curve detection works upstream of the

 curve detector, it's also worth briefly considering how curves are used

 downstream in the model. It's easiest to see their mark on the next

layer, 4a, where they're used to construct more sophisticated shapes.

Many

 of these shapes look for individual curves at different spatial

positions, such as circles and evolutes. These shapes often reappear

across different branches of 4a, such as the 3x3 and 5x5 branch.

![](Curve%20Circuits_files/figure.png)Layer

 4a also constructs a series of curve detectors, mostly in the 5x5

branch that specializes in 3d geometry. However, we believe they should

be thought of less as pure abstract shapes and more as corresponding to

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.


Curve Detectors

Nick Cammarata
Gabriel Goh
Shan Carter
Ludwig Schubert
Michael Petrov
Chris Olah

### Contents

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

 vision model we've explored in detail contains neurons which detect

curves. Curve detectors in vision models have been hinted at in the

 curve in our earlier overview of early vision, but wanted to examine

them in more depth. This article is the first part of a three article

deep dive into curve detectors: their behavior, how they're built from

earlier neurons, and their prevalence across models.

We're doing

this because we believe that the interpretability community disagrees on

 several crucial questions. In particular, are neural network

representations composed of meaningful features — that is, features

tracking articulable properties of images? On the one hand, there are a

number of papers reporting on seemingly meaningful features, such as eye

 detectors, head detectors, car detectors, and so forth .

 At the same time, there's a significant amount of skepticism, only

partially reflected in the literature. One concern is that features

which seem superficially to be meaningful may in fact not be what they

 rather than the kind of meaningful features described earlier. Finally,

 even if some meaningful features exist, it's possible they don't play

an especially important role in the network.

 Some reconcile these results by concluding that if one observes, for

example, what appears to be a dog head detector, it is actually a

detector for special textures correlated with dog heads.

This

disagreement really matters. If every neuron was meaningful, and their

connections formed meaningful circuits, we believe it would open a path

to completely reverse engineering and interpreting neural networks. Of

course, we know not every neuron is meaningful, As

 discussed in Zoom In, the main issue we see is what we call

polysemantic neurons which respond to multiple different features,

seemingly as a way to compress many features into a smaller number of

neurons. We're hopeful this can be worked around. but we

think it's close enough for this path to be tractable. However, our

position is definitely not the consensus view. Moreover, it seems too

good to be true, and rings of the similar failed promises in other

fieldsFor example, genetics seems to have

been optimistic in the past that genes had individual functions and that

 the human genome project would allow us to "mine miracles," a position

We

 believe that curve detectors are a good vehicle for making progress on

this disagreement. Curve detectors seem like a modest step from

edge-detecting Gabor filters, which the community widely agrees often

form in the first convolutional layer. Furthermore, artificial curves

are simple to generate, opening up lots of possibilities for rigorous

investigation. And the fact that they're only a couple convolutional

layers deep means we can follow every string of neurons back to the

input. At the same time, the underlying algorithm the model has

implemented for curve detection is quite sophisticated. If this paper

persuades skeptics that at least curve detectors exist, that seems like a

 substantial step forward. Similarly, if it surfaces a more precise

point of disagreement, that would also advance the dialogue.

[A Simplified Story of Curve Neurons

-----------------------------------](#a-simplified-story-of-curve-neurons)Before

 running detailed experiments, let's look at a high level and slightly

simplified story of how the curve 10 neurons in 3b work.

 curve detector implements a variant of the same algorithm: it responds

to a wide variety of curves, preferring curves of a particular

orientation and gradually firing less as the orientation changes. Curve

neurons are invariant to cosmetic properties such as brightness,

texture, and color.

 are normalized by the neuron's maximum activation. We'll examine

 across ImageNet, and usually activating weakly when they do fire. When

they activate strongly, it's in response to curves with similar

orientation and curvature to their feature visualization.

---

It's

 worth stepping back and reflecting on how surprising the existence of

seemingly meaningful features like curve detectors is. There's no

explicit incentive for the network to form meaningful neurons. It's not

like we optimized these neurons to be curve detectors! Rather,

InceptionV1 is trained to classify images into categories many levels of

 abstraction removed from curves and somehow curve detectors fell out of

 gradient descent.

Moreover, detecting curves across a wide

variety of natural images is a difficult and arguably unsolved problem

in classical computer visionThis is our

sense from trying to implement programmatic curve detection to compare

them to curve neurons. We found that practitioners generally had to

choose between several algorithms, each with significant trade-offs such

 as robustness to different kinds of visual "noise" (for instance,

texture), even in images much less complex than the natural images in

 in general, is a very challenging one and, except for toy examples,

there are no good solutions." Additionally, many classical curve

detection algorithms are too slow to run in real-time, or require often

intractable amounts of memory.. InceptionV1 seems to learn a

 flexible and general solution to this problem, implemented using five

convolutional layers. We'll see in the next article that the algorithm

used is straightforward and understandable, and we've since

reimplemented it by hand.

What exactly are we claiming when we say

 these neurons detect curves? We think part of the reason there is

sometimes disagreement about whether neurons detect particular stimuli

is that there are a variety of claims one may be making. It's pretty

easy to show that, empirically, when a curve detector fires strongly the

 stimulus is reliably a curve. But there are several other claims which

might be more contentious:

* **Causality**

Curve detectors genuinely detect a curve feature, rather than another

stimulus correlated with curves. We believe our feature visualization

and visualizing attribution experiments establish a causal link, since

"running it in reverse" produces a curve.

* **Generality:**

 Curve detectors respond to a wide variety of curve stimuli. They

tolerate a wide range of radii and are largely invariant to cosmetic

attributes like color, brightness, and texture. We believe that our

experiments explicitly testing these invariances with synthetic stimuli

are the most compelling evidence of this.

* **Purity:**

 Curve detectors are not polysemantic and they have no meaningful

secondary function. Images that cause curve detectors to activate

weakly, such as edges or angles, are a natural extension of the

algorithm that InceptionV1 uses to implement curve detection. We believe

 our experiments classifying dataset examples at different activation

magnitudes and visualizing their attributions show that any secondary

function would need to be rare. In the next article, exploring the

mechanics of the algorithm implementing curve detectors, we'll provide

further evidence for this claim.

* **Family:** Curve neurons collectively span all orientations of curves.

[Feature Visualization

 uses optimization to find the input to a neural network that maximizes a

 given objective. The objective we often use is to make the neuron fire

as strongly as possible, but we'll use other objectives throughout in

this article. One reason feature visualization is powerful is that it

tells us about causality. Since the input starts with random noise and

optimizes pixels rather than a generative prior, we can be confident

that any property in the resulting image contributed to the objective.

 feature visualizations is a bit of a skill, and these images might feel

 disorienting if you haven't spent much time with them before. The most

important thing to take away is the curve shape. You may also notice

that there are bright, opposite hue colors on each side of the curve:

this reflects a preference to see a change in color at the boundary of

the curve. Finally, if you look carefully, you will notice small lines

perpendicular to the boundary of the curve. We call this weak preference

 for small perpendicular lines "combing" and will discuss it later.

Every

 time we use feature visualization to make curve neurons fire as

strongly as possible we get images of curves, even when we explicitly

Feature

 visualization finds images that maximally cause a neuron to fire, but

are these superstimuli representative of the neuron's behavior? When we

see a feature visualization, we often imagine that the neuron fires

strongly for stimuli qualitatively similar to it, and gradually becomes

weaker as the stimuli exhibit those visual features less. But one could

imagine cases where the neuron's behavior is completely different in the

 non-extreme activations, or cases where it does fire weakly for messy

versions of the extreme stimulus, but also has a secondary class of

stimulus to which it responds weakly.

If we want to understand how

 a neuron behaves in practice, there's no substitute to simply looking

at how it actually responds to images from the dataset.

[Dataset Analysis

 because some experiments required non-trivial manual labor. However,

the core ideas in this section will apply to all curve detectors in 3b.

 fire? When it fires, how often does it fire strongly? And when it

doesn't fire, is it often strongly inhibited, or just on the verge of

firing? We can answer these questions by visualizing the distribution of

 activations across the dataset.

When studying ReLU networks, we

find it helpful to look at the distribution of pre-activation values.

Since ReLU just truncates the left hand side, it's easy to reason about

the post-activation values, but it also shows us how close the neuron

 has a pre-activation mean of about -200. It fires in just 11% of cases

across the dataset, since negative values will be set to zero by the

ReLU activation function.

 observation that neural network activations generally follow an

exponential distribution was first made to us by Brice Ménard, who

observed it to be the case over all but the first layer of several

networks. This is mildly surprising both because of how perfectly they

seem to follow an exponential distribution, and also because one often

 looking at pre-ReLU values for 3b:379 activations, we see that both

positive and negative values follow an exponential distribution. Since

all negative values will be lifted to zero by the ReLU, 3b:379

activations are sparse, with only 11% of stimuli across the dataset

causing activations.To understand different parts

of this distribution qualitatively we can render a quilt of images by

 to activate different amounts. The quilt shows a pattern. Images that

cause the strongest activations have curves that are similar to the

neuron's feature visualization. Images that cause weakly positive

activations are imperfect curves, either too flat, off-orientation, or

with some other defect. Images that cause pre-ReLU activations near zero

 tend to be straight lines or images with no arcs, although some images

are of curves about 45 degrees off orientation. Finally, the images that

 cause the strongest negative activations have curves with an

orientation more than 45 degrees away from the neuron's ideal curve.

 of images reveal patterns across a wide range of activations, but they

can be misleading. Since a neuron's activation to a receptive-field

sized crop of an image is just a single number, we can't be sure which

part of the image caused it. As a result, we could be fooled by spurious

[Visualizing Attribution

 These methods attempt to describe which pixels or earlier neurons are

responsible for causing neurons to fire. In the general case of complex

non-linear functions, there is a lot of disagreement over which

attribution methods are principled and whether they are reliable .

 But in the linear-case, attribution is generally agreed upon, with most

Since

 a neuron's pre-activation function and bias value is a linear function

of neurons in the layer before it, we can use this generally agreed upon

 attribution method. In particular, curve detectors in 3b's

pre-activation value is a linear function of 3a. The attribution tensor

describing how all neurons in the previous layer influenced a given

neuron is the activations pointwise multiplied by the weights.

We

normally use feature visualization to create a superstimulus that

activates a single neuron, but we can also use it to activate linear

combinations of neurons. By applying feature visualization to the

attribution tensor, we are creating the stimulus that maximally

 to fire. Additionally, we will use the absolute value of the

attribution tensor, which shows us features that caused the neuron to

fire and also features that inhibited it. This can be useful for seeing

curve-related visual properties that influenced our cure neuron, even if

 that influence was to make it fire less.

 is the activations of the previous hidden layer. In practice, we find

it helpful to parameterize these attribution visualizations to be

grayscale and transparent, making the visualization easier to read for

non-experts . Example code can be found in the notebook.

 the above experiment visualizes every neuron in 3a, attribution is a

powerful and flexible tool that could be used to apply to studies of

circuits in a variety of ways. For instance, we could visualize how an

 before 3b, visualizing the image's activation vector and attribution

vector to curve neurons at each family along the way. Each activation

vector would show what a family saw in the image, and each attribution

In

 the next section we'll look at a less sophisticated technique for

extracting information from dataset images: blindfolding yourself from

seeing neuron activations and classifying images by hand.

[Human Comparison

----------------](#human-comparison)Nick

 Cammarata, an author of this paper, manually labelled over 800 images

into four groups: curve, imperfect curve, unrelated image, or opposing

 While labeling, Nick could only see the image's pixels, not additional

information such as the neuron's activations or attribution

visualizations. He used the following rubric in labeling:

* **Curves**:

 The Image has a curve with a similar orientation to the neuron's

feature visualization. The curve goes across most of the width of the

image.

* **Imperfect Curve**: The image has a

curve that is similar to the neuron's feature visualization, but has at

least one significant defect. Perhaps it is too flat, has an angle

interrupting the arc, or the orientation is slightly off.

* **Unrelated**: The image doesn't have a curve.

 during labeling Nick felt it was often difficult to place samples into

groups, as many images seemed to fall within the boundaries of the

rubric. We were surprised when we saw that activations clearly separate

into different levels of activation..

 probability of each group by 3b:379 activation across our hand-labelled

 dataset of around 850 images. We see that the different human labels

separate into different ranges of activations.Still,

 there are many images that cause the neuron to activate but aren't

classified as curves or imperfect curves. When we visualize attribution

 examples that activate 3b:379 but were labelled "unrelated" by humans

often contain subtle curves that are revealed by visualizing the image's

 that occurs when looking at one kind of stimulus for a long time. As a

result, he found it hard to tell whether subtle curves were simply

perceptual illusions. By visualizing attribution, we can reveal the

curves that the neuron sees in the image, showing us curves that our

**How important are different points on the activation spectrum?**

 seems to be highly selective for curve stimuli when it fires strongly,

this is only a tiny fraction of cases where it fires. Most of the time,

it doesn't fire at all, and when it does it's usually very weakly.

To

 see this, we can look at the probability density over activation

magnitudes from all ImageNet examples, split into the same

per-activation-magnitude (x-axis) ratio of classes as our hand labelled

dataset.

 our hand-labelled dataset uniformly samples from activations, images of

 curves are rare within the dataset and 3b:379 activations follow an

exponential distribution. In this plot we show 3b:379 activations split

into the conditional probabilities of different groups at a given

activation level within our hand-labelled dataset.From

 this perspective, we can't even see the cases where our neuron fires

strongly! Probability density exponentially decays as we move right, and

 so these activations are rare. To some extent, this is what we should

expect if these neurons really detect curves, since clear-cut curves

rarely occur in images.

 only weakly fire or didn't fire, this graph seems to show that the

majority of stimuli classified as curves also fall in these cases, as a

result of neurons firing strongly being many orders of magnitude rarer.

This seems to be at least partly due to labeling error and the rarity of

 curves (see discussion later). But it makes things a bit hard to reason

 about. This is why we haven't provided a precision-recall curve: recall

 would be dominated by the cases where the neuron didn't fire strongly

and be dominated by potential labeling error as a result.

It's not

 clear that probability density is really the right way to think about

the behavior of a neuron. The vast majority of cases are cases where the

 neuron didn't fire: are those actually important to think about? And if

 a neuron frequently barely fires, how important is that for

understanding the role of a neuron in the network?

 This measure can be thought of as giving an approximation at how much

that activation value influences the output of the neuron, and by

extension network behavior. There's still reason to think that high

activation cases may be disproportionately important beyond this (for

example, in max pooling only the highest value matters), but

contribution to expected value seems like a reasonable estimate.If

 one wanted to push further on exploring the importance of different

parts of the activation spectrum, they might take some notion of

attribution (methods for estimating the influence of one neuron on later

 neurons in a particular case) and estimate the contribution to the

expected value of the attribution to the logit. A simple version of this

 to the expected value of different activations, which shows how much

each activation value influences the output of a neuron. Since curves

are rare within the dataset, weak neuron activations contribute most to

 was really a curve detector in a meaningful sense. Even if it's highly

selective when it fires strongly, how can that be what matters when it

isn't even visible on a probability density plot? Contribution to

expected value shows us that even by a conservative measure, curves and

imperfect curves form 55%. This seems consistent with the hypothesis

that it really is a curve detector, and the other stimuli causing it to

fire are labeling errors or cases where noisy images cause the neuron to

 misfire.

 activations seem to correspond roughly to a human labelled judgement of

 whether images contain curves. Additionally, visualizing the

attribution vector of these images tells us that the reason these images

 fire is because of the curves in the images, and we're not being fooled

 by spurious correlations. But these experiments are not enough to

defend the claim that curve neurons detect images of curves. Since

images of curves appear infrequently in the dataset, using it to

systematically study curve images is difficult. Our next few experiments

 will focus on this directly, studying how curve neurons respond to the

space of reasonable curve images.

[Joint Tuning Curves

-------------------](#joint-tuning-curves)Our

 first two experiments suggest that each curve detector responds to

curves at a different orientation. Our next experiment will help verify

that they really do detect rotated versions of the same feature, and

characterize how sensitive each unit is to changes in orientation.

We do this by creating a **joint tuning curve**In

 neuroscience, tuning curves — charts of neural response to a continuous

 stimulus parameter — came to prominence in the early days of vision

research. Observation of receptive fields and orientation-specific

responses in neurons gave rise to some of the earliest theories about

how low-level visual features might combine to create higher-level

representations. Since then they have been a mainstay technique in the

 collect dataset examples that maximally activate neuron. We rotate them

 by increments of 1 degree from 0 to 360 degrees and record activations.The

 activations are shifted so that the points where each neuron responds

are aligned. The curves are then averaged to create a typical response

 tuning curves are useful for measuring neuron activations across

perturbations in natural images, we're limited by the kinds of

perturbations we can do on these images. In our next experiment we'll

get access to a larger range of perturbations by rendering artificial

stimuli from scratch.

[Synthetic Curves

----------------](#synthetic-curves)While

 the dataset gives us almost every imaginable curve, they don't come

labelled with data such as orientation or radius, making it hard to

answer questions that require systematically measuring responses to

visual properties. How sensitive are curve detectors to curvature? What

orientations do they respond to? Does it matter what colors are

involved? One way to get more insight into these questions is to draw

our own curves. Using synthetic stimuli like this is a common method in

visual neuroscience, and we've found it to also be very helpful in the

study of artificial neural networks. The experiments in this section are

 specifically inspired by similar experiments probing for curve

detecting biological neurons .

Since

 dataset examples suggest curve detectors are most sensitive to

orientation and curvature, we'll use them as parameters in our curve

renderer. We can use this to measure how changes in each property causes

 to fire. We find it helpful to present this as a heatmap, in order to

get a higher resolution perspective on what causes the neuron to fire.

 find that simple drawings can be extraordinarily exciting. The curve

images that cause the strongest excitations — up to 24 standard

deviations above the average dataset activation! — have similar

orientation and curvature to the neuron's feature visualization.

The

 triangular geometry shows that curve detectors respond to a wider range

 of orientations in curves with higher curvature. This is because curves

 with more curvature contain more orientations. Consider that a line

contains no curve orientations and a circle contains every curve

orientation. Since the synthetic images closer to the top are closer to

lines, their activations are more narrow.

The wisps show that tiny

 changes in orientation or curvature can cause dramatic changes in

activations, which indicate that curve detectors are fragile and

non-robust. Sadly, this is a more general problem across neuron

Varying

 curves along just two variables reveals barely-perceptible

perturbations that sway activations several standard deviations. This

suggests that the higher dimensional pixel-space contains more

pernicious exploits. We're excited about the research direction of

carefully studying neuron-specific adversarial attacks, particularly in

early vision. One benefit of studying early vision families is that it's

 tractible to follow the whole circuit back to the input, and this could

 be made simpler by extracting the important parts of a circuit and

studying it in isolation. Perhaps this simplified environment could give

 us clues into how to make neurons more robust or even protect whole

models against adversarial attacks.

In addition to testing

orientation and curvature, we can also test other variants like whether

the curve shapes are filled, or if they have color. Dataset analyses

hints that curve detectors are invariant to cosmetic properties like

lighting, and color, and we can confirm this with synthetic stimuli.

----------------](#synthetic-angles)Both

 our synthetic curve experiments and dataset analysis show that although

 curves are sensitive to orientation, they have a wide tolerance for the

 radius of curves. At the extreme, curve neurons partially respond to

edges in a narrow band of orientations, which can be seen as a curve

with infinite radius. This may cause us to think curve neurons actually

respond to lots of shapes with the right orientation, rather than curves

 specifically. While we cannot systematically render all possible

shapes, we think angles are a good test case for studying this

hypothesis.

In the following experiment we vary synthetic angles

similarly to our synthetic curves, with radius on the y axis and

orientation across the x axis.

 activations form two distinct lines, with the strongest activations

where they touch. Each line is where one of the two lines in the angle

aligns with the tangent of the curve. The two lines touch where the

angle most similar to a curve with an orientation that matches the

neuron's feature visualization. The weaker activations on the right side

 of the activations have the same cause, but with the inhibitory half of

 the angle stimulus facing outwards instead of inwards.

![](Curve%20Detectors_files/figure-f4e54f9da80ab5da3ac54a096ca11be4.svg)The

 first stimuli we looked at were synthetic curves and the second stimuli

 was synthetic angles. In the next examples we show a series of stimuli

that transition from angles to curves. Each column's strongest

activation is stronger than the column before it since rounder stimuli

are closer to curves, causing curve neurons to fire more strongly.

Additionally, as each stimulus becomes rounder, their "triangles of

activation" become increasingly filled as the two lines from the

original angle stimuli transition into a smooth arc.

 transition from angles on the left to curves on the right, making the

stimuli rounder at each step. Each step we see the maximum activation

for each neuron increase, and the activation "triangle" fill in as the

 interface is useful for seeing how different curve neurons respond to

changes in multiple stimuli properties, but it's bulky. In the next

section we'll be exploring curve families across different layers, and

it will be helpful to have a more compact way to view activations of a

curve neuron family. For this, we'll introduce a *radial tuning curve*.

[Radial Tuning Curve

---------------------------------](#the-curve-families-of-inceptionv1)So

 far we've been looking at a set of curve neurons in 3b. But InceptionV1

 actually contains curve neurons in four contiguous layers, with 3b

being the third of these layers.

 which we sometimes shorten to "2", is the third convolutional layer in

InceptionV1. It contains two types of curve detectors: concentric curves

 and combed edges.

Concentric curves are small curve detectors

that have a preference for multiple curves at the same orientation with

increasing radii. We believe this feature has a role in the development

of curve detectors in 3a and 3b that are tolerant of a wide range of

radii.

![](Curve%20Detectors_files/imgsappnick-personal1XjsJAYh6F.png)Combed

 edges detect several lines protruding perpendicularly from a larger

line. These protruding lines also detect curves, making them a type of

curve detector. These neurons are used to construct later curve

detectors and play a part in the [combing effect](#combing-effect).

Looking

 at conv2d2 activations we see that curves respond to one contiguous

range like the ones in 3b, but also weakly activate to a range on the

opposite side, 180 degrees away. We call this secondary range **echoes**.

 3a non-concentric curve detectors have formed. In many ways they

resemble the curve detectors in 3b, and in the next article we'll see

how they're used to build 3b curves. One difference is that the 3a

curves have echoes.

You

 may notice that there are two large angular gaps at the top of the

radial tuning curve for 3b, and smaller ones at the bottom. Why is that?

 4a the network constructs many complex shapes such as spirals and

boundary detectors, and it is also the first layer to construct 3d

geometry. It has several curve detectors, but we believe they are better

 thought of as corresponding to specific worldly objects rather than

 appears to be a upwards facing curve detector with confusing secondary

behavior at two other angles. But dataset examples reveal its secret:

it's detecting the tops of cups and pans viewed from an angle. In this

sense, it is better viewed as a tilted 3d circle detector.

 is a good example of how neural network interpretability can be

subjective. We usually think of abstract concepts like curves and

worldly objects like coffee cups as belonging as different kinds of

things — and for most of the network they are separate. But there's a

transition period where we have to make a judgement call, and 4a is that

 transition.

---------------------------](#repurposing-curve-detectors)We

 started studying curve neurons to better understand neural networks,

not because we were intrinsically interested in curves. But during our

investigation we became aware that curve detection is important for

fields like aerial imaging, self-driving cars, and medical research, and

 there's a breadth of literature from classical computer vision on curve

 detection in each domain. We've prototyped a technique that leverages

the curve neuron family to do a couple different curve related computer

vision tasks.

One task is *curve extraction*

 , the task of highlighting the pixels of the image that are part of

curves. Visualizing attribution to curve neurons, as we've been doing in

 this article, can be seen as a form of curve extraction. Here we

compare it to the commonly used Canny edge detection algorithm on an

x-ray of blood vessels known as an angiogram, taken from  , Figure 2.1.

 attribution visualization clearly separates and illuminates the lines

and curves, and displays less visual artifacts. However, it displays a

strong [combing effect](#combing-effect) — unwanted

perpendicular lines emanating from the edge being traced. We're unsure

how harmful these lines are in practice for this application, but we

think it's possible to remove them by editing the circuits of curve

neurons.

We don't mean to suggest we've created a competitive

curve tracing algorithm. We haven't done a detailed comparison to state

of the art curve detection algorithms, and believe it's likely that

classical algorithms tuned for precisely this goal outperform our

approach. Instead, our goal here is to explore how leveraging internal

neural network representations opens a vast space of visual operations,

of which curve extraction is just one point.

[### Spline Parameterization](#spline-parameterization)We

 can access more parts of this space by changing what we optimize. So

far we've been optimizing pixels, but we can also create a

Occlusion![](Curve%20Detectors_files/source_006.png)Our

 splines can trace curves even if they have significant occlusion.

Furthermore, we can use attribution to construct complex occlusion

rules. For instance, we can strongly penalize our spline for overlapping

 with a particular object or texture, disincentivizing the spline from

 curve neurons are robust to a wide variety of natural visual features,

 seemingly unrelated visual operation is image segmentation. This can be

 done in an unsupervised way using non-negative matrix factorization

(NMF) . We can

visualize attribution to each of these factors with our spline

parameterization to trace the curves of different objects in the image.

 of factoring the activations of a single image, we can jointly

factorize lots of butterflies to find the neurons in the network that

respond to butterflies in general. One big difference between factoring

activations and normal image segmentation is that we get groups of

neurons rather than pixels. These neuron groups can be applied to find

butterflies in images in general, and by composing this with

differentiable spline parameterization we get a single optimization we

can apply to any image that automatically finds butterflies and gives us

 equations to splines that fit them.

 this above example we manipulated butterflies and curves without having

 to worry about the details of either. We delegated the intricacies of

recognizing butterflies of many species and orientations to the neurons,

 letting us work with the abstract concept of butterflies.

We

think this is one exciting way to fuse classical computer vision with

deep learning. There is plenty of low hanging fruit in extending the

technique shown above, as our spline parameterization is just an early

prototype and our optimizations are using a neural network that's half a

 decade old. However, we're more excited by investigations of how users

can explore the space between tasks than improvements in any particular

task. Once a task is set

in stone, training a neural network for exactly that job will likely

give the best results. But real world tasks are rarely specified with

precision, and the harder challenge is to explore the space of tasks to

find which to commit to.

For instance, a more developed version of

 our algorithm that automatically finds the splines of butterflies in an

 image could be used as a basis for turning video footage of butterflies

 But an animator may wish to add texture neurons and change to a soft

brush parameterization to add a rotoscoping style to their animation.

Since they have full access to every neuron in the render, they could

manipulate attribution to fur neuron families and specific dog breeds,

changing how fur is rendered on specific species of dogs across the

entire movie. Since none of these algorithms require retraining a neural

 network or any training data, in-principle an animator could explore

this space of algorithms in real time, which is important because tight

feedback loops can be crucial in unlocking creative potential.

[The Combing Phenomenon

----------------------](#the-combing-phenomenon)One

 curious aspect of curve detectors is that they seem to be excited by

small lines perpendicular to the curve, both inwards and outwards. You

can see this most easily by inspecting feature visualizations. We call

this phenomenon "combing."

Combing seems to occur across curve

detectors from many models, including models trained on Places365

instead of ImageNet. In fact, there's some weak evidence it occurs in

biological neural networks as well: a team that ran a process similar to

 feature visualization on a biological neuron in a Macaque monkey's V4

region of the visual cortex found a circular shape with outwardly

protruding lines to be one of the highest activating stimuli.

One

 hypothesis is that many important curves in the modern world have

perpendicular lines, such as the spokes of a wheel or the markings on

the rim of a clock.

 related hypothesis is that combing might allow curve detectors to be

used for fur detection in some contexts. Another hypothesis is that a

curve has higher "contrast" with perpendicular lines running towards.

Recall that in the dataset examples, the strongest negative pre-ReLU

activations were curves at opposite orientations. If a curve detector

wants to see a strong change in orientation between the curve and the

space around it, it may consider perpendicular lines to be more contrast

 than a solid color.

Finally, we think it's possible that combing

is really just a convenient way to implement curve detectors — a side

effect of a shortcut in circuit construction rather than an

intrinsically useful feature. In conv2d1, edge detectors are inhibited

by perpendicular lines in conv2d0. One of the things a line or curve

detector needs to do is check that the image is not just a single

repeating texture, but that it has a strong line surrounded by contrast.

 It seems to do this by weakly inhibiting parallel lines alongside the

tangent. Being excited by a perpendicular line may be an easy way to

implement a "inhibit an excitatory neuron" pattern which allows for

capped inhibition, without creating dedicated neurons at the previous

layer.

Combing is not unique to curves. We also observe it in

lines, and basically any shape feature like curves that is derivative of

 lines. A lot more work could be done exploring the combing phenomenon.

Why does combing form? Does it persist in adversarially robust models?

Is it an example of what Ilyas et al  call a "non-robust feature"?

[Conclusion

----------](#conclusion)Compared

 to fields like neuroscience, artificial neural networks make careful

investigation easy. We can read and write to every weight in the neural

network, use gradients to optimize stimuli, and analyze billions of

realistic activations across a dataset. Composing these tools lets us

run a wide range of experiments that show us different perspectives on a

 neuron. If every perspective shows the same story, it's unlikely we're

missing something big.

Given this, it may seem odd to invest so

much energy into just a handful of neurons. We agree. We first estimated

 it would take a week to understand the curve family. Instead, we spent

months exploring the fractal of beauty and structure we found.

Many

 paths led to new techniques for studying neurons in general, like

synthetic stimuli or using circuit editing to ablate neurons behavior.

Others are only relevant for some families, such as the equivariance

motif or our hand-trained "artificial artificial neural network" that

reimplements curve detectors. A couple were curve-specific, like

exploring curve detectors as a type of curve analysis algorithms.

If

 our broader goal is fully reverse-engineer neural networks it may seem

concerning that studying just one family took so much effort. However,

from our experience studying neuron families at a variety of depths,

 shows you feature visualizations, dataset examples, and soon weights in

 just a few seconds. Since feature visualization shows strong evidence

of causal behavior and dataset examples show what neurons respond to in

practice, these are collectively strong evidence of what a neuron does.

In fact, we understood the basics of curves at our first glance at them.

While it's usually possible to understand the main function of a

 neuron family at a glance, researchers engaging in closer inquiry of

neuron families will be rewarded with deeper beauty.

When we started, we were nervous that 10 neurons was too narrow a topic

for a paper, but now we realize a complete investigation would take a

book.

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

Visualizing and understanding convolutional networks

Complex pattern selectivity in macaque primary visual cortex revealed by large-scale two-photon imaging

Distributed representations of words and phrases and their compositionality

Visualizing and understanding recurrent networks

Learning to generate reviews and discovering sentiment

Object detectors emerge in deep scene cnns

Feature Visualization

Network Dissection: Quantifying Interpretability of Deep Visual Representations

On Interpretability and Feature Representations: An Analysis of the Sentiment Neuron

Measuring the tendency of CNNs to Learn Surface Statistical Regularities

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Approximating cnns with bag-of-local-features models works surprisingly well on imagenet

Adversarial examples are not bugs, they are features

On the importance of single directions for generalization

Visualizing higher-layer features of a deep network

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Axiomatic attribution for deep networks

Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization

PatternNet and PatternLRP--Improving the interpretability of neural networks

The (Un)reliability of saliency methods

Sanity checks for saliency maps

Differentiable Image Parameterizations

An overview of early vision in inceptionv1

Responses to contour features in macaque area V4

Discrete neural clusters encode orientation, curvature and corners in macaque V4

Curve tracing and curve detection in images

Synthetic Abstractions

Neural painters: A learned differentiable constraint for generating brushstroke paintings

The Building Blocks of Interpretability

Using Artificial Intelligence to Augment Human Intelligence