Research Debt

 .mountain-illustration {

 margin-top: 20px;

 margin-bottom: 10px;

 width: 180%;

 margin-left: -40%;

 }

 @media (min-width: 768px) {

 .mountain-illustration {

 max-width: 1200px;

 margin-left: auto;

 width: 100%;

 }

 }

 dt-article > h1:first-of-type {

 margin-top: 0;

 }

Achieving a research-level understanding of most topics is like

climbing a mountain. Aspiring researchers must struggle to understand

vast bodies of work that came before them, to learn techniques, and to

gain intuition. Upon reaching the top, the new researcher begins doing

novel work, throwing new stones onto the top of the mountain and making

it a little taller for whoever comes next.

Mathematics is a striking example of this. For centuries,

countless minds have climbed the mountain range of mathematics and laid

new boulders at the top. Over time, different peaks formed, built on top

 of particularly beautiful results. Now the peaks of mathematics are so

numerous and steep that no person can climb them all. Even with a

lifetime of dedicated effort, a mathematician may only enjoy some of

their vistas.

People expect the climb to be hard. It reflects the tremendous

progress and cumulative effort that’s gone into mathematics. The climb

is seen as an intellectual pilgrimage, the labor a rite of passage. But

the climb could be massively easier. It’s entirely possible to build

The climb isn’t progress: the climb is a mountain of debt.

The Debt

--------

Programmers talk about technical debt: there are ways to write

software that are faster in the short run but problematic in the long

run. Managers talk about institutional debt: institutions can grow

quickly at the cost of bad practices creeping in. Both are easy to

accumulate but hard to get rid of.

Research can also have debt. It comes in several forms:

* **Poor Exposition** – Often, there is no good explanation of

important ideas and one has to struggle to understand them. This problem

 is so pervasive that we take it for granted and don’t appreciate how

much better things could be.

* **Undigested Ideas** – Most ideas start off rough and hard to

understand. They become radically easier as we polish them, developing

the right analogies, language, and ways of thinking.

* **Bad abstractions and notation** – Abstractions and notation

are the user interface of research, shaping how we think and

communicate. Unfortunately, we often get stuck with the first formalisms

 to develop even when they’re bad. For example, an object with extra

electrons is negative, and pi is wrong.

* **Noise** – Being a researcher is like standing in the middle

of a construction site. Countless papers scream for your attention and

there’s no easy way to filter or summarize them.Because

 most work is explained poorly, it takes a lot of energy to understand

each piece of work. For many papers, one wants a simple one sentence

explanation of it, but needs to fight with it to get that sentence.

Because the simplest way to get the attention of interested parties is

to get everyone’s attention, we get flooded with work. Because we

The insidious thing about research debt is that it’s normal.

Everyone takes it for granted, and doesn’t realize that things could be

different. For example, it’s normal to give very mediocre explanations

of research, and people perceive that to be the ceiling of explanation

quality. On the rare occasions that truly excellent explanations come

along, people see them as one-off miracles rather than a sign that we

could systematically be doing better.

Interpretive Labor

------------------

There’s a tradeoff between the energy put into explaining an idea,

and the energy needed to understand it. On one extreme, the explainer

can painstakingly craft a beautiful explanation, leading their audience

to understanding without even realizing it could have been difficult. On

 the other extreme, the explainer can do the absolute minimum and

Many explanations are not one-to-one. People give lectures,

write books, or communicate online. In these one-to-many cases, each

member of the audience pays the cost of understanding, even though the

 are coefficients for the trade off between energy on the explanation

side and energy on the understanding side. That is a is the energy spent

 on explaining and b is the corresponding effort needed to understand.

 example, Christopher’s average blog post is read by over 100,000

people; if he can save each reader just one second, he’s saved humanity

30 hours.

![](Research%20Debt_files/publish-one-many-crop.jpg)

In research, we often have a group of researchers all trying to

understand each other. Just like before, the cost of explaining stays

constant as the group grows, but the cost of understanding increases

with each new member. At some size, the effort to understand everyone

else becomes too much. As a defense mechanism, people specialize,

focusing on a narrower area of interest. The maintainable size of the

field is controlled by how its members trade off the energy between

communicating and understanding.

Research debt is the accumulation of missing interpretive

labor. It’s extremely natural for young ideas to go through a stage of

debt, like early prototypes in engineering. The problem is that we often

 stop at that point. Young ideas aren’t ending points for us to put in a

 paper and abandon. When we let things stop there the debt piles up. It

becomes harder to understand and build on each other’s work and the

field fragments.

Clear Thinking

--------------

It’s worth being clear that research debt isn’t just about ideas

not being explained well. It’s a lack of digesting ideas – or, at least,

 a lack of the public version of ideas being digested.Often,

 some individuals have a much more developed version of an idea than is

publicly shared. There are a lot of reasons for not sharing it (in

Developing good abstractions, notations, visualizations, and so

 forth, is improving the user interfaces for ideas. This helps both with

 understanding ideas for the first time and with thinking clearly about

them. Conversely, if we can’t explain an idea well, that’s often a sign

that we don’t understand it as well as we could.

It shouldn’t be that surprising that these two largely go hand

in hand. Part of thinking is having a conversation with ourselves.

Research Distillation

---------------------

Research distillation is the opposite of research debt. It can be

incredibly satisfying, combining deep scientific understanding, empathy,

 and design to do justice to our research and lay bare beautiful

insights.

Distillation is also hard. It’s tempting to think of explaining

 an idea as just putting a layer of polish on it, but good explanations

often involve transforming the idea. This kind of refinement of an idea

can take just as much effort and deep understanding as the initial

discovery.

This leaves us with no easy way out. We can’t solve research

debt by having one person write a textbook: their energy is spread too

thin to polish every idea from scratch. We can’t outsource distillation

to less skilled non-experts: refining and explaining ideas requires

creativity and deep understanding, just as much as novel research.

Research distillation doesn’t have to be you, but it does have to be us.

Where are the Distillers?

-------------------------

Like the theoretician, the experimentalist or the research

engineer, the research distiller is an integral role for a healthy

research community. Right now, almost no one is filling it.

Why do researchers not work on distillation? One possibility is

 perverse incentives, like wanting your work to look difficult. Those

certainly exist, but we don’t think they’re the main factor.

 There are a lot of perverse incentives that push against explaining

things well, sharing data, and so forth. This is especially true when

the work you are doing isn’t that interesting or isn’t reproducible and

you want to obscure that. Or if you have a lot of competitors and don’t

want them to catch up.

However, our experience is that most good

researchers don’t seem that motivated by these kind of factors. Instead,

 the main issue is that it isn’t worthwhile for them to divert energy

from pursuing results to distill things. Perhaps things are different in

 other fields, or I’m not cynical enough.

Lots of people want to work on research distillation.

Unfortunately, it’s very difficult to do so, because we don’t support

them.There is a strange kind of informal

support for people working on research distillation. Christopher has

personally benefitted a great deal from this. But it’s unreliable and

not widely advertised, which makes it hard to build a career on.

An aspiring research distiller lacks many things that are easy

 to take for granted: a career path, places to learn, examples and role

models. Underlying this is a deeper issue: their work isn’t seen as a

real research contribution. We need to fix this.

An Ecosystem for Distillation

-----------------------------

If you are excited to distill ideas, seek clarity, and build

beautiful explanations, we are letting you down. You have something

precious to contribute and we aren’t supporting you the way we should.

* Distill Infrastructure – Tools for making beautiful interactive essays.

This is just a start: there’s a lot more that needs to be done. A

complete ecosystem for this kind of work needs several other components,

 including places where one can learn these skills and reliable sources

of employment doing this kind of work. We’re optimistic that will come

with time.

---

Further Reading

---------------

![](Research%20Debt_files/headstand.jpg)

* **Visual Mathematics:**

 is particularly striking, but there are several lovely examples of new

* **Explorable Explanations:**

 There’s a loose community exploring how the interactive medium

enabled by computers can be used to communicate and think in ways

previously impossible. These ideas start, as many ideas in computing do,

 with work done by Douglas Engelbart and Alan Kay.

There are also explorations of how we can augment our ability

 to think in this new medium, bringing previously inaccessible ideas

* **Research Distribution:**

* **Open-Notebook Science:**

 This seems really important. Traditionally, if one doesn’t turn

research into a paper, it’s basically as though you didn’t do it. This

creates a strong incentive for all research to be dressed up as an

important paper, increasing noise.

* **Discussion of Debt and Distillation:**

 A number of mathematicians have discussed what we call research

 I’d draw particular attention to Thurston’s account of accidentally

killing a field by drowning it in research debt in section 6 of .

The Tau Manifesto

Interpretive Labor

The utopia of rules: On technology, stupidity, and the secret joys of bureaucracy

The Mythical Man-Month

Visual Complex Analysis

Visual Group Theory

A visual explanation of Jensen's inequality

Visual Differential Geometry

A Research Center for Augmenting Human Intellect

Personal Dynamic Media

Explorable explanations

Visualizing Algorithms

How to Fold a Julia Fractal

Back to the Future of Handwriting Recognition

Media for thinking the unthinkable

Learnable programming

Thought as a Technology

Toward an Exploratory Medium for Mathematics

On Proof and Progress in Mathematics

A beginner’s guide to forcing

Recoltes et Semailles

Statement on conceptual contributions in theory


Self-Organising Textures

Eyvind Niklasson
Alexander Mordvintsev
Ettore Randazzo
Michael Levin

Abstract: This article is part of the     Differentiable Self-organizing Systems Thread,     an experimental format collecting invited short articles delving into     differentiable self-organizing systems, interspersed with critical     commentary from several experts in adjacent fields.

### Contents

* [NCA as pattern generators](#nca-as-pattern-generators)

* [Related work](#related-work)

[Feature Visualization](#feature-visualization)

* [NCA with Inception](#nca-with-inception)

[Other interesting findings](#other-interesting-findings)

* [Robustness](#robustness)

* [Hidden States](#hidden-states)

[Conclusion](#conclusion)

![](Self-Organising%20Textures_files/multiple-pages.svg)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Self-classifying MNIST Digits](https://distill.pub/2020/selforg/mnist/)

 The inductive bias imposed by using cellular automata is powerful. A

system of individual agents running the same learned local rule can

solve surprisingly complex tasks. Moreover, individual agents, or cells,

 can learn to coordinate their behavior even when separated by large

distances. By construction, they solve these tasks in a massively

 way. Each cell must be able to take on the role of any other cell - as a

 result they tend to generalize well to unseen situations.

In this work, we apply NCA to the task of texture synthesis. This

task involves reproducing the general appearance of a texture template,

as opposed to making pixel-perfect copies. We are going to focus on

texture losses that allow for a degree of ambiguity. After training NCA

models to reproduce textures, we subsequently investigate their learned

behaviors and observe a few surprising effects. Starting from these

investigations, we make the case that the cells learn distributed,

local, algorithms.

Patterns, textures and physical processes

-----------------------------------------

![](Self-Organising%20Textures_files/zebra.jpg)

A pair of Zebra. Zebra are said to have unique stripes.

Zebra stripes are an iconic texture. Ask almost anyone to identify

zebra stripes in a set of images, and they will have no trouble doing

so. Ask them to describe what zebra stripes look like, and they will

gladly tell you that they are parallel stripes of slightly varying

width, alternating in black and white. And yet, they may also tell you

that no two zebra have the same set of stripes

 Perhaps an apocryphal claim, but at the very lowest level every zebra

will be unique. Ourp point is - “zebra stripes” as a concept in human

understanding refers to the general structure of a black and white

striped pattern and not to a specific mapping from location to colour..

 This is because evolution has programmed the cells responsible for

creating the zebra pattern to generate a pattern of a certain quality,

with certain characteristics, as opposed to programming them with the

blueprints for an exact bitmap of the edges and locations of stripes to

be moulded to the surface of the zebra’s body.

Put another way, patterns and textures are ill-defined concepts. The

Cambridge English Dictionary defines a pattern as “any regularly

repeated arrangement, especially a design made from repeated lines,

shapes, or colours on a surface”. This definition falls apart rather

quickly when looking at patterns and textures that impart a feeling or

quality, rather than a specific repeating property. A coloured fuzzy

rug, for instance, can be considered a pattern or a texture, but is

composed of strands pointing in random directions with small random

variations in size and color, and there is no discernable regularity to

the pattern. Penrose tilings do not repeat (they are not translationally

 invariant), but show them to anyone and they’ll describe them as a

pattern or a texture. Most patterns in nature are outputs of locally

interacting processes that may or may not be stochastic in nature, but

are often based on fairly simple rules. There is a large body of work on

 models which give rise to such patterns in nature; most of it is

inspired by Turing’s seminal paper on morphogenesis.

Such patterns are very common in developmental biology .

 In addition to coat colors and skin pigmentation, invariant large-scale

 patterns, arising in spite of stochastic low-level dynamics, are a key

feature of peripheral nerve networks, vascular networks, somites (blocks

 of tissue demarcated in embryogenesis that give rise to many organs),

and segments of anatomical and genetic-level features, including whole

body plans (e.g., snakes and centipedes) and appendages (such as

demarcation of digit fields within the vertebrate limb).

 These kinds of patterns are generated by reaction-diffusion processes,

bioelectric signaling, planar polarity, and other cell-to-cell

communication mechanisms.

 Patterns in biology are not only structural, but also physiological, as

 in the waves of electrical activity in the brain and the dynamics of

gene regulatory networks. These gene regulatory networks, for example,

can support computation sufficiently sophisticated as to be subject to

Liar paradoxes See [liar paradox](https://en.wikipedia.org/wiki/Liar_paradox).

 In principle, gene regulatory networks can express paradoxical

behaviour, such as that expression of factor A represses the expression

 Studying the emergence and control of such patterns can help us to

understand not only their evolutionary origins, but also how they are

recognized (either in the visual system of a second observer or in

adjacent cells during regeneration) and how they can be modulated for

the purposes of regenerative medicine.

As a result, when having any model learn to produce textures or

patterns, we want it to learn a generative process for the pattern. We

can think of such a process as a means of sampling from the distribution

 governing this pattern. The first hurdle is to choose an appropriate

loss function, or qualitative measure of the pattern. To do so, we

employ ideas from Gatys et. al .

 NCA become the parametrization for an image which we “stylize” in the

style of the target pattern. In this case, instead of restyling an

existing image, we begin with a fully unconstrained setting: the output

of an untrained, randomly initialized, NCA. The NCA serve as the

“renderer” or “generator”, and a pre-trained differentiable model serves

 as a distinguisher of the patterns, providing the gradient necessary

for the renderer to learn to produce a pattern of a certain style.

### From Turing, to Cellular Automata, to Neural Networks

NCA are well suited for generating textures. To understand why, we’ll

 demonstrate parallels between texture generation in nature and NCA.

Given these parallels, we argue that NCA are a good model class for

texture generation.

#### PDEs

In “The Chemical Basis of Morphogenesis” ,

 Alan Turing suggested that simple physical processes of reaction and

diffusion, modelled by partial differential equations, lie behind

pattern formation in nature, such as the aforementioned zebra stripes.

Extensive work has since been done to identify PDEs modeling

reaction-diffusion and evaluating their behaviour. One of the more

celebrated examples is the Gray-Scott model of reaction diffusion (,).

 This process has a veritable zoo of interesting behaviour, explorable

by simply tuning the two parameters. We strongly encourage readers to

visit this [interactive atlas](http://mrob.com/pub/comp/xmorphia/)

 of the different regions of the Gray-Scott reaction diffusion model to

get a sense for the extreme variety of behaviour hidden behind two

To tackle the problem of reproducing our textures, we propose a more

general version of the above systems, described by a simple Partial

Differential Equation (PDE) over the state space of an image.

Intuitively, we have defined a system where every point of the image

changes with time, in a way that depends on how the image currently

changes across space, with respect to its immediate neighbourhood.

Readers may start to recognize the resemblance between this and another

system based on immediately local interactions.

#### To CAs

Differential equations governing natural phenomena are usually

evaluated using numerical differential equation solvers. Indeed, this is

 sometimes the **only** way to solve them, as many PDEs and

 ODEs of interest do not have closed form solutions. This is even the

 Numerically solving PDEs and ODEs is a vast and well-established field.

 One of the biggest hammers in the metaphorical toolkit for numerically

evaluating differential equations is discretization: the process of

converting the variables of the system from continuous space to a

discrete space, where numerical integration is tractable. When using

some ODEs to model a change in a phenomena over time, for example, it

makes sense to advance through time in discrete steps, possibly of

variable size.

We now show that numerically integrating the aforementioned PDE is

The logical approach to discretizing the space the PDE operates on is

 to discretize the continuous 2D image space into a 2D raster grid.

Boundary conditions are of concern but we can address them by moving to a

 toroidal world where each dimension wraps around on itself.

Similarly to space, we choose to treat time in a discretized fashion

and evaluate our NCA at fixed-sized time steps. This is equivalent to

explicit Euler integration. However, here we make an important deviation

 from traditional PDE numerical integration methods for two reasons.

First, if all cells are updated synchronously, initial conditions s0s\_0s0

 must vary from cell-to-cell in order to break the symmetry. Second, the

 physical implementation of the synchronous model would require the

existence of a global clock, shared by all cells. One way to work around

 the former is by initializing the grid with random noise, but in the

spirit of self organisation we instead choose to decouple the cell

updates by asynchronously evaluating the CA. We sample a subset of all

cells at each time-step to update. This introduces both asynchronicity

in time (cells will sometimes operate on information from their

neighbours that is several timesteps old), and asymmetry in space,

solving both aforementioned issues.

Our next step towards representing a PDE with cellular automata is to

\begin{bmatrix}

-1 & 0 & 1\\-2 & 0 & 2 \\-1 & 0 & 1

\end{bmatrix}

&

\begin{bmatrix}

-1 & -2 & -1\\ 0 & 0 & 0 \\1 & 2 & 1

\end{bmatrix}

&

\begin{bmatrix}

1 & 2 & 1\\2 & -12 & 2 \\1 & 2 & 1

\end{bmatrix}

\\

Sobel\_x & Sobel\_y & Laplacian

With all the pieces in place, we now have a space-discretized version

 of our PDE that looks very much like a Cellular Automata: the time

evolution of each discrete point in the raster grid depends only on its

immediate neighbours. These discrete operators allow us to formalize our

 PDE as a CA. To double check that this is true, simply observe that as

our grid becomes very fine, and the asynchronous updates approach

uniformity, the dynamics of these discrete operators will reproduce the

continuous dynamics of the original PDE as we defined it.

#### To Neural Networks

The final step in implementing the above general PDE for texture

generation is to translate it to the language of deep learning.

Fortunately, all the operations involved in iteratively evaluating the

generalized PDE exist as common operations in most deep learning

frameworks. We provide both a Tensorflow and a minimal PyTorch

implementation for reference, and refer readers to these for details on

our implementation.

### NCA as pattern generators

#### Model:

![](Self-Organising%20Textures_files/texture_model.svg)

Texture NCA model.

We build on the Growing CA NCA model ,

 complete with built-in quantization of weights, stochastic updates, and

 the batch pool mechanism to approximate long-term training. For further

 details on the model and motivation, we refer readers to this work.

#### Loss function:

![](Self-Organising%20Textures_files/texture_training.svg)

Texture NCA model.

 in the form of the raw activation values of the neurons in these

layers. Finally, we run our NCA forward for between 32 and 64

 of activations of these neurons with the NCA as input and their

activations with the template image as input. We keep the weights of VGG

 frozen and use ADAM  to update the weights of the NCA.

#### Dataset:

 The aim of this dataset is to provide a benchmark for measuring the

ability of vision models to recognize and categorize textures and

describe textures using words. The textures were collected to match 47

“attributes” such as “bumpy” or “polka-dotted”. These 47 attributes were

 in turn distilled from a set of common words used to describe textures

identified by Bhusan, Rao and Lohse .

#### Results:

After a few iterations of training, we see the NCA converge to a

solution that at first glance looks similar to the input template, but

not pixel-wise identical. The very first thing to notice is that the

This is not completely unexpected. In *Differentiable Parametrizations*,

 the authors noted that the images produced when backpropagating into

image space would end up different each time the algorithm was run due

to the stochastic nature of the parametrizations. To work around this,

they introduced some tricks to maintain **alignment**

between different visualizations. In our model, we find that we attain

such alignment along the temporal dimension without optimizing for it; a

 welcome surprise. We believe the reason is threefold. First, reaching

and maintaining a static state in an NCA appears to be non-trivial in

comparison to a dynamic one, so much so that in Growing CA a pool of NCA

 states at various iteration times had to be maintained and sampled as

starting states to simulate loss being applied after a time period

longer than the NCAs iteration period, to achieve a static stability. We

 employ the same sampling mechanism here to prevent the pattern from

decaying, but in this case the loss doesn’t enforce a static fixed

target; rather it guides the NCA towards any one of a number of states

that minimizes the style loss. Second, we apply our loss after a random

number of iterations of the NCA. This means that, at any given time

step, the pattern must be in a state that minimizes the loss. Third, the

 stochastic updates, local communication, and quantization all limit and

 regularize the magnitude of updates at each iteration. This encourages

changes to be small between one iteration and the next. We hypothesize

that these properties combined encourage the NCA to find a solution

where each iteration is **aligned** with the previous

iteration. We perceive this alignment through time as motion, and as we

iterate the NCA we observe it traversing a manifold of locally aligned

solutions.

 based on the aforementioned findings and qualitative observation of the

 NCA. We proceed to demonstrate some exciting behaviours of NCA trained

on different template images.

An NCA trained to create a pattern in the style of **chequered\_0121.jpg**.

We notice that:

* Initially, a non-aligned grid of black and white quadrilaterals is formed.

 to more closely approximate squares. Quadrilaterals of both colours

either emerge or disappear. Both of these behaviours seem to be an

attempt to find local consistency.

* After a longer time, the grid tends to achieve perfect consistency.

Such behaviour is not entirely unlike what one would expect in a

hand-engineered algorithm to produce a consistent grid with local

communication. For instance, one potential hand-engineered approach

would be to have cells first try and achieve local consistency, by

choosing the most common colour from the cells surrounding them, then

attempting to form a diamond of correct size by measuring distance to

the four edges of this patch of consistent colour, and moving this

boundary if it were incorrect. Distance could be measured by using a

hidden channel to encode a gradient in each direction of interest, with

each cell decreasing the magnitude of this channel as compared to its

neighbour in that direction. A cell could then localize itself within a

diamond by measuring the value of two such gradient channels. The

appearance of such an algorithm would bear resemblance to the above -

with patches of cells becoming either black, or white, diamonds then

resizing themselves to achieve consistency.

An NCA trained to create a pattern in the style of **bubbly\_0101.jpg**.

In this video, the NCA has learned to reproduce a texture based on a

template of clear bubbles on a blue background. One of the most

interesting behaviours we observe is that the density of the bubbles

remains fairly constant. If we re-initialize the grid states, or

interactively destroy states, we see a multitude of bubbles re-forming.

However, as soon as two bubbles get too close to each other, one of them

 spontaneously collapses and disappears, ensuring a constant density of

If we speed the animation up, we see that different bubbles move at

different speeds, yet they never collide or touch each other. Bubbles

also maintain their structure by self-correcting; a damaged bubble can

re-grow.

This behaviour is remarkable because it arises spontaneously, without

 any external or auxiliary losses. All of these properties are learned

from a combination of the template image, the information stored in the

layers of VGG, and the inductive bias of the NCA. The NCA learned a rule

 that effectively approximates many of the properties of the bubbles in

the original image. Moreover, it has learned a process that generates

this pattern in a way that is robust to damage and looks realistic to

humans.

An NCA trained to create a pattern in the style of **interlaced\_0172.jpg**.

Here we see one of our favourite patterns: a simple geometric

“weave”. Again, we notice the NCA seems to have learned an algorithm for

 producing this pattern. Each “thread” alternately joins or detaches

from other threads in order to produce the final pattern. This is

strikingly similar to what one would attempt to implement, were one

asked to programmatically generate the above pattern. One would try to

design some sort of stochastic algorithm for weaving individual threads

together with other nearby threads.

An NCA trained to create a pattern in the style of **banded\_0037.jpg**.

Here, misaligned stripe fragments travel up or down the stripe until

either they merge to form a single straight stripe or a stripe shrinks

and disappears. Were this to be implemented algorithmically with local

communication, it is not infeasible that a similar algorithm for finding

 consistency among the stripes would be used.

### Related work

This foray into pattern generation is by no means the first. There

has been extensive work predating deep-learning, in particular

suggesting deep connections between spatial patterning of anatomical

structure and temporal patterning of cognitive and computational

processes (e.g., reviewed in ).

 Hans Spemann, one of the heroes of classical developmental biology,

said “Again and again terms have been used which point not to physical

but to psychical analogies. It was meant to be more than a poetical

metaphor. It was meant to my conviction that the suitable reaction of a

germ fragment, endowed with diverse potencies, in an embryonic ‘field’…

is not a common chemical reaction, like all vital processes, are

comparable, to nothing we know in such degree as to vital processes of

which we have the most intimate knowledge.” .

 More recently, Grossberg quantitatively laid out important

similarities between developmental patterning and computational

neuroscience . As briefly touched

upon, the inspiration for much of the work came from Turing’s work on

pattern generation through local interaction, and later papers based on

this principle. However, we also wish to acknowledge some works that we

feel have a particular kinship with ours.

#### Patch sampling

Early work in pattern generation focused on texture sampling. Patches

 were often sampled from the original image and reconstructed or

rejoined in different ways to obtain an approximation of the texture.

This method has also seen recent success with the work of Gumin .

#### Deep learning

Gatys et. al’s work ,

referenced throughout, has been seminal with regards to the idea that

statistics of certain layers in a pre-trained network can capture

textures or styles in an image. There has been extensive work building

on this idea, including playing with other parametrisations for image

generation  and optimizing the generation process .

Other work has focused on using a convolutional generator combined

with path sampling and trained using an adversarial loss to produce

textures of similar quality .

#### Interactive Evolution of Camouflage

 Craig Reynolds uses a texture description language, consisting of

generators and operators, to parametrize a texture patch, which is

presented to human viewers who have to decide which patches are the

worst at “camouflaging” themselves against a chosen background texture.

The population is updated in an evolutionary fashion to maximize

“camouflage”, resulting in a texture exhibiting the most camouflage (to

human eyes) after a number of iterations. We see strong parallels with

our work - instead of a texture generation language, we have an NCA

parametrize the texture, and instead of human reviewers we use VGG as an

 evaluator of the quality of a generated pattern. We believe a

fundamental difference lies in the solution space of an NCA. A texture

generation language comes with a number of inductive biases and learns a

 deterministic mapping from coordinates to colours. Our method appears

to learn more general algorithms and behaviours giving rise to the

target pattern.

Feature visualization

---------------------

![](Self-Organising%20Textures_files/butterfly_eye.jpg)

A butterfly with an “eye-spot” on the wings.

We have now explored some of the fascinating behaviours learned by

the NCA when presented with a template image. What if we want to see

them learn even more “unconstrained” behaviour?

Some butterflies have remarkably lifelike eyes on their wings. It’s

unlikely the butterflies are even aware of this incredible artwork on

their own bodies. Evolution placed these there to trigger a response of

fear in potential predators or to deflect attacks from them .

 It is likely that neither the predator nor the butterfly has a concept

 regarding the consciousness of the other, but evolution has identified a

 region of morphospace for this organism that exploits

pattern-identifying features of predators to trick them into fearing a

harmless bug instead of consuming it.

Even more remarkable is the fact that the individual cells composing

the butterfly’s wings can self assemble into coherent, beautiful, shapes

 The coordination required to produce these features implies

self-organization over hundreds or thousands of cells to generate a

coherent image of an eye that evolved simply to act as a visual stimuli

for an entirely different species, because of the local nature of

cell-to-cell communication. Of course, this pales in comparison to the

morphogenesis that occurs in animal and plant bodies, where structures

consisting of millions of cells will specialize and coordinate to

generate the target morphology.

 Just as neuroscientists and biologists have often treated cells and

cell structures and neurons as black-box models to be investigated,

measured and reverse-engineered, there is a large contemporary body of

work on doing the same with neural networks. For instance the work by

Boettiger  .

We can explore this idea with minimal effort by taking our

pattern-generating NCA and exploring what happens if we task it to enter

 a state that excites a given neuron in Inception. One of the common

resulting NCAs we notice is eye and eye-related shapes - such as the

video below - likely as a result of having to detect various animals in

ImageNet. In the same way that cells form eye patterns on the wings of

butterflies to excite neurons in the brains of predators, our NCA’s

population of cells has learned to collaborate to produce a pattern that

 excites certain neurons in an external neural network.

An NCA trained to excite **mixed4a\_472** in Inception.

### NCA with Inception

#### Model:

We use a model identical to the one used for exploring pattern

generation, but with a different discriminator network: Imagenet-trained

 Inception v1 network .

#### Loss function:

Our loss maximizes the activations of chosen neurons, when evaluated

on the output of the NCA. We add an auxiliary loss to encourage the

#### Dataset:

There is no explicit dataset for this task. Inception is trained on

ImageNet. The layers and neurons we chose to excite are chosen

qualitatively using OpenAI Microscope.

#### Results:

Similar to the pattern generation experiment, we see quick

convergence and a tendency to find temporally dynamic solutions. In

other words, resulting NCAs do not stay still. We also observe that the

majority of the NCAs learn to produce solitons of various kinds. We

discuss a few below, but encourage readers to explore them in the demo.

An NCA trained to excite **mixed4c\_439** in Inception.

Solitons in the form of regular circle-like shapes with internal

structure are quite commonly observed in the inception renderings. Two

solitons approaching each other too closely may cause one or both of

them to decay. We also observe that solitons can divide into two new

solitons.

An NCA trained to excite **mixed3b\_454** in Inception.

In textures that are composed of threads or lines, or in certain

excitations of Inception neurons where the resulting NCA has a

“thread-like” quality, the threads grow in their respective directions

and will join other threads, or grow around them, as required. This

behaviour is similar to the regular lines observed in the striped

patterns during pattern generation.

Other interesting findings

--------------------------

### Robustness

#### Switching manifolds

We encode local information flow within the NCA using the same fixed

Laplacian and gradient filters. As luck would have it, these can be

defined for most underlying manifolds, giving us a way of placing our

cells on various surfaces and in various configurations without having

to modify the learned model. Suppose we want our cells to live in a

hexagonal world. We can redefine our kernels as follows:

![](Self-Organising%20Textures_files/hex_kernels.svg)

Hexagonal grid convolutional filters.

Our model, trained in a purely square environment, works out of the

box on a hexagonal grid! Play with the corresponding setting in the demo

 to experiment with this. Zooming in allows observation of the

individual hexagonal or square cells. As can be seen in the demo, the

cells have no problem adjusting to a hexagonal world and producing

identical patterns after a brief period of re-alignment.

![](Self-Organising%20Textures_files/coral_square.png)

![](Self-Organising%20Textures_files/coral_hex.png)

The same texture evaluated on a square and hexagonal grid, respectively.

#### Rotation

![](Self-Organising%20Textures_files/mond_rot0.png)

![](Self-Organising%20Textures_files/mond_rot1.png)

![](Self-Organising%20Textures_files/mond_rot2.png)

![](Self-Organising%20Textures_files/mond_rot3.png)

eralises to this new rotated paradigm without issue.

In theory, the cells can be evaluated on any manifold where one can

define approximations to the Sobel kernel and the Laplacian kernel. We

demonstrate this in our demo by providing an aforementioned “hexagonal”

world for the cells to live in. Instead of having eight equally-spaced

neighbours, each cell now has six equally-spaced neighbours. We further

demonstrate this versatility by rotating the Sobel and Laplacian

kernels. Each cell receives an innate global orientation based on these

kernels, because they are defined with respect to the coordinate system

of the state. Redefining the Sobel and Laplacian kernel with a rotated

coordinate system is straightforward and can even be done on a per-cell

level. Such versatility is exciting because it mirrors the extreme

robustness found in biological cells in nature. Cells in most tissues

will generally continue to operate whatever their location, direction,

or exact placement relative to their neighbours. We believe this

versatility in our model could even extend to a setting where the cells

are placed on a manifold at random, rather than on an ordered grid.

#### Time-synchronization

Two NCAs running next to each other, at different speeds,

with some stochasticity in speed. They can communicate through their

shared edge; the vertical boundary between them in the center of the

state space.

Stochastic updates teach the cells to be robust to asynchronous

updates. We investigate this property by taking it to an extreme and

 The result is surprisingly stable; the CA is still able to construct

and maintain a consistent texture across the combined manifold. The time

 discrepancy between the two CAs sharing the state is far larger than

anything the NCA experiences during training, showing remarkable

robustness of the learned behaviour. Parallels can be drawn to organic

matter self repairing, for instance a fingernail can regrow in adulthood

 despite the underlying finger already having fully developed; the two

do not need to be sync. This result also hints at the possibility of

designing distributed systems without having to engineer for a global

clock, synchronization of compute units or even homogenous compute

capacity.

An NCA is evaluated for a number of steps. The surrounding

border of cells are then also turned into NCA cells. The cells have no

difficulty communicating with the “finished” pattern and achieving

consistency.

An even more drastic example of this robustness to time

asynchronicity can be seen above. Here, an NCA is iterated until it

achieves perfect consistency in a pattern. Then, the state space is

expanded, introducing a border of new cells around the existing state.

This border quickly interfaces with the existing cells and settles in a

consistent pattern, with almost no perturbation to the already-converged

 inner state.

#### Failure cases

The failure modes of a complex system can teach us a great deal about

 its internal structure and process. Our model has many quirks and

sometimes these prevent it from learning certain patterns. Below are

some examples.

![](Self-Organising%20Textures_files/fail_mondrian.jpeg)

![](Self-Organising%20Textures_files/fail_sprinkle.jpeg)

![](Self-Organising%20Textures_files/fail_chequerboard.jpeg)

Three failure cases of the NCA. Bottom row shows target

texture samples, top row are corresponding NCA outputs. Failure modes

include incorrect colours, chequerboard artefacts, and incoherent image

structure.

Some patterns are reproduced somewhat accurately in terms of

structure, but not in colour, while some are the opposite. Others fail

completely. It is difficult to determine whether these failure cases

have their roots in the parametrization (the NCA), or in the

hard-to-interpret gradient signals from VGG, or Inception. Existing work

 with style transfer suggests that using a loss on Gram matrices in VGG

can introduce instabilities ,

 that are similar to the ones we see here. We hypothesize that this

effect explains the failures in reproducing colors. The structural

failures, meanwhile, may be caused by the NCA parameterization, which

makes it difficult for cells to establish long-distance communication

with one another.

### Hidden states

When biological cells communicate with each other, they do so through

 a multitude of available communication channels. Cells can emit or

absorb different ions and proteins, sense physical motion or “stiffness”

 of other cells, and even emit different chemical signals to diffuse

over the local substrate .

There are various ways to visualize communication channels in real

cells. One of them is to add to cells a potential-activated dye. Doing

so gives a clear picture of the voltage potential the cell is under with

 respect to the surrounding substrate. This technique provides useful

insight into the communication patterns within groups of cells and helps

 scientists visualize both local and global communication over a variety

 of time-scales.

As luck would have it, we can do something similar with our Neural

Cellular Automata. Our NCA model contains 12 channels. The first three

are visible RGB channels and the rest we treat as latent channels which

are visible to adjacent cells during update steps, but excluded from

loss functions. Below we map the first three principle components of the

 hidden channels to the R,G, and B channels respectively. Hidden

channels can be considered “floating,” to abuse a term from circuit

theory. In other words, they are not pulled to any specific final state

or intermediate state by the loss. Instead, they converge to some form

of a dynamical system which assists the cell in fulfilling its objective

 with respect to its visible channels. There is no pre-defined

assignment of different roles or meaning to different hidden channels,

and there is almost certainly redundancy and correlation between

different hidden channels. Such correlation may not be visible when we

visualize the first three principal components in isolation. But this

concern aside, the visualization yields some interesting insights

anyways.

 An NCA trained to excite **mixed4b\_70**

 in Inception. Notice the hidden states appear to encode information

about structure. “Threads” along the major diagonal (NW - SE) appear

primarily green, while those running along the anti-diagonal appear

blue, indicating that these have differing internal states, despite

being effectively indistinguishable in RGB space.

In the principal components of this coral-like texture, we see a

pattern which is similar to the visible channels. However, the “threads”

 pointing in each diagonal direction have different colours - one

diagonal is green and the other is a pale blue. This suggests that one

of the things encoded into the hidden states is the direction of a

“thread”, likely to allow cells that are inside one of these threads to

keep track of which direction the thread is growing, or moving, in.

 An NCA trained to produce a texture based on DTD image **cheqeuered\_0121**.

 Notice the structure of squares - with a gradient occurring inside the

structure of each square, evidencing that structure is being encoded in

hidden state.

The chequerboard pattern likewise lends itself to some qualitative

analysis and hints at a fairly simple mechanism for maintaining the

shape of squares. Each square has a clear gradient in PCA space across

the diagonal, and the values this gradient traverses differ for the

white and black squares. We find it likely the gradient is used to

provide a local coordinate system for creating and sizing the squares.

 An NCA trained to excite **mixed4c\_208**

 in Inception. The visible body of the eye is clearly demarcated in the

hidden states. There is also a “halo” which appears to modulate growth

of any solitons immediately next to each other. This halo is barely

visible in the RGB channels.

We find surprising insight in NCA trained on Inception as well. In

this case, the structure of the eye is clearly encoded in the hidden

state with the body composed primarily of one combination of principal

components, and an halo, seemingly to prevent collisions of the eye

solitons, composed of another set of principal components.

Analysis of these hidden states is something of a dark art; it is not

 always possible to draw rigorous conclusions about what is happening.

We welcome future work in this direction, as we believe qualitative

analysis of these behaviours will be useful for understanding more

complex behaviours of CAs. We also hypothesize that it may be possible

to modify or alter hidden states in order to affect the morphology and

behaviour of NCA.

Conclusion

----------

In this work, we selected texture templates and individual neurons as

 targets and then optimized NCA populations so as to produce similar

excitations in a pre-trained neural network. This procedure yielded NCAs

 that could render nuanced and hypnotic textures. During our analysis,

we found that these NCAs have interesting and unexpected properties.

Many of the solutions for generating certain patterns in an image appear

 similar to the underlying model or physical behaviour producing the

pattern. For example, our learned NCAs seem to have a bias for treating

objects in the pattern as individual objects and letting them move

freely across space. While this effect was present in many of our

models, it was particularly strong in the bubble and eye models. The NCA

 is forced to find algorithms that can produce such a pattern with

purely local interaction. This constraint seems to produce models that

favor high-level consistency and robustness.

![](Self-Organising%20Textures_files/multiple-pages.svg)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Self-classifying MNIST Digits](https://distill.pub/2020/selforg/mnist/)

Growing Neural Cellular Automata

Image segmentation via Cellular Automata

Self-classifying MNIST Digits

Differentiable Image Parameterizations

The chemical basis of morphogenesis

Turing patterns in development: what about the horse part?

A unified design space of synthetic stripe-forming networks

On the Formation of Digits and Joints during Limb Development

Modeling digits. Digit patterning is controlled by a Bmp-Sox9-Wnt Turing network modulated by morphogen gradients

Pattern formation mechanisms of self-organizing reaction-diffusion systems

Bioelectric
 gene and reaction networks: computational modelling of genetic,
biochemical and bioelectrical dynamics in pattern regulation

Turing-like patterns can arise from purely bioelectric mechanisms

Dissipative structures in biological systems: bistability, oscillations, spatial patterns and waves

Gene networks and liar paradoxes

Texture Synthesis Using Convolutional Neural Networks

The chemical basis of morphogenesis. 1953

Pattern formation by interacting chemical fronts

Complex patterns in a simple system

Very Deep Convolutional Networks for Large-Scale Image Recognition

Adam: A Method for Stochastic Optimization

Describing Textures in the Wild

The texture lexicon: Understanding the categorization of visual texture terms and their relationship to texture images

Re-membering
 the body: applications of computational neuroscience to the top-down
control of regeneration of limbs and other complex organs

Embryonic Development and Induction

Communication, Memory, and Development

WaveFunctionCollapse

Texture Networks: Feed-forward Synthesis of Textures and Stylized Images

TextureGAN: Controlling deep image synthesis with texture patches

Interactive evolution of camouflage

A parametric texture model based on joint statistics of complex wavelet coefficients

Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration

The evolutionary significance of butterfly eyespots

Live Cell Imaging of Butterfly Pupal and Larval Wings In Vivo

Focusing on butterfly eyespot focus: uncoupling of white spots from eyespot bodies in nymphalid butterflies

OpenAI Microscope

The neural origins of shell structure and pattern in aquatic mollusks

Emergent complexity in simple neural systems

Going deeper with convolutions

Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses

Stem cell migration and mechanotransduction on linear stiffness gradient hydrogels


Sequence Modeling with CTC

Introduction

------------

Consider speech recognition. We have a dataset of audio clips and

corresponding transcripts. Unfortunately, we don’t know how the characters in

the transcript align to the audio. This makes training a speech recognizer

harder than it might at first seem.

Without this alignment, the simple approaches aren’t available to us. We

could devise a rule like “one character corresponds to ten inputs”. But

people’s rates of speech vary, so this type of rule can always be broken.

Another alternative is to hand-align each character to its location in the

audio. From a modeling standpoint this works well — we’d know the ground truth

for each input time-step. However, for any reasonably sized dataset this is

prohibitively time consuming.

This problem doesn’t just turn up in speech recognition. We see it in many

other places. Handwriting recognition from images or sequences of pen strokes

is one example. Action labelling in videos is another.

![](Sequence%20Modeling%20with%20CTC_files/handwriting_recognition.svg)

**Handwriting recognition:** The input can be

 (x,y)(x,y)(x,y) coordinates of a pen stroke or

 pixels in an image.

![](Sequence%20Modeling%20with%20CTC_files/speech_recognition.svg)

**Speech recognition:** The input can be a spectrogram or some

 other frequency based feature extractor.

Connectionist Temporal Classification (CTC) is a way to get around not

knowing the alignment between the input and the output. As we’ll see, it’s

especially well suited to applications like speech and handwriting

recognition.

---

To be a bit more formal, let’s consider mapping input sequences

We want to find an accurate mapping from XXX’s to YYY’s.

There are challenges which get in the way of us

using simpler supervised learning algorithms. In particular:

* Both XXX and YYY

 can vary in length.

* The ratio of the lengths of XXX and YYY

 can vary.

* We don’t have an accurate alignment (correspondence of the elements) of

 XXX and Y.Y.Y.

The CTC algorithm overcomes these challenges. For a given XXX

it gives us an output distribution over all possible YYY’s. We

can use this distribution either to *infer* a likely output or to assess

the *probability* of a given output.

Not all ways of computing the loss function and performing inference are

tractable. We’ll require that CTC do both of these efficiently.

**Loss Function:** For a given input, we’d like to train our

model to maximize the probability it assigns to the right answer. To do this,

we’ll need to efficiently compute the conditional probability

p(Y∣X).p(Y \mid X).p(Y∣X). The function p(Y∣X)p(Y \mid X)p(Y∣X) should

also be differentiable, so we can use gradient descent.

**Inference:** Naturally, after we’ve trained the model, we

want to use it to infer a likely YYY given an X.X.X.

This means solving

Y∗=argmaxYp(Y∣X).

Y∗=Yargmaxp(Y∣X).

Ideally Y∗Y^\*Y∗ can be found efficiently. With CTC we’ll settle

for an approximate solution that’s not too expensive to find.

The Algorithm

-------------

The CTC algorithm can assign a probability for any YYY

given an X.X.X. The key to computing this probability is how CTC

thinks about alignments between inputs and outputs. We’ll start by looking at

these alignments and then show how to use them to compute the loss function and

perform inference.

### Alignment

The CTC algorithm is *alignment-free* — it doesn’t require an

alignment between the input and the output. However, to get the probability of

an output given an input, CTC works by summing over the probability of all

possible alignments between the two. We need to understand what these

alignments are in order to understand how the loss function is ultimately

calculated.

To motivate the specific form of the CTC alignments, first consider a naive

approach. Let’s use an example. Assume the input has length six and Y=Y

=Y= [c, a, t]. One way to align XXX and YYY

is to assign an output character to each input step and collapse repeats.

![](Sequence%20Modeling%20with%20CTC_files/naive_alignment.svg)

This approach has two problems.

* Often, it doesn’t make sense to force every input step to align to

 some output. In speech recognition, for example, the input can have stretches

 of silence with no corresponding output.

* We have no way to produce outputs with multiple characters in a row.

 Consider the alignment [h, h, e, l, l, l, o]. Collapsing repeats will

 produce “helo” instead of “hello”.

To get around these problems, CTC introduces a new token to the set of

allowed outputs. This new token is sometimes called the *blank* token. We’ll

refer to it here as ϵ.\epsilon.ϵ. The

ϵ\epsilonϵ token doesn’t correspond to anything and is simply

removed from the output.

The alignments allowed by CTC are the same length as the input. We allow any

alignment which maps to YYY after merging repeats and removing

ϵ\epsilonϵ tokens:

![](Sequence%20Modeling%20with%20CTC_files/ctc_alignment_steps.svg)

If YYY has two of the same character in a row, then a valid

alignment must have an ϵ\epsilonϵ between them. With this rule

in place, we can differentiate between alignments which collapse to “hello” and

those which collapse to “helo”.

Let’s go back to the output [c, a, t] with an input of length six. Here are

a few more examples of valid and invalid alignments.

![](Sequence%20Modeling%20with%20CTC_files/valid_invalid_alignments.svg)

The CTC alignments have a few notable properties. First, the allowed

alignments between XXX and YYY are monotonic.

If we advance to the next input, we can keep the corresponding output the

same or advance to the next one. A second property is that the alignment of

XXX to YYY is many-to-one. One or more input

elements can align to a single output element but not vice-versa. This implies

a third property: the length of YYY cannot be greater than the

length of X.X.X.

### Loss Function

The CTC alignments give us a natural way to go from probabilities at each

time-step to the probability of an output sequence.

![](Sequence%20Modeling%20with%20CTC_files/full_collapse_from_audio.svg)

To be precise, the CTC objective

for a single (X,Y)(X, Y)(X,Y) pair is:

p(Y∣X)=p(Y \mid X) \;\; =p(Y∣X)=

∑A∈AX,Y\sum\_{A \in \mathcal{A}\_{X,Y}}A∈AX,Y∑

∏t=1Tpt(at∣X)\prod\_{t=1}^T \; p\_t(a\_t \mid X)t=1∏Tpt(at∣X)

 The CTC conditional **probability**

**marginalizes** over the set of valid alignments

 computing the **probability** for a single alignment step-by-step.

Models trained with CTC typically use a recurrent neural network (RNN) to

estimate the per time-step probabilities, pt(at∣X).p\_t(a\_t \mid X).pt(at∣X).

An RNN usually works well since it accounts for context in the input, but we’re

free to use any learning algorithm which produces a distribution over output

classes given a fixed-size slice of the input.

If we aren’t careful, the CTC loss can be very expensive to compute. We

could try the straightforward approach and compute the score for each alignment

summing them all up as we go. The problem is there can be a massive number of

alignments.

 For a YYY of length UUU without any repeat

 characters and an XXX of length TTT the size

 of the set is (T+UT−U).{T + U \choose T - U}.(T−UT+U). For T=100T=100T=100 and

 U=50U=50U=50 this number is almost 1040.10^{40}.1040.

For most problems this would be too slow.

Thankfully, we can compute the loss much faster with a dynamic programming

algorithm. The key insight is that if two alignments have reached the same

output at the same step, then we can merge them.

![](Sequence%20Modeling%20with%20CTC_files/all_alignments.svg)

 Summing over all alignments can be very expensive.

![](Sequence%20Modeling%20with%20CTC_files/merged_alignments.svg)

 Dynamic programming merges alignments, so it’s much faster.

Since we can have an ϵ\epsilonϵ before or after any token in

YYY, it’s easier to describe the algorithm

using a sequence which includes them. We’ll work with the sequence

Z=[ϵ, y1, ϵ, y2, …, ϵ, yU, ϵ]

Z=[ϵ, y1, ϵ, y2, …, ϵ, yU, ϵ]

which is YYY with an ϵ\epsilonϵ at

the beginning, end, and between every character.

Let’s let α\alphaα be the score of the merged alignments at a

given node. More precisely, αs,t\alpha\_{s, t}αs,t is the CTC score of

the subsequence Z1:sZ\_{1:s}Z1:s after ttt input steps.

As we’ll see, we’ll compute the final CTC score, P(Y∣X)P(Y \mid X)P(Y∣X),

from the α\alphaα’s at the last time-step. As long as we know

the values of α\alphaα at the previous time-step, we can compute

αs,t.\alpha\_{s, t}.αs,t. There are two cases.

**Case 1:**

![](Sequence%20Modeling%20with%20CTC_files/cost_no_skip.svg)

In this case, we can’t jump over zs−1z\_{s-1}zs−1, the previous

token in Z.Z.Z. The first reason is that the previous token can

be an element of YYY, and we can’t skip elements of

Y.Y.Y. Since every element of YYY in

ZZZ is followed by an ϵ\epsilonϵ, we can

identify this when zs=ϵ. z\_{s} = \epsilon.zs=ϵ. The second reason is

that we must have an ϵ\epsilonϵ between repeat characters in

Y.Y.Y. We can identify this when

zs=zs−2.z\_s = z\_{s-2}.zs=zs−2.

To ensure we don’t skip zs−1z\_{s-1}zs−1, we can either be there

at the previous time-step or have already passed through at some earlier

time-step. As a result there are two positions we can transition from.

αs,t=\alpha\_{s, t} \; =αs,t=

 The CTC probability of the two valid subsequences after

 t−1t-1t−1 input steps.

pt(zs∣X)p\_t(z\_{s} \mid X)pt(zs∣X)

 The probability of the current character at input step t.t.t.

![](Sequence%20Modeling%20with%20CTC_files/cost_regular.svg)

**Case 2:**

In the second case, we’re allowed to skip the previous token in

Z.Z.Z. We have this case whenever zs−1z\_{s-1}zs−1 is

an ϵ\epsilonϵ between unique characters. As a result there are

three positions we could have come from at the previous step.

αs,t=\alpha\_{s, t} \; =αs,t=

 The CTC probability of the three valid subsequences after

 t−1t-1t−1 input steps.

pt(zs∣X)p\_t(z\_{s} \mid X)pt(zs∣X)

 The probability of the current character at input step t.t.t.

Below is an example of the computation performed by the dynamic programming

algorithm. Every valid alignment has a path in this graph.

 output

Y=Y =Y= [a, b]

 input, XXX

![](Sequence%20Modeling%20with%20CTC_files/ctc_cost.svg)

 Node (s,t)(s, t)(s,t) in the diagram represents

 αs,t\alpha\_{s, t}αs,t – the CTC score of

 the subsequence Z1:sZ\_{1:s}Z1:s after

 ttt input steps.

There are two valid starting nodes and two valid final nodes since the

ϵ\epsilonϵ at the beginning and end of the sequence is

optional. The complete probability is the sum of the two final nodes.

Now that we can efficiently compute the loss function, the next step is to

compute a gradient and train the model. The CTC loss function is differentiable

with respect to the per time-step output probabilities since it’s just sums and

products of them. Given this, we can analytically compute the gradient of the

loss function with respect to the (unnormalized) output probabilities and from

there run backpropagation as usual.

For a training set D\mathcal{D}D, the model’s parameters

are tuned to minimize the negative log-likelihood

∑(X,Y)∈D−logp(Y∣X)

\sum\_{(X, Y) \in \mathcal{D}} -\log\; p(Y \mid X)

(X,Y)∈D∑−logp(Y∣X)

instead of maximizing the likelihood directly.

### Inference

After we’ve trained the model, we’d like to use it to find a likely output

for a given input. More precisely, we need to solve:

Y∗=argmaxYp(Y∣X)

Y∗=Yargmaxp(Y∣X)

One heuristic is to take the most likely output at each time-step. This

gives us the alignment with the highest probability:

A∗=argmaxA∏t=1Tpt(at∣X)

A∗=Aargmaxt=1∏Tpt(at∣X)

We can then collapse repeats and remove ϵ\epsilonϵ tokens to

get Y.Y.Y.

For many applications this heuristic works well, especially when most of the

probability mass is alloted to a single alignment. However, this approach can

sometimes miss easy to find outputs with much higher probability. The problem

is, it doesn’t take into account the fact that a single output can have many

alignments.

Here’s an example. Assume the alignments [a, a, ϵ\epsilonϵ]

and [a, a, a] individually have lower probability than [b, b, b]. But

the sum of their probabilities is actually greater than that of [b, b, b]. The

naive heuristic will incorrectly propose Y=Y =Y= [b] as

the most likely hypothesis. It should have chosen Y=Y =Y= [a].

To fix this, the algorithm needs to account for the fact that [a, a, a] and [a,

a, ϵ\epsilonϵ] collapse to the same output.

We can use a modified beam search to solve this. Given limited

computation, the modified beam search won’t necessarily find the

most likely Y.Y.Y. It does, at least, have

the nice property that we can trade-off more computation

(a larger beam-size) for an asymptotically better solution.

A regular beam search computes a new set of hypotheses at each input step.

The new set of hypotheses is generated from the previous set by extending each

hypothesis with all possible output characters and keeping only the top

candidates.

![](Sequence%20Modeling%20with%20CTC_files/beam_search.svg)

 A standard beam search algorithm with an alphabet of

 {ϵ,a,b}\{\epsilon, a, b\}{ϵ,a,b} and a beam size

 of three.

We can modify the vanilla beam search to handle multiple alignments mapping to

the same output. In this case instead of keeping a list of alignments in the

beam, we store the output prefixes after collapsing repeats and removing

ϵ\epsilonϵ characters. At each step of the search we accumulate

scores for a given prefix based on all the alignments which map to it.

 The CTC beam search algorithm with an output alphabet

 {ϵ,a,b}\{\epsilon, a, b\}{ϵ,a,b}

 and a beam size of three.

A proposed extension can map to two output prefixes if the character is a

repeat. This is shown at T=3T=3T=3 in the figure above

where ‘a’ is proposed as an extension to the prefix [a]. Both [a] and [a, a] are

valid outputs for this proposed extension.

When we extend [a] to produce [a,a], we only want include the part of the

previous score for alignments which end in ϵ.\epsilon.ϵ. Remember, the

ϵ\epsilonϵ is required between repeat characters. Similarly,

when we don’t extend the prefix and produce [a], we should only include the part

of the previous score for alignments which don’t end in ϵ.\epsilon.ϵ.

Given this, we have to keep track of two probabilities for each prefix

in the beam. The probability of all alignments which end in

ϵ\epsilonϵ and the probability of all alignments which don’t

end in ϵ.\epsilon.ϵ. When we rank the hypotheses at

each step before pruning the beam, we’ll use their combined scores.

The implementation of this algorithm doesn’t require much code, but it is

dense and tricky to get right. Checkout this

[gist](https://gist.github.com/awni/56369a90d03953e370f3964c826ed4b0)

for an example implementation in Python.

In some problems, such as speech recognition, incorporating a language model

over the outputs significantly improves accuracy. We can include the language

model as a factor in the inference problem.

Y∗=argmaxYY^\* \enspace = \enspace {\mathop{\text{argmax}}\limits\_{Y}}

 Y∗=Yargmax

p(Y∣X)⋅p(Y \mid X) \quad \cdotp(Y∣X)⋅

 The CTC conditional probability.

p(Y)α⋅p(Y)^\alpha \quad \cdotp(Y)α⋅

 The language model probability.

L(Y)βL(Y)^\betaL(Y)β

 The “word” insertion bonus.

The function L(Y)L(Y)L(Y) computes the length of

YYY in terms of the language model tokens and acts as a word

insertion bonus. With a word-based language model L(Y)L(Y)L(Y)

counts the number of words in Y.Y.Y. If we use a character-based

language model then L(Y)L(Y)L(Y) counts the number of characters

in Y.Y.Y. The language model scores are only included when a

prefix is extended by a character (or word) and not at every step of the

algorithm. This causes the search to favor shorter prefixes, as measured by

L(Y)L(Y)L(Y), since they don’t include as many language model

updates. The word insertion bonus helps with this. The parameters

α\alphaα and β\betaβ are usually set by

cross-validation.

The language model scores and word insertion term can be included in the

beam search. Whenever we propose to extend a prefix by a character, we can

include the language model score for the new character given the prefix so

far.

Properties of CTC

-----------------

We mentioned a few important properties of CTC so far. Here we’ll go

into more depth on what these properties are and what trade-offs they offer.

### Conditional Independence

One of the most commonly cited shortcomings of CTC is the conditional

independence assumption it makes.

![](Sequence%20Modeling%20with%20CTC_files/conditional_independence.svg)

 Graphical model for CTC.

The model assumes that every output is conditionally independent of

the other outputs given the input. This is a bad assumption for many

sequence to sequence problems.

Say we had an audio clip of someone saying “triple A”.

 Another valid transcription could

be “AAA”. If the first letter of the predicted transcription is ‘A’, then

the next letter should be ‘A’ with high probability and ‘r’ with low

probability. The conditional independence assumption does not allow for this.

![](Sequence%20Modeling%20with%20CTC_files/triple_a.svg)

 If we predict an ‘A’ as the first letter then the suffix ‘AA’ should get much

 more probability than ‘riple A’. If we predict ‘t’ first, the opposite

 should be true.

In fact speech recognizers using CTC don’t learn a language model over the

output nearly as well as models which are conditionally dependent.

 However, a separate language model can

be included and usually gives a good boost to accuracy.

The conditional independence assumption made by CTC isn’t always a bad

thing. Baking in strong beliefs over output interactions makes the model less

adaptable to new or altered domains. For example, we might want to use a speech

recognizer trained on phone conversations between friends to transcribe

customer support calls. The language in the two domains can be quite different

even if the acoustic model is similar. With a CTC acoustic model, we can easily

swap in a new language model as we change domains.

### Alignment Properties

The CTC algorithm is *alignment-free*. The objective function

marginalizes over all alignments. While CTC does make strong assumptions about

the form of alignments between XXX and YYY, the

model is agnostic as to how probability is distributed amongst them. In some

problems CTC ends up allocating most of the probability to a single alignment.

However, this isn’t guaranteed.

We could force the model to choose a single

alignment by replacing the sum with a max in the objective function,

p(Y∣X)=maxA∈AX,Y∏t=1Tp(at∣X).

p(Y∣X)=A∈AX,Ymaxt=1∏Tp(at∣X).

As mentioned before, CTC only allows *monotonic* alignments. In

problems such as speech recognition this may be a valid assumption. For other

problems like machine translation where a future word in a target sentence

can align to an earlier part of the source sentence, this assumption is a

deal-breaker.

Another important property of CTC alignments is that they are

*many-to-one*. Multiple inputs can align to at most one output. In some

cases this may not be desirable. We might want to enforce a strict one-to-one

correspondence between elements of XXX and

Y.Y.Y. Alternatively, we may want to allow multiple output

elements to align to a single input element. For example, the characters

“th” might align to a single input step of audio. A character based CTC model

would not allow that.

The many-to-one property implies that the output can’t have more time-steps

than the input.

 If YYY has rrr consecutive

 repeats, then the length of YYY must be less than

 the length of XXX by 2r−1.2r - 1.2r−1.

This is usually not a problem for speech and handwriting recognition since the

input is much longer than the output. However, for other problems where

YYY is often longer than XXX, CTC just won’t

work.

CTC in Context

--------------

In this section we’ll discuss how CTC relates to other commonly used

algorithms for sequence modeling.

### HMMs

At a first glance, a Hidden Markov Model (HMM) seems quite different from

CTC. But, the two algorithms are actually quite similar. Understanding the

relationship between them will help us understand what advantages CTC has over

HMM sequence models and give us insight into how CTC could be changed for

various use cases.

Let’s use the same notation as before,

XXX is the input sequence and YYY

is the output sequence with lengths TTT and

UUU respectively. We’re interested in learning

p(Y∣X).p(Y \mid X).p(Y∣X). One way to simplify the problem is to apply

Bayes’ Rule:

p(Y∣X)∝p(X∣Y)p(Y).

p(Y \mid X) \; \propto \; p(X \mid Y) \; p(Y).

p(Y∣X)∝p(X∣Y)p(Y).

The p(Y)p(Y)p(Y) term can be any language model, so let’s focus on

p(X∣Y).p(X \mid Y).p(X∣Y). Like before we’ll let

A\mathcal{A}A be a set of allowed alignments between

XXX and Y.Y.Y. Members of

A\mathcal{A}A have length T.T.T.

Let’s otherwise leave A\mathcal{A}A unspecified for now. We’ll

come back to it later. We can marginalize over alignments to get

p(X∣Y)=∑A∈Ap(X,A∣Y).

p(X \mid Y)\; = \; \sum\_{A \in \mathcal{A}} \; p(X, A \mid Y).

p(X∣Y)=A∈A∑p(X,A∣Y).

To simplify notation, let’s remove the conditioning on YYY, it

will be present in every p(⋅).p(\cdot).p(⋅). With two assumptions we can

write down the standard HMM.

p(X)=p(X) \quad =p(X)=

 The probability of the input

∑A∈A∏t=1T\sum\_{A \in \mathcal{A}} \; \prod\_{t=1}^T∑A∈A∏t=1T

 Marginalizes over alignments

p(xt∣at)⋅p(x\_t \mid a\_t) \quad \cdotp(xt∣at)⋅

 The emission probability

p(at∣at−1)p(a\_t \mid a\_{t-1})p(at∣at−1)

 The transition probability

The first assumption is the usual Markov property. The state

ata\_tat is conditionally independent of all historic states given

the previous state at−1.a\_{t-1}.at−1. The second is that the observation

xtx\_txt is conditionally independent of everything given the

current state at.a\_t.at.

![](Sequence%20Modeling%20with%20CTC_files/hmm.svg)

 The graphical model for an HMM.

Now we can take just a few steps to transform the HMM into CTC and see how

the two models relate. First, let’s assume that the transition probabilities

p(at∣at−1)p(a\_t \mid a\_{t-1})p(at∣at−1) are uniform. This gives

p(X)∝∑A∈A∏t=1Tp(xt∣at).

p(X)∝A∈A∑t=1∏Tp(xt∣at).

There are only two differences from this equation and the CTC loss function.

The first is that we are learning a model of XXX given

YYY as opposed to YYY given X.X.X.

The second is how the set A\mathcal{A}A is produced. Let’s deal

with each in turn.

To do this, we apply Bayes’ rule and rewrite the model as

p(X)∝∑A∈A∏t=1Tp(at∣xt)p(xt)p(at)

p(X)∝A∈A∑t=1∏Tp(at)p(at∣xt)p(xt)

∝∑A∈A∏t=1Tp(at∣xt)p(at).

∝A∈A∑t=1∏Tp(at)p(at∣xt).

If we assume a uniform prior over the states aaa and condition on all of

XXX instead of a single element at a time, we arrive at

p(X)∝∑A∈A∏t=1Tp(at∣X).

p(X)∝A∈A∑t=1∏Tp(at∣X).

The above equation is essentially the CTC loss function, assuming the set

A\mathcal{A}A is the same. In fact, the HMM framework does not specify what

A\mathcal{A}A should consist of. This part of the model can be designed on a

per-problem basis. In many cases the model doesn’t condition on YYY and the

set A\mathcal{A}A consists of all possible length TTT sequences from the

output alphabet. In this case, the HMM can be drawn as an *ergodic* state

transition diagram in which every state connects to every other state. The

figure below shows this model with the alphabet or set of unique hidden states

as {a,b,c}.\{a, b, c\}.{a,b,c}.

In our case the transitions allowed by the model are strongly related to

Y.Y.Y. We want the HMM to reflect this. One possible model could

be a simple linear state transition diagram. The figure below shows this with

the same alphabet as before and Y=Y =Y= [a, b]. Another commonly

used model is the *Bakis* or left-right HMM. In this model any

transition which proceeds from the left to the right is allowed.

![](Sequence%20Modeling%20with%20CTC_files/ergodic_hmm.svg)

**Ergodic HMM:** Any node can be either a starting or

 final state.

![](Sequence%20Modeling%20with%20CTC_files/linear_hmm.svg)

**Linear HMM:** Start on the left, end on the right.

![](Sequence%20Modeling%20with%20CTC_files/ctc_hmm.svg)

**CTC HMM:** The first two nodes are the starting

 states and the last two nodes are the final states.

In CTC we augment the alphabet with ϵ\epsilonϵ and the HMM model allows a

subset of the left-right transitions. The CTC HMM has two start

states and two accepting states.

One possible source of confusion is that the HMM model differs for any unique

Y.Y.Y. This is in fact standard in applications such as speech recognition. The

state diagram changes based on the output Y.Y.Y. However, the functions which

estimate the observation and transition probabilities are shared.

Let’s discuss how CTC improves on the original HMM model. First, we can think

of the CTC state diagram as a special case HMM which works well for many

problems of interest. Incorporating the blank as a hidden state in the HMM

allows us to use the alphabet of YYY as the other hidden states. This model

also gives a set of allowed alignments which may be a good prior for some

problems.

Perhaps most importantly, CTC is discriminative. It models p(Y∣X)p(Y \mid

 X)p(Y∣X) directly, an idea that’s been important in the past with other

discriminative improvements to HMMs.

Discriminative training let’s us apply powerful learning algorithms like the

RNN directly towards solving the problem we care about.

### Encoder-Decoder Models

The encoder-decoder is perhaps the most commonly used framework for sequence

modeling with neural networks. These models have an encoder

and a decoder. The encoder maps the input sequence XXX into a

hidden representation. The decoder consumes the hidden representation and

produces a distribution over the outputs. We can write this as

H=encode(X)p(Y∣X)=decode(H).

\begin{aligned}

H\enspace &= \enspace\textsf{encode}(X) \\[.5em]

p(Y \mid X)\enspace &= \enspace \textsf{decode}(H).

\end{aligned}

Hp(Y∣X)=encode(X)=decode(H).

The encode(⋅)\textsf{encode}(\cdot)encode(⋅) and

decode(⋅)\textsf{decode}(\cdot)decode(⋅) functions are typically RNNs. The

decoder can optionally be equipped with an attention mechanism. The hidden

state sequence HHH has the same number of time-steps as the

input, T.T.T. Sometimes the encoder subsamples the input. If the

encoder subsamples the input by a factor sss then

HHH will have T/sT/sT/s time-steps.

We can interpret CTC in the encoder-decoder framework. This is helpful to

understand the developments in encoder-decoder models that are applicable to

CTC and to develop a common language for the properties of these

models.

**Encoder:** The encoder of a CTC model can be just about any

encoder we find in commonly used encoder-decoder models. For example the

encoder could be a multi-layer bidirectional RNN or a convolutional network.

There is a constraint on the CTC encoder that doesn’t apply to the others. The

input length cannot be sub-sampled so much that T/sT/sT/s

is less than the length of the output.

**Decoder:** We can view the decoder of a CTC model as a simple

linear transformation followed by a softmax normalization. This layer should

project all TTT steps of the encoder output

HHH into the dimensionality of the output alphabet.

We mentioned earlier that CTC makes a conditional independence assumption over

the characters in the output sequence. This is one of the big advantages that

other encoder-decoder models have over CTC — they can model the

dependence over the outputs. However in practice, CTC is still more commonly

used in tasks like speech recognition as we can partially overcome the

conditional independence assumption by including an external language model.

Practitioner’s Guide

--------------------

So far we’ve mostly developed a conceptual understanding of CTC. Here we’ll go

through a few implementation tips for practitioners.

**Software:** Even with a solid understanding of CTC, the

implementation is difficult. The algorithm has several edge cases and a fast

implementation should be written in a lower-level programming language.

Open-source software tools make it much easier to get started:

* Baidu Research has open-sourced

 [warp-ctc](https://github.com/baidu-research/warp-ctc). The

 package is written in C++ and CUDA. The CTC loss function runs on either

 the CPU or the GPU. Bindings are available for Torch, TensorFlow and

 [PyTorch](https://github.com/awni/warp-ctc).

* TensorFlow has built in

 [CTC loss](https://www.tensorflow.org/api_docs/python/tf/nn/ctc_loss)

 functions for the CPU.

* Nvidia also provides a GPU implementation of CTC in

 [cuDNN](https://developer.nvidia.com/cudnn) versions 7 and up.

**Numerical Stability:** Computing the CTC loss naively is

numerically unstable. One method to avoid this is to normalize the

α\alphaα’s at each time-step. The original publication

has more detail on this including the adjustments to the gradient.

 In practice this works well enough

for medium length sequences but can still underflow for long sequences.

A better solution is to compute the loss function in log-space with the

log-sum-exp trick.

When computing the sum of two probabilities in log space use the identity

log(ea+eb)=max{a,b}+log(1+e−∣a−b∣)

\log(e^a + e^b) = \max\{a, b\} + \log(1 + e^{-|a-b|})

log(ea+eb)=max{a,b}+log(1+e−∣a−b∣)

Most programming languages have a stable function to compute

log(1+x)\log(1 + x)log(1+x) when

xxx is close to zero.

Inference should also be done in log-space using the log-sum-exp trick.

**Beam Search:** There are a couple of good tips to know about

when implementing and using the CTC beam search.

The correctness of the beam search can be tested as follows.

1. Run the beam search algorithm on an arbitrary input.

2. Save the inferred output Y¯\bar{Y}Y¯ and the corresponding

 score c¯.\bar{c}.c¯.

3. Compute the actual CTC score ccc for

 Y¯.\bar{Y}.Y¯.

4. Check that c¯≈c\bar{c} \approx cc¯≈c with the former being no

 greater than the latter. As the beam size increases the inferred output

 Y¯\bar{Y}Y¯ may change, but the two numbers should grow

 closer.

A common question when using a beam search decoder is the size of the beam

to use. There is a trade-off between accuracy and runtime. We can check if the

beam size is in a good range. To do this first compute the CTC score for the

inferred output ci.c\_i.ci. Then compute the CTC score for the ground

truth output cg.c\_g.cg. If the two outputs are not the same, we

should have cg<ci.c\_g \lt c\_i.cg<ci. If ci<<cgc\_i << c\_gci<<cg then

the ground truth output actually has a higher probability under the model and

the beam search failed to find it. In this case a large increase to the beam

size may be warranted.

Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition

Exploring Neural Transducers for End-to-End Speech Recognition

Large scale discriminative training of hidden Markov models for speech recognition

Connectionist Temporal Classification : Labelling Unsegmented Sequence Data with Recurrent Neural Networks

Phone recognition on the TIMIT database

Supervised Sequence Labelling with Recurrent Neural Networks

Towards End-To-End Speech Recognition with Recurrent Neural Networks

Deep Speech: Scaling up end-to-end speech recognition

A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks

LipNet: End-to-End Sentence-level Lipreading

Connectionist Temporal Modeling for Weakly Supervised Action Labeling

An application of recurrent neural networks to discriminative keyword spotting

An End-to-End Architecture for Keyword Spotting and Voice Activity Detection

Sequence Transduction with Recurrent Neural Networks

Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling

Sequence Modeling via Segmentations

Segmental Recurrent Neural Networks

Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition

Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Sequence to Sequence Learning with Neural Networks

Attention and Augmented Recurrent Neural Networks