Self-classifying MNIST Digits

Ettore Randazzo
Alexander Mordvintsev
Eyvind Niklasson
Michael Levin
Sam Greydanus

### Contents

[Model](#model)

[Experiments](#experiment-1)

* [Self-classify, persist & mutate](#experiment-1)

* [Stabilizing classification](#experiment-2)

[Robustness](#robustness)

[Related Work](#related-work)

[Discussion](#discussion)

![](Self-classifying%20MNIST%20Digits_files/multiple-pages.svg)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Growing Neural Cellular Automata](https://distill.pub/2020/growing-ca/)

[Self-Organising Textures](https://distill.pub/selforg/2021/textures/)

Growing Neural Cellular Automata

 demonstrated how simple cellular automata (CAs) can learn to

self-organise into complex shapes while being resistant to

perturbations. Such a computational model approximates a solution to an

open question in biology, namely, how do cells cooperate to create a

complex multicellular anatomy and work to regenerate it upon damage? The

 model parameterizing the cells’ rules is parameter-efficient,

end-to-end differentiable, and illustrates a new approach to modeling

the regulation of anatomical homeostasis. In this work, we use a version

 of this model to show how CAs can be applied to a common task in

Our question is closely related to another unsolved problem in

developmental and regenerative biology: how cell groups decide whether

an organ or tissue pattern is correct, or whether current anatomy needs

to be remodeled (anatomical surveillance and repair toward a specific

target morphology). For example, when scientists surgically transplanted

 a salamander tail to its flank, it slowly remodeled into a limb - the

organ that belongs at this location .

 Similarly, tadpoles with craniofacial organs in the wrong positions

usually become normal frogs because they remodel their faces, placing

the eye, mouth, nostrils, etc. in their correct locations. Cell groups

move around and stop when the correct frog-specific anatomical

configuration has been achieved .

 All of these examples illustrate the ability of biological systems to

determine their current anatomical structure and decide whether it

matches a species-specific target morphology .

 Despite the recent progress in molecular biology of genes necessary for

 this process, there is still a fundamental knowledge gap concerning the

 algorithms sufficient for cell collectives to measure and classify

their own large-scale morphology. More broadly, it is important to

create computational models of swarm intelligence that explicitly define

 and distinguish the dynamics of the basal cognition of single cells

versus cell collectives .

### The self-classifying MNIST task

Suppose a population of agents is arranged on a grid. They do not

know where they are in the grid and they can only communicate with their

 immediate neighbors. They can also observe whether a neighbor is

missing. Now suppose these agents are arranged to form the shape of a

digit. Given that all the agents operate under the same rules, can they

form a communication protocol such that, after a number of iterations of

 if some agents were to be removed and added to form a new digit from a

preexisting one, would they be able to know which the new digit is?

Because digits are not rotationally invariant (i.e. 6 is a rotation

of 9), we presume the agents must be made aware of their orientation

with respect to the grid. Therefore, while they do not know where they

are, they do know where up, down, left and right are. The biological

analogy here is a situation where the remodeling structures exist in the

 context of a larger body and a set of morphogen gradients or tissue

polarity that indicate directional information with respect to the three

 major body axes. Given these preliminaries, we introduce the

self-classifying MNIST task.

Each sample of the MNIST dataset

consists of a 28x28 image with a single monochrome channel that is

classically displayed in greyscale. The label is an integer in [0,9][0,9][0,9].

Our goal is for all cells that make up the digit to correctly output

the label of the digit. To convey this structural information to the

cells, we make a distinction between alive and dead cells by rescaling

the values of the image to [0, 1]. Then we treat a cell as alive if its

value in the MNIST sample is larger than 0.1. The intuition here is that

 we are placing living cells in a cookie cutter and asking them to

identify the global shape of the cookie cutter. We visualize the label

output by assigning a color to each cell, as you can see above. We use

the same mapping between colors and labels throughout the article.

Please note that there is a slider in the interactive demo controls

which you can use to adjust the color palette if you have trouble

differentiating between the default colors.

Model

-----

### Target labels

The work in Growing CA used RGB images as targets, and optimized the

first three state channels to approximate those images. For our

experiments, we treat the last ten channels of our cells as a

pseudo-distribution over each possible label (digit). During inference,

we simply pick the label corresponding to the channel with the highest

output value.

### Alive cells and cell states

In Growing CA we assigned a cell’s state to be “dead” or “alive”

based on the strength of its alpha channel and the activity of its

neighbors. This is similar to the rules of Conway’s Game of Life .

 In the Growing CA model, “alive” cells are cells which update their

state and dead cells are “frozen” and do not undergo updates. In

contrast to biological life, what we call “dead” cells aren’t dead in

the sense of being non-existent or decayed, but rather frozen: they are

visible to their neighbors and maintain their state throughout the

simulation. In this work, meanwhile, we use input pixel values to

determine whether cells are alive or dead and perform computations with

 It is important to note that the values of MNIST pixels are exposed to

the cell update rule as an immutable channel of the cell state. In other

 words, we make cells aware of their own pixel intensities as well as

those of their neighbors. Given 19 mutable cell state channels (nine

general purpose state channels for communication and ten output state

channels for digit classification) and an immutable pixel channel, each

cell perceives 19 + 1 state channels and only outputs state updates for

the 19 mutable state channels.

**A note on digit topology.**Keen

readers may notice that our model requires each digit to be a single

connected component in order for classification to be possible, since

any disconnected components will be unable to propagate information

between themselves. We made this design decision in order to stay true

to our core biological analogy, which involves a group of cells that is

trying to identify its global shape. Even though the vast majority of

samples from MNIST are fully connected, some aren’t. We do not expect

our models to classify non-connected minor components correctly, but we

do not remove them This choice complicates

 comparison between the MNIST train/test accuracies of neural network

classifiers vs. CAs. However, such a comparison is not in scope of this

article..

### Perception

**A note on model size.** Like the Growing CA model, our

 MNIST CA is small by the standards of deep learning - it has less than

25k parameters. Since this work aims to demonstrate a novel approach to

classification, we do not attempt to maximise the validation accuracy of

 the model by increasing the number of parameters or any other tuning.

We suspect that, as with other deep neural network models, one would

observe a positive correlation between accuracy and model size.

Experiment 1: Self-classify, persist and mutate

-----------------------------------------------

In our first experiment, we use the same training paradigm as was

discussed in Growing CA. We train with a pool of initial samples to

allow the model to learn to persist and then perturb the converged

states. However, our perturbation is different. Previously, we destroyed

 the states of cells at random in order to make the CAs resistant to

destructive perturbations (analogous to traumatic tissue loss). In this

context, perturbation has a slightly different role to play. Here we aim

 to build a CA model that not only has regenerative properties, but also

Biologically, this corresponds to a teratogenic influence during

development, or alternatively, a case of an incorrect or incomplete

remodeling event such as metamorphosis or rescaling. The distinction

between training our model from scratch and training it to accommodate

perturbations is subtle but important. An important feature of life is

the ability to react adaptively to external perturbations that are not

accounted for in the normal developmental sequence of events. If our

virtual cells simply learned to recognize a digit and then entered some

dormant state and did not react to any further changes, we would be

missing this key property of living organisms. One could imagine a

trivial solution in the absence of perturbations, where a single wave of

 information is passed from the boundaries of the digit inwards and then

 back out, in such a way that all cells could agree on a correct

classification. By introducing perturbations to new digits, the cells

have to be in constant communication and achieve a “dynamic homeostasis”

 - continually “kept on their toes” in anticipation of new or further

communication from their neighbours.

In our model, we achieve this dynamic homeostasis by randomly

mutating the underlying digit at training time. Starting from a certain

digit and after some time evolution, we sample a new digit, erase all

cell states that are not present in both digits and bring alive the

cells that were not present in the original digit but are present in the

 new digit. This kind of mutation teaches CAs to learn to process new

information and adapt to changing conditions. It also exposes the cells

to training states where all of the cells that remain after a

perturbation are misclassifying the new digit and must recover from this

 catastrophic mutation. This in turn forces our CAs to learn to change

their own classifications to adapt to changing global structure.

We use a pixel-wise (cell-wise) cross entropy loss on the last ten

channels of each pixel, applying it after letting the CA evolve for 20

steps.

 Your browser does not support the video tag.

A first attempt at having the neural CAs classify digits. Each digit is a

 separate evolution of the neural CA, with the visualisations collated.

Halfway through, the underlying digit is swapped for a new one - a

“mutation”.

The video above shows the CA classifying a batch of digits for 200

steps. We then mutate the digits and let the system evolve and classify

for a further 200 steps.

The results look promising overall and we can see how our CAs are

able to recover from mutations. However, astute observers may notice

that often not all cells agree with each other. Often, the majority of

the digit is classified correctly, but some outlier cells are still

convinced they are part of a different digit, often switching back and

forth in an oscillating pattern, causing a flickering effect in the

visualization. This is not ideal, since we would like the population of

cells to reach stable, total, agreement. The next experiment

troubleshoots this undesired behaviour.

Experiment 2: Stabilizing classification

----------------------------------------

 which we define as the mean percentage of cells that have a correct

output. We track this metric both before and after mutation.

Average accuracy across the cells in a digit over time.

In the figure above, we show the mean percentage of correctly

classified pixels in the test set over the course of 400 steps. At step

200, we randomly mutate the digit. Accordingly, we see a brief drop in

accuracy as the cells re-organise and eventually come to agreement on

what the new digit is.

We immediately notice an interesting phenomenon: the cell accuracy

appears to decrease over time after the cells have come to an agreement.

 However, the graph does not necessarily reflect the qualitative issue

of unstable labels that we set out to solve. The slow decay in accuracy

may be a reflection of the lack of total agreement, but doesn’t capture

the stark instability issue.

Average total agreement among cells across the test set in MNIST, over time.

This metric does a better job of capturing the issues we are seeing.

The total agreement starts at zero and then spikes up to roughly 78%,

only to lose more than 10% agreement over the next 100 steps. Again,

behaviour after mutation does not appear to be significantly different.

Our model is not only unstable in the short term, exhibiting flickering,

 but is also unstable over longer timescales. As time goes on, cells are

 becoming less sure of themselves. Let’s inspect the inner states of the

 CA to see why this is happening.

The figure above shows the time evolution of the average magnitude of

 the state values of active cells (solid line), and the average

magnitude of the residual updates (dotted line). Two important things

are happening here: 1) the average magnitude of each cell’s internal

states is growing monotonically on this timescale; 2) the average

magnitude of the residual updates is staying roughly constant. We

theorize that, unlike 1), a successful CA model should stabilize the

magnitude of its internal states once cells have reached an agreement.

In order for this to happen, its residual updates should approach zero

over time, unlike what we observed in 2).

**Using an** **L2L\_2L2** **loss.** One

 problem with cross entropy loss is that it tends to push raw logit

values indefinitely higher. Another problem is that two sets of logits

can have vastly different values but essentially the same prediction

over classes. As such, training the CA with cross-entropy loss neither

requires nor encourages a shared reference range for logit values,

making it difficult for the cells to effectively communicate and

stabilize. Finally, we theorize that large magnitudes in the

classification channels may in turn lead the remaining

(non-classification) state channels to transition to a high magnitude

regime. More specifically, we believe that *cross-entropy loss causes

unbounded growth in classification logits, which prevents residual

updates from approaching zero, which means that neighboring cells

 loss and use one-hot vectors as targets. Intuitively, this solution

should be more stable since the raw state channels for classification

are never pushed out of the range [0,1][0, 1][0,1]

 and a properly classified digit in a cell will have exactly one

classification channel set to 1 and the rest to 0. In summary, an L2L\_2L2

 loss should decrease the magnitude of all the internal state channels

while keeping the classification targets in a reasonable range.

 Here we add noise to the residual updates by sampling from a normal

 Your browser does not support the video tag.

The video above shows a batch of runs with the augmentations in

place. Qualitatively, the result looks much better as there is less

flickering and more total agreement. Let’s check the quantitative

metrics to see if they, too, show improvement.

Comparison of average accuracy and total agreement

 when using cross-entropy and when using L2L\_2L2 loss.

| Model | Top accuracy | Accuracy at 200  | Top agreement | Agreement at 200 |

| --- | --- | --- | --- | --- |

| CE | **96.2 at 80** | **95.3** | 77.9 at 80 | 66.2 |

| L2L\_2L2 | 95.0 at 95 | 94.7 | 85.5 at 175 | 85.2 |

| L2L\_2L2 + Noise | 95.4 at 65 | **95.3** | **88.2 at 190** | **88.1** |

The figure and table above show that cross-entropy achieves the

highest accuracy of all models at roughly 80 steps. However, the

### Internal states

 loss stabilizes the magnitude of the states, and how residual updates

quickly decay to small values as the system nears agreement.

 Your browser does not support the video tag.

Visualisation of internal state channel values during mutations. Note

the accelerated timeline after a few seconds showing the relative

stability of the channel values.

To further validate our results, we can visualize the dynamics of the

 internal states of the final model. For visualization purposes, we have

 as most state values are less than one but a few are much larger. The

states converge to stable configurations quickly and the state channels

exhibit spatial continuity with the neighbouring states. More

specifically, we don’t see any stark discontinuities in state values of

neighbouring pixels. Applying a mutation causes the CA to readapt to the

 new shape and form a new classification in just a few steps, after

which its internal values are stable.

Robustness

----------

Recall that during training we used random digit mutations to ensure

that the resulting CA would be responsive to external changes. This

allowed us to learn a dynamical system of agents which interact to

produce stable behavior at the population level, even when perturbed to

form a different digit from the original. Biologically, this model helps

 us understand the mutation insensitivity of some large-scale anatomical

 control mechanisms. For example, planaria continuously accumulate

mutations over millions of years of somatic inheritance but still always

 regenerate the correct morphology in nature (and exhibit no genetic

strains with new morphologies) .

This robustness to change was critically important to our interactive

 demo, since the cells needed to reclassify drawings as the user changed

 them. For example, when the user converted a six to an eight, the cells

 needed to quickly re-classify themselves to an eight. We encourage the

reader to play with the interactive demo and experience this for

themselves. In this section, we want to showcase a few behaviours we

found interesting.

 Your browser does not support the video tag.

The video above shows how the CA is able to interactively adjust to

our own writing and to change classification when the drawing is

updated.

### Robustness to out-of-distribution shapes

In the field of machine learning, researchers take great interest in

how their models perform on out-of-distribution data. In the

experimental sections of this article, we evaluated our model on the

test set of MNIST. In this section, we go further and examine how the

model reacts to digits drawn by us and not sampled from MNIST at all. We

 vary the shapes of the digits until the model is no longer capable of

classifying them correctly. Every classification model inherently

contains certain inductive biases that render them more or less robust

to generalizing to out-of-distribution data. Our model can be seen as a

recurrent convolutional model and thus we expect it to exhibit some of

the key properties of traditional convolutional models such as

translation invariance. However, we strongly believe that the

self-organising nature of this model introduces a novel inductive bias

which may have interesting properties of its own. Biology offers

examples of “repairing to novel configurations”: 2-headed planaria, once

 created, regenerate to this new configuration which was not present in

the evolutionary “training set” .

 Your browser does not support the video tag.

Demonstration of some of the failure cases of the CA.

Above, we can see that our CA fails to classify some variants of 1

and 9. This is likely because MNIST training data is not sufficiently

representative of all writing styles. We hypothesize that more varied

and extensive datasets would improve performance. The model often

oscillates between two attractors (of competing digit labels) in these

situations. This is interesting because this behavior could not arise

from static classifiers such as traditional convolutional neural

networks.

 Your browser does not support the video tag.

By construction, our CA is translation invariant. But perhaps

surprisingly, we noticed that our model is also scale-invariant for

out-of-distribution digit sizes up to a certain point. Alas, it does not

 generalize well enough to classify digits of arbitrary lengths and

widths.

 Your browser does not support the video tag.

Demonstration of the behaviour of the model with chimeric configurations.

It is also interesting to see how our CA classifies “chimeric

digits”, which are shapes composed of multiple digits. First, when

creating a 3-5 chimera, the classification of 3 appears to dominate that

 of the 5. Second, when creating a 8-9 chimera, the CAs reach an

oscillating attractor where sections of the two digits are correctly

classified. Third, when creating a 6-9 chimera, the CAs converge to an

oscillating attractor but the 6 is misclassified as a 4.

These phenomena are important in biology as scientists begin to develop

predictive models for the morphogenetic outcome of chimeric cell

collectives. We still do not have a framework for knowing in advance

what anatomical structures will form from a combination of, for example

leg-and-tail blastema cells in an axolotl, heads of planaria housing

stem cells from species with different head shapes, or composite embryos

 consisting of, for example, frog and axolotl blastomeres .

 Likewise, designing information signals that induce the emergence of

desired tissue patterns from a chimeric cellular collective, in vitro or

 in vivo, remains an open problem.

Related Work

------------

This article is follow-up work to Growing Neural Cellular Automata ,

 and it is meant to be read after the latter. In this article, we

purposefully skim over details of the original model and we refer the

**MNIST and CA.** Since CAs are easy to apply to two

dimensional grids, many researchers wondered if they could use them to

somehow classify the MNIST dataset. We are aware of work that combines

 To the best of our knowledge, we are the first to train end-to-end

differentiable Neural CAs for classification purposes and we are the

first to introduce the self-classifying variant of MNIST wherein each

pixel in the digit needs to coordinate locally in order to reach a

global agreement about its label.

Discussion

----------

This article serves as a proof-of-concept for how simple

self-organising systems such as CA can be used for classification when

trained end-to-end through backpropagation.

Our model adapts to writing and erasing and is surprisingly robust to

 certain ranges of digit stretching and brush widths. We hypothesize

that self-organising models with constrained capacity may be inherently

robust and have good generalisation properties. We encourage future work

 to test this hypothesis.

From a biological perspective, our work shows we can teach things to a

 collective of cells that they could not learn individually (by training

 or engineering a single cell). Training cells in unison (while

communicating with each other) allows them to learn more complex

behaviour than any attempt to train them one by one, which has important

 implications for strategies in regenerative medicine. The current focus

 on editing individual cells at the genetic or molecular signaling level

 faces fundamental barriers when trying to induce desired complex,

system-level outcomes (such as regenerating or remodeling whole organs).

 The inverse problem of determining which cell-level rules (e.g.,

genetic information) must be changed to achieve a global outcome is very

 difficult. In contrast and complement to this approach, we show the

first component of a roadmap toward developing effective strategies for

communication with cellular collectives. Future advances in this field

may be able to induce desired outcomes by using stimuli at the system’s

input layer (experience), not hardware rewiring, to re-specify outcomes

at the tissue, organ, or whole-body level .

![](Self-classifying%20MNIST%20Digits_files/multiple-pages.svg)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Growing Neural Cellular Automata](https://distill.pub/2020/growing-ca/)

[Self-Organising Textures](https://distill.pub/selforg/2021/textures/)

Growing Neural Cellular Automata

The transformation of a tail into limb after xenoplastic transplantation

Normalized
 shape and location of perturbed craniofacial structures in the Xenopus
tadpole reveal an innate ability to achieve correct morphology

Top-down models in biology: explanation and control of complex living systems above the molecular level

On Having No Head: Cognition throughout Biological Systems

The biogenic approach to cognition

Gradient-based learning applied to document recognition

MATHEMATICAL GAMES

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Auto-Encoding Variational Bayes

Practical Variational Inference for Neural Networks

Noisy Networks for Exploration

Planarian regeneration as a model of anatomical homeostasis: Recent progress in biophysical and computational approaches

Long-range neural and gap junction protein-mediated cues control polarity during planarian regeneration

Bioelectrical Mechanisms for Programming Growth and Form: Taming Physiological Networks for Soft Body Robotics

Interspecies chimeras

Reservoir Computing Hardware with Cellular Automata

Energy-Efficient Pattern Recognition Hardware With Elementary Cellular Automata

Asynchronous network of cellular automaton-based neurons for efficient implementation of Boltzmann machines

An Approach to Searching for Two-Dimensional Cellular Automata for Recognition of Handwritten Digits

Biologically inspired cellular automata learning and prediction model for handwritten pattern recognition

Pattern Classification with Rejection Using Cellular Automata-Based Filtering

The body electric 2.0: recent advances in developmental bioelectricity for regenerative and synthetic bioengineering

Re-membering
 the body: applications of computational neuroscience to the top-down
control of regeneration of limbs and other complex organs


 What are computers for?

-------------------------

 Historically, different answers to this question – that is,

 different visions of computing – have helped inspire and

 determine the computing systems humanity has ultimately

 built. Consider the early electronic computers. ENIAC, the

 world’s first general-purpose electronic computer, was

 commissioned to compute artillery firing tables for the United

 States Army. Other early computers were also used to solve

 numerical problems, such as simulating nuclear explosions,

 predicting the weather, and planning the motion of rockets. The

 machines operated in a batch mode, using crude input and output

 devices, and without any real-time interaction. It was a vision

 of computers as number-crunching machines, used to speed up

 calculations that would formerly have taken weeks, months, or more

 for a team of humans.

 In the 1950s a different vision of what computers are for began to

 develop. That vision was crystallized in 1962, when Douglas

 Engelbart proposed that computers could be used as a way

 of augmenting human

 intellect. In this view, computers weren’t primarily

 tools for solving number-crunching problems. Rather, they were

 real-time interactive systems, with rich inputs and outputs, that

 humans could work with to support and expand their own

 problem-solving process. This vision of intelligence augmentation

 (IA) deeply influenced many others, including researchers such as

 Alan Kay at Xerox PARC, entrepreneurs such as Steve Jobs at Apple,

 and led to many of the key ideas of modern computing systems. Its

 ideas have also deeply influenced digital art and music, and

 fields such as interaction design, data visualization,

 computational creativity, and human-computer interaction.

 Research on IA has often been in competition with research on

 artificial intelligence (AI): competition for funding, competition

 for the interest of talented researchers. Although there has

 always been overlap between the fields, IA has typically focused

 on building systems which put humans and machines to work

 together, while AI has focused on complete outsourcing of

 intellectual tasks to machines. In particular, problems in AI are

 often framed in terms of matching or surpassing human performance:

 beating humans at chess or Go; learning to recognize speech and

 images or translating language as well as humans; and so on.

 This essay describes a new field, emerging today out of a

 synthesis of AI and IA. For this field, we suggest the

 name *artificial intelligence augmentation* (AIA): the use

 of AI systems to help develop new methods for intelligence

 augmentation. This new field introduces important new fundamental

 questions, questions not associated to either parent field. We

 believe the principles and systems of AIA will be radically

 different to most existing systems.

 Our essay begins with a survey of recent technical work hinting at

 artificial intelligence augmentation, including work

 on *generative interfaces* – that is, interfaces

 which can be used to explore and visualize generative machine

 learning models. Such interfaces develop a kind of cartography of

 generative models, ways for humans to explore and make meaning

 from those models, and to incorporate what those models

 “know” into their creative work.

 Our essay is not just a survey of technical work. We believe now

 is a good time to identify some of the broad, fundamental

 questions at the foundation of this emerging field. To what

 extent are these new tools enabling creativity? Can they be used

 to generate ideas which are truly surprising and new, or are the

 ideas cliches, based on trivial recombinations of existing ideas?

 Can such systems be used to develop fundamental new interface

 primitives? How will those new primitives change and expand the

 way humans think?

 Using generative models to invent meaningful creative operations

------------------------------------------------------------------

 Let’s look at an example where a machine learning model makes a

 new type of interface possible. To understand the interface,

 imagine you’re a type designer, working on creating a new

 fontWe shall egregiously abuse the distinction between

 a font and a typeface. Apologies to any type designers who may be

 reading.. After sketching some initial designs, you

 wish to experiment with bold, italic, and condensed variations.

 Let’s examine a tool to generate and explore such variations, from

 any initial design. For reasons that will soon be explained the

 quality of results is quite crude; please bear with us.

#### Starting Font

 …

 …

 …

 …

 …

 …

#### Modifications

Bold

Italic

Condensed

#### Result

 Of course, varying the bolding (i.e., the weight), italicization

 and width are just three ways you can vary a font. Imagine that

 instead of building specialized tools, users could build their own

 tool merely by choosing examples of existing fonts. For instance,

 suppose you wanted to vary the degree of serifing on a font. In

 the following, please select 5 to 10 sans-serif fonts from the top

 box, and drag them to the box on the left. Select 5 to 10 serif

 fonts and drag them to the box on the right. As you do this, a

 machine learning model running in your browser will automatically

 infer from these examples how to interpolate your starting font in

 either the serif or sans-serif direction:

Drag **sans-serif** examples here

Drag **serif** examples here

#### Starting font

 …

 …

 …

 …

 …

 …

#### Apply your new tool

#### Result

 In fact, we used this same technique to build the earlier bolding

 italicization, and condensing tool. To do so, we used the

 following examples of bold and non-bold fonts, of italic and

 non-italic fonts, and of condensed and non-condensed fonts:

#### Bold

#### Italic

#### Condensed

 To build these tools, we used what’s called a *generative

 model*; the particular model we use was trained

 by James Wexler. To

 understand generative models, consider that *a priori*

 describing a font appears to require a lot of data. For

 instance, if the font is 646464 by 646464 pixels, then we’d expect

 glyph. But we can use a generative model to find a much simpler

 description.

 We do this by building a neural network which takes a small number

 of input variables, called *latent variables*, and produces

 as output the entire glyph. For the particular model we use, we

 have 404040 latent space dimensions, and map that into the

 4,0964,0964,096-dimensional space describing all the pixels in the glyph.

 In other words, the idea is to map a low-dimensional space into a

 higher-dimensional space:

Neural network

 The generative model we use is a type of neural network known as

 a variational autoencoder

 (VAE). For our purposes, the details of the generative

 model aren’t so important. The important thing is that by

 changing the latent variables used as input, it’s possible to get

 different fonts as output. So one choice of latent variables will

 give one font, while another choice will give a different font:

…

…

…

 You can think of the latent variables as a compact, high-level

 representation of the font. The neural network takes that

 high-level representation and converts it into the full pixel

 data. It’s remarkable that just 404040 numbers can capture the

 apparent complexity in a glyph, which originally required 4,0964,0964,096

 variables.

 The generative model we use is learnt from a training set of more

 than 505050 thousand

 fonts Bernhardsson

 scraped from the open web. During training, the weights and

 biases in the network are adjusted so that the network can output

 a close approximation to any desired font from the training set,

 provided a suitable choice of latent variables is made. In some

 sense, the model is learning a highly compressed representation of

 all the training fonts.

 In fact, the model doesn’t just reproduce the training fonts. It

 can also generalize, producing fonts not seen in training. By

 being forced to find a compact description of the training

 examples, the neural net learns an abstract, higher-level model of

 what a font is. That higher-level model makes it possible to

 generalize beyond the training examples already seen, to produce

 realistic-looking fonts.

 Ideally, a good generative model would be exposed to a relatively

 small number of training examples, and use that exposure to

 generalize to the space of all possible human-readable fonts.

 That is, for any conceivable font – whether existing or

 perhaps even imagined in the future – it would be possible

 to find latent variables corresponding exactly to that font. Of

 course, the model we’re using falls far short of this ideal

 – a particularly egregious failure is that many fonts

 generated by the model omit the tail on the capital

 “Q” (you can see this in the examples above). Still,

 it’s useful to keep in mind what an ideal generative model would

 do.

 Such generative models are similar in some ways to how scientific

 theories work. Scientific theories often greatly simplify the

 description of what appear to be complex phenomena, reducing large

 numbers of variables to just a few variables from which many

 aspects of system behavior can be deduced. Furthermore, good

 scientific theories sometimes enable us to generalize to discover

 new phenomena.

 As an example, consider ordinary material objects. Such objects

 have what physicists call a *phase* – they may be a

 liquid, a solid, a gas, or perhaps something more exotic, like a

 superconductor

 or [Bose-Einstein

 complex, involving perhaps 102310^{23}1023 or so molecules. But the

 laws of thermodynamics and statistical mechanics enable us to find

 a simpler description, reducing that complexity to just a few

 variables (temperature, pressure, and so on), which encompass much

 of the behavior of the system. Furthermore, sometimes it’s

 possible to generalize, predicting unexpected new phases of

 matter. For example, in 1924, physicists used thermodynamics and

 statistical mechanics to predict a remarkable new phase of matter,

 Bose-Einstein condensation, in which a collection of atoms may all

 occupy identical quantum states, leading to surprising large-scale

 quantum interference effects. We’ll come back to this predictive

 ability in our later discussion of creativity and generative

 models.

 Returning to the nuts and bolts of generative models, how can we

 use such models to do example-based reasoning like that in the

 tool shown above? Let’s consider the case of the bolding tool. In

 that instance, we take the average of all the latent vectors for

 the user-specified bold fonts, and the average for all the

 user-specified non-bold fonts. We then compute the difference

 between these two average vectors:

bolding vectoraverage ofnon-bold fontsaverage ofbold fonts

 We’ll refer to this as the *bolding vector*. To make some

 given font bolder, we simply add a little of the bolding vector to

 the corresponding latent vector, with the amount of bolding vector

 added controlling the boldness of the resultIn

 practice, sometimes a slightly different procedure is used. In

 some generative models the latent vectors satisfy some constraints

 – for instance, they may all be of the same length. When

 that’s the case, as in our model, a more sophisticated

 “adding” operation must be used, to ensure the length

 remains the same. But conceptually, the picture of adding the

 bolding vector is the right way to think.:

 This technique was introduced

 by Larsen *et al*, and

 vectors like the bolding vector are sometimes called

 *attribute vectors*. The same idea is use to implement all

 the tools we’ve shown. That is, we use example fonts to creating

 a bolding vector, an italicizing vector, a condensing vector, and

 a user-defined serif vector. The interface thus provides a way of

 exploring the latent space in those four directions.

 The tools we’ve shown have many drawbacks. Consider the following

 example, where we start with an example glyph, in the middle, and

 either increase or decrease the bolding (on the right and left,

 respectively):

original

 Examining the glyphs on the left and right we see many unfortunate

 artifacts. Particularly for the rightmost glyph, the edges start to get

 rough, and the serifs begin to disappear. A better generative

 model would reduce those artifacts. That’s a good long-term

 research program, posing many intriguing problems. But even with

 the model we have, there are also some striking benefits to the

 use of the generative model.

 To understand these benefits, consider a naive approach to

 bolding, in which we simply add some extra pixels around a glyph’s

 edges, thickening it up. While this thickening perhaps matches a

 non-expert’s way of thinking about type design, an expert does

 something much more involved. In the following we show the

 results of this naive thickening procedure versus what is actually

 done, for Georgia and Helvetica:

####

#### Naive bolding

#### Actual bolding

Georgia

Helvetica

 As you can see, the naive bolding procedure produces quite

 different results, in both cases. For example, in Georgia, the

 left stroke is only changed slightly by bolding, while the right

 stroke is greatly enlarged, but only on one side. In both

 fonts, bolding doesn’t change the height of the font, while the

 naive approach does.

 As these examples show, good bolding is *not* a trivial

 process of thickening up a font. Expert type designers have many

 heuristics for bolding, heuristics inferred from much previous

 experimentation, and careful study of historical

 examples. Capturing all those heuristics in a conventional program

 would involve immense work. The benefit of using the generative

 model is that it automatically learns many such heuristics.

 For example, a naive bolding tool would rapidly fill in the

 enclosed negative space in the enclosed upper region of the letter

 “A”. The font tool doesn’t do this. Instead, it goes

 to some trouble to preserve the enclosed negative space, moving

 the A’s bar down, and filling out the interior strokes more slowly

 than the exterior. This principle is evident in the examples

 shown above, especially Helvetica, and it can also be seen in the

 operation of the font tool:

original

original

Height is constant

original

Even non-standard fonts are properly bolded

 The heuristic of preserving enclosed negative space is not *a

 priori* obvious. However, it’s done in many professionally

 designed fonts. If you examine examples like those shown above

 it’s easy to see why: it improves legibility. During training,

 our generative model has automatically inferred this principle

 from the examples it’s seen. And our bolding interface then makes

 this available to the user.

 In fact, the model captures many other heuristics. For instance,

 in the above examples the heights of the fonts are (roughly)

 preserved, which is the norm in professional font design. Again,

 what’s going on isn’t just a thickening of the font, but rather

 the application of a more subtle heuristic inferred by the

 generative model. Such heuristics can be used to create fonts

 with properties which would otherwise be unlikely to occur to

 users. Thus, the tool expands ordinary people’s ability to

 explore the space of meaningful fonts.

 The font tool is an example of a kind of cognitive technology. In

 particular, the primitive operations it contains can be

 internalized as part of how a user thinks. In this it resembles a

 program such as *Photoshop* or a spreadsheet or 3D graphics

 programs. Each provides a novel set of interface primitives,

 primitives which can be internalized by the user as fundamental

 new elements in their thinking. This act of internalization of new

 primitives is fundamental to much work on intelligence

 augmentation.

 The ideas shown in the font tool can be extended to other domains.

 Using the same interface, we can use a generative model to

 manipulate images of human faces using qualities such as

 expression, gender, or hair color. Or to manipulate sentences

 using length, sarcasm, or tone. Or to manipulate molecules using

 chemical properties:

 #alternate-uses .root {

 max-width: 300px;

 margin: 0 auto;

 padding: 0 20px;

 }

 @media(min-width: 600px) {

 #alternate-uses .root {

 max-width: 760px;

 display: grid;

 grid-template-columns: 1fr 1fr 1fr;

 grid-gap: 50px;

 }

 }

 .note {

 margin-top: 12px;

 grid-column-end: span 2;

 font-size: 10px;

 line-height: 1.5em;

 text-align: left;

 color: rgba(0, 0, 0, 0.4);

 }

#### Faces

Smiling

 Images from *Sampling Generative Networks* by White.

#### Sentences

“Everyone knows that a rich, single man must want a wife.”

“A rich, single man obviously must want a wife.”

“A single man must want a wife.”

Length

#### Molecules

Delayed fluorescence decay rate

 Such generative interfaces provide a kind of cartography of

 generative models, ways for humans to explore and make meaning

 using those models.

 We saw earlier that the font model automatically infers relatively

 deep principles about font design, and makes them available to

 users. While it’s great that such deep principles can be

 inferred, sometimes such models infer other things that are wrong,

 or undesirable. For example, White

 points out the addition of a smile vector in some face

 models will make faces not just smile more, but also appear more

 feminine. Why? Because in the training data more women than men

 were smiling. So these models may not just learn deep facts about

 the world, they may also internalize prejudices or erroneous

 beliefs. Once such a bias is known, it is often possible to make

 corrections. But to find those biases requires careful auditing

 of the models, and it is not yet clear how we can ensure such

 audits are exhaustive.

 More broadly, we can ask why attribute vectors work, when they

 work, and when they fail? At the moment, the answers to these

 questions are poorly understood.

 For the attribute vector to work requires that taking any starting

 font, we can construct the corresponding bold version by adding

 the *same* vector in the latent space. However, *a

 priori* there is no reason using a single constant vector to

 displace will work. It may be that we should displace in many

 different ways. For instance, the heuristics used to bold serif

 and sans-serif fonts are quite different, and so it seems likely

 that very different displacements would be involved:

#### Assumption

#### Reality

 Of course, we could do something more sophisticated than using a

 single constant attribute vector. Given pairs of example fonts

 (unbold, bold) we could train a machine learning algorithm to take

 as input the latent vector for the unbolded version and output the

 latent vector for the bolded version. With additional training

 data about font weights, the machine learning algorithm could

 learn to generate fonts of arbitrary weight. Attribute vectors

 are just an extremely simple approach to doing these kinds of

 operation.

 For these reasons, it seems unlikely that attribute vectors will

 last as an approach to manipulating high-level features. Over the

 next few years much better approaches will be developed. However,

 we can still expect interfaces offering operations broadly similar

 to those sketched above, allowing access to high-level and

 potentially user-defined concepts. That interface pattern doesn’t

 depend on the technical details of attribute vectors.

 Interactive Generative Adversarial Models

-------------------------------------------

 Let’s look at another example using machine learning models to

 augment human creativity. It’s the interactive generative

 adversarial networks, or iGANs, introduced

 by Zhu *et al* in 2016.

 One of the examples of Zhu *et al* is the use of iGANs in

 an interface to generate images of consumer products such as

 shoes. Conventionally, such an interface would require the

 programmer to write a program containing a great deal of knowledge

 about shoes: soles, laces, heels, and so on. Instead of doing

 this, Zhu *et al* train a generative model using 505050

 thousand images of shoes, downloaded from Zappos. They then use

 that generative model to build an interface that lets a user

 roughly sketch the shape of a shoe, the sole, the laces, and so

 on:

Excerpted from Zhu *et

 al*.

 The visual quality is low, in part because the generative model

 Zhu *et al* used is outdated by modern (2017) standards

 – with more modern models, the visual quality would be much

 higher.

 But the visual quality is not the point. Many interesting things

 are going on in this prototype. For instance, notice how the

 overall shape of the shoe changes considerably when the sole is

 filled in – it becomes narrower and sleeker. Many small

 details are filled in, like the black piping on the top of the

 white sole, and the red coloring filled in everywhere on the

 shoe’s upper. These and other facts are automatically deduced

 from the underlying generative model, in a way we’ll describe

 shortly.

 The same interface may be used to sketch landscapes. The only

 difference is that the underlying generative model has been

 trained on landscape images rather than images of shoes. In this

 case it becomes possible to sketch in just the colors associated

 to a landscape. For example, here’s a user sketching in some green

 grass, the outline of a mountain, some blue sky, and snow on the

 mountain:

Excerpted from Zhu *et

 al*.

 The generative models used in these interfaces are different than

 for our font model. Rather than using variational autoencoders,

 they’re based on generative

 adversarial networks (GANs). But the underlying idea is

 still to find a low-dimensional latent space which can be used to

 represent (say) all landscape images, and map that latent space to

 a corresponding image. Again, we can think of points in the

 latent space as a compact way of describing landscape images.

 Roughly speaking, the way the iGANs works is as follows. Whatever

 the current image is, it corresponds to some point in the latent

 space:

 Suppose, as happened in the earlier video, the user now sketches

 in a stroke outlining the mountain shape. We can think of the

 stroke as a constraint on the image, picking out a subspace of the

 latent space, consisting of all points in the latent space whose

 image matches that outline:

Subspace of all images that satisfy the mountain constraint

 The way the interface works is to find a point in the latent space

 which is near to the current image, so the image is not changed

 too much, but also coming close to satisfying the imposed

 constraints. This is done by optimizing an objective function

 which combines the distance to each of the imposed constraints, as

 well as the distance moved from the current point. If there’s

 just a single constraint, say, corresponding to the mountain

 stroke, this looks something like the following:

 We can think of this, then, as a way of applying constraints to

 the latent space to move the image around in meaningful ways.

 The iGANs have much in common with the font tool we showed

 earlier. Both make available operations that encode much subtle

 knowledge about the world, whether it be learning to understand

 what a mountain looks like, or inferring that enclosed negative

 space should be preserved when bolding a font. Both the iGANs and

 the font tool provide ways of understanding and navigating a

 high-dimensional space, keeping us on the natural space of fonts

 or shoes or landscapes. As Zhu *et al* remark:

>

>  [F]or most of us, even a simple image manipulation in Photoshop

>  presents insurmountable difficulties… any less-than-perfect

>  edit immediately makes the image look completely unrealistic. To

>  put another way, classic visual manipulation paradigm does not

>  prevent the user from “falling off” the manifold of

>  natural images.

>

 Like the font tool, the iGANs is a cognitive technology. Users

 can internalize the interface operations as new primitive elements

 in their thinking. In the case of shoes, for example, they can

 learn to think in terms of the difference they want to apply,

 adding a heel, or a higher top, or a special highlight. This is

 richer than the traditional way non-experts think about shoes

 (“Size 11, black” *etc*). To the extent that

 non-experts do think in more sophisticated ways –

 “make the top a little higher and sleeker” –

 they get little practice in thinking this way, or seeing the

 consequences of their choices. Having an interface like this

 enables easier exploration, the ability to develop idioms and the

 ability to plan, to swap ideas with friends, and so on.

 Two models of computation

---------------------------

 Let’s revisit the question we began the essay with, the question

 of what computers are for, and how this relates to intelligence

 augmentation.

 One common conception of computers is that they’re problem-solving

 machines: “computer, what is the result of firing this

 artillery shell in such-and-such a wind [and so on]?”;

 “computer, what will the maximum temperature in Tokyo be in

 5 days?”; “computer, what is the best move to take

 when the Go board is in this position?”; “computer,

 how should this image be classified?”; and so on.

 This is a conception common to both the early view of computers as

 number-crunchers, and also in much work on AI, both historically

 and today. It’s a model of a computer as a way of outsourcing

 cognition. In speculative depictions of possible future AI,

 this *cognitive outsourcing* model often shows up in the

 view of an AI as an oracle, able to solve some large class of

 problems with better-than-human performance.

 But a very different conception of what computers are for is

 possible, a conception much more congruent with work on

 intelligence augmentation.

 To understand this alternate view, consider our subjective

 experience of thought. For many people, that experience is verbal:

 they think using language, forming chains of words in their heads,

 similar to sentences in speech or written on a page. For other

 people, thinking is a more visual experience, incorporating

 representations such as graphs and maps. Still other people mix

 mathematics into their thinking, using algebraic expressions or

 diagrammatic techniques, such as Feynman diagrams and Penrose

 diagrams.

 In each case, we’re thinking using representations invented by

 other people: words, graphs, maps, algebra, mathematical diagrams,

 and so on. We internalize these cognitive technologies as we grow

 up, and come to use them as a kind of substrate for our thinking.

 For most of history, the range of available cognitive technologies

 has changed slowly and incrementally. A new word will be

 introduced, or a new mathematical symbol. More rarely, a radical

 new cognitive technology will be developed. For example, in 1637

 Descartes published his “Discourse on Method”,

 explaining how to represent geometric ideas using algebra, and

 vice versa:

-1.5-1-0.50.511.5-1.5-1-0.50.511.5

x2 + y2 = 1

While it may seem obvious now, it is remarkable that a

geometric shape and an equation can represent the same underlying

mathematical object.

 This enabled a radical change and expansion in how we think about

 both geometry and algebra.

 Historically, lasting cognitive technologies have been invented

 only rarely. But modern computers are a meta-medium enabling the

 rapid invention of many new cognitive technologies. Consider a

 relatively banal example, such

 as *Photoshop*. Adept *Photoshop* users routinely

 have formerly impossible thoughts such as: “let’s apply the

 clone stamp to the such-and-such layer.”. That’s an

 instance of a more general class of thought: “computer, [new

 type of action] this [new type of representation for a newly

 imagined class of object]”. When that happens, we’re using

 computers to expand the range of thoughts we can think.

 It’s this kind of *cognitive transformation* model which

 underlies much of the deepest work on intelligence augmentation.

 Rather than outsourcing cognition, it’s about changing the

 operations and representations we use to think; it’s about

 changing the substrate of thought itself. And so while cognitive

 outsourcing is important, this cognitive transformation view

 offers a much more profound model of intelligence augmentation.

 It’s a view in which computers are a means to change and expand

 human thought itself.

 Historically, cognitive technologies were developed by human

 inventors, ranging from the invention of writing in Sumeria and

 Mesoamerica, to the modern interfaces of designers such as Douglas

 Engelbart, Alan Kay, and others.

 Examples such as those described in this essay suggest that AI

 systems can enable the creation of new cognitive technologies.

 Things like the font tool aren’t just oracles to be consulted when

 you want a new font. Rather, they can be used to explore and

 discover, to provide new representations and operations, which can

 be internalized as part of the user’s own thinking. And while

 these examples are in their early stages, they suggest AI is not

 just about cognitive outsourcing. A different view of AI is

 possible, one where it helps us invent new cognitive technologies

 which transform the way we think.

 In this essay we’ve focused on a small number of examples, mostly

 involving exploration of the latent space. There are many other

 examples of artificial intelligence augmentation. To give some

 flavor, without being comprehensive:

 the sketch-rnn system, for neural

 network assisted drawing;

 the Wekinator, which enables

 users to rapidly build new musical instruments and artistic

 systems; TopoSketch, for developing

 animations by exploring latent spaces; machine learning models for

 designing overall typographic

 layout; and a generative model which enables

 interpolation between musical

 phrases. In each case, the systems use machine learning

 to enable new primitives which can be integrated into the user’s

 thinking. More broadly, artificial intelligence augmentation will

 draw on fields such as computational

 creativity and interactive machine

 learning.

 Finding powerful new primitives of thought

--------------------------------------------

 We’ve argued that machine learning systems can help create

 representations and operations which serve as new primitives in

 human thought. What properties should we look for in such new

 primitives? This is too large a question to be answered

 comprehensively in a short essay. But we will explore it briefly.

 Historically, important new media forms often seem strange when

 introduced. Many such stories have passed into popular culture:

 the near riot at the premiere of Stravinsky and Nijinksy’s

 “Rite of Spring”; the consternation caused by the

 early cubist paintings, leading

 *The New York Times* to

 comment: “What do they mean? Have those

 responsible for them taken leave of their senses? Is it art or

 madness? Who knows?”

 Another example comes from physics. In the 1940s, different

 formulations of the theory of quantum electrodynamics were

 developed independently by the physicists Julian Schwinger,

 Shin’ichirō Tomonaga, and Richard Feynman. In their work,

 Schwinger and Tomonaga used a conventional algebraic approach,

 along lines similar to the rest of physics. Feynman used a more

 radical approach, based on what are now known as Feynman diagrams,

 for depicting the interaction of light and matter:

Image by [Joel

 Attribution-Share Alike 3.0 Unported license

 Initially, the Schwinger-Tomonaga approach was easier for other

 physicists to understand. When Feynman and Schwinger presented

 their work at a 1948 workshop, Schwinger was immediately

 acclaimed. By contrast, Feynman left his audience mystified. As

 James Gleick put it in his biography of

 Feynman:

>

>  It struck Feynman that everyone had a favorite principle or

>  theorem and he was violating them all… Feynman knew he had

>  failed. At the time, he was in anguish. Later he said simply:

>  “I had too much stuff. My machines came from too far

>  away.”

>

 Of course, strangeness for strangeness’s sake alone is not

 useful. But these examples suggest that breakthroughs in

 representation often appear strange at first. Is there any

 underlying reason that is true?

 Part of the reason is because if some representation is truly new,

 then it will appear different than anything you’ve ever seen

 before. Feynman’s diagrams, Picasso’s paintings, Stravinsky’s

 music: all revealed genuinely new ways of making meaning. Good

 representations sharpen up such insights, eliding the familiar to

 show that which is new as vividly as possible. But because of

 that emphasis on unfamiliarity, the representation will seem

 strange: it shows relationships you’ve never seen before. In some

 sense, the task of the designer is to identify that core

 strangeness, and to amplify it as much as possible.

 Strange representations are often difficult to understand. At

 first, physicists preferred Schwinger-Tomonaga to Feynman. But as

 Feynman’s approach was slowly understood by physicists, they

 realized that although Schwinger-Tomonaga and Feynman were

 mathematically equivalent, Feynman was more powerful. As Gleick

 puts it:

>

>  Schwinger’s students at Harvard were put at a competitive

>  disadvantage, or so it seemed to their fellows elsewhere, who

>  suspected them of surreptitiously using the diagrams anyway. This

>  was sometimes true… Murray Gell-Mann later spent a semester

>  staying in Schwinger’s house and loved to say afterward that he

>  had searched everywhere for the Feynman diagrams. He had not

>  found any, but one room had been locked…

>

 These ideas are true not just of historical representations, but

 also of computer interfaces. However, our advocacy of strangeness

 in representation contradicts much conventional wisdom about

 interfaces, especially the widely-held belief that they should be

 “user friendly”, i.e., simple and immediately useable

 by novices. That most often means the interface is cliched, built

 from conventional elements combined in standard ways. But while

 using a cliched interface may be easy and fun, it’s an ease

 similar to reading a formulaic romance novel. It means the

 interface does not reveal anything truly surprising about its

 subject area. And so it will do little to deepen the user’s

 understanding, or to change the way they think. For mundane tasks

 that is fine, but for deeper tasks, and for the longer term, you

 want a better interface.

 Ideally, an interface will surface the deepest principles

 underlying a subject, revealing a new world to the user. When you

 learn such an interface, you internalize those principles, giving

 you more powerful ways of reasoning about that world. Those

 principles are the diffs in your understanding. They’re all you

 really want to see, everything else is at best support, at worst

 unimportant dross. The purpose of the best interfaces isn’t to be

 user-friendly in some shallow sense. It’s to be user-friendly in

 a much stronger sense, reifying deep

 principles about the world, making them the working

 conditions in which users live and create. At that point what once

 appeared strange can instead becomes comfortable and familiar,

 part of the pattern of thoughtA powerful instance of

 these ideas is when an interface reifies general-purpose

 principles. An example is an

 interface one of us developed

 based on the principle of conservation of energy. Such

 general-purpose principles generate multiple unexpected

 relationships between the entities of a subject, and so are a

 particularly rich source of insights when reified in an

 interface..

 What does this mean for the use of AI models for intelligence

 augmentation?

 Aspirationally, as we’ve seen, our machine learning models will

 help us build interfaces which reify deep principles in ways

 meaningful to the user. For that to happen, the models have to

 discover deep principles about the world, recognize those

 principles, and then surface them as vividly as possible in an

 interface, in a way comprehensible by the user.

 Of course, this is a tall order! The examples we’ve shown are just

 barely beginning to do this. It’s true that our models do

 sometimes discover relatively deep principles, like the

 preservation of enclosed negative space when bolding a font. But

 this is merely implicit in the model. And while we’ve built a tool

 which takes advantage of such principles, it’d be better if the

 model automatically inferred the important principles learned, and

 found ways of explicitly surfacing them through the interface.

 (Encouraging progress toward this has been made

 by InfoGANs, which use

 information-theoretic ideas to find structure in the latent

 space.) Ideally, such models would start to get at true

 explanations, not just in a static form, but in a dynamic form,

 manipulable by the user. But we’re a long way from that point.

 Do these interfaces inhibit creativity?

-----------------------------------------

 It’s tempting to be skeptical of the expressiveness of the

 interfaces we’ve described. If an interface constrains us to

 explore only the natural space of images, does that mean we’re

 merely doing the expected? Does it mean these interfaces can only

 be used to generate visual cliches? Does it prevent us from

 generating anything truly new, from doing truly creative work?

 To answer these questions, it’s helpful to identify two different

 modes of creativity. This two-mode model is over-simplified:

 creativity doesn’t fit so neatly into two distinct categories. Yet

 the model nonetheless clarifies the role of new interfaces in

 creative work.

 The first mode of creativity is the everyday creativity of a

 craftsperson engaged in their craft. Much of the work of a font

 designer, for example, consists of competent recombination of the

 best existing practices. Such work typically involves many

 creative choices to meet the intended design goals, but not

 developing key new underlying principles.

 For such work, the generative interfaces we’ve been discussing are

 promising. While they currently have many limitations, future

 research will identity and fix many deficiencies. This is

 happening rapidly with GANs: the original

 GANs had many limitations,

 but models soon appeared that were better adapted to

 images, improved the

 resolution, reduced artifactsSo much work has been

 done on improving resolution and reducing artifacts it seems

 unfair to single out any small set of papers, and to omit the many

 others., and so on. With enough iterations it’s

 plausible these generative interfaces will become powerful tools

 for craft work.

 The second mode of creativity aims toward developing new

 principles that fundamentally change the range of creative

 expression. One sees this in the work of artists such as Picasso

 or Monet, who violated existing principles of painting, developing

 new principles which enabled people to see in new ways.

 Is it possible to do such creative work, while using a generative

 interface? Don’t such interfaces constrain us to the space of

 natural images, or natural fonts, and thus actively prevent us

 from exploring the most interesting new directions in creative

 work?

 The situation is more complex than this.

 In part, this is a question about the power of our generative

 models. In some cases, the model can only generate recombinations

 of existing ideas. This is a limitation of an ideal GAN, since a

 perfectly trained GAN generator will reproduce the training

 distribution. Such a model can’t directly generate an image based

 on new fundamental principles, because such an image wouldn’t look

 anything like it’s seen in its training data.

 Artists such as [Mario

 Klingemann](http://quasimondo.com/) and [Mike

 Tyka](http://www.miketyka.com/) are now using GANs to create interesting

 artwork. They’re doing that using “imperfect” GAN

 models, which they seem to be able to use to explore interesting

 new principles; it’s perhaps the case that bad GANs may be more

 artistically interesting than ideal GANs. Furthermore, nothing

 says an interface must only help us explore the latent space.

 Perhaps operations can be added which deliberately take us out

 of the latent space, or to less probable (and so more

 surprising) parts of the space of natural images.

 Of course, GANs are not the only generative models. In a

 sufficiently powerful generative model, the generalizations

 discovered by the model may contain ideas going beyond what humans

 have discovered. In that case, exploration of the latent space may

 enable us to discover new fundamental principles. The model would

 have discovered stronger abstractions than human experts. Imagine

 a generative model trained on paintings up until just before the

 time of the cubists; might it be that by exploring that model it

 would be possible to discover cubism? It would be an analogue to

 something like the prediction of Bose-Einstein condensation, as

 discussed earlier in the essay. Such invention is beyond today’s

 generative models, but seems a worthwhile aspiration for future

 models.

 Our examples so far have all been based on generative models. But

 there are some illuminating examples which are not based on

 generative models. Consider the pix2pix system developed

 by Isola *et al*. This

 system is trained on pairs of images, e.g., pairs showing the

 edges of a cat, and the actual corresponding cat. Once trained,

 it can be shown a set of edges and asked to generate an image for

 an actual corresponding cat. It often does this quite well:

 .cat-grid .row {

 grid-template-columns: 1fr 1fr 0.5fr;

 align-items: center;

 }

 .cat-grid .row {

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 }

 .cat-grid .row:last-child {

 border-bottom: none;

 }

#### Input

#### Output

[Live demo by Christopher Hesse](https://affinelayer.com/pixsrv/)

 When supplied with unusual constraints, pix2pix can produce

 striking images:

[Bread cat by Ivy Tsai](https://twitter.com/ivymyt/status/834174687282241537)

[Cat beholder by Marc Hesse](https://affinelayer.com/pixsrv/beholder.jpg)

Spiral cat

 This is perhaps not high creativity of a Picasso-esque level. But

 it is still surprising. It’s certainly unlike images most of us

 have ever seen before. How does pix2pix and its human user achieve

 this kind of result?

 Unlike our earlier examples, pix2pix is not a generative model.

 This means it does not have a latent space or a corresponding

 space of natural images. Instead, there is a neural network,

 called, confusingly, a generator – this is not meant in the

 same sense as our earlier generative models – that takes as

 input the constraint image, and produces as output the filled-in

 image.

 The generator is trained adversarially against a discriminator

 network, whose job is to distinguish between pairs of images

 generated from real data, and pairs of images generated by the

 generator.

 While this sounds similar to a conventional GAN, there is a

 crucial difference: there is no latent vector input to the

 generatorActually, Isola *et

 al* experimented with adding such a latent vector to

 the generator, but found it made little difference to the

 resulting images.. Rather, there is simply an input

 constraint. When a human inputs a constraint unlike anything seen

 in training, the network is forced to improvise, doing the best it

 can to interpret that constraint according to the rules it has

 previously learned. The creativity is the result of a forced

 merger of knowledge inferred from the training data, together with

 novel constraints provided by the user. As a result, even

 relatively simple ideas – like the bread- and beholder-cats

 – can result in striking new types of images, images not

 within what we would previously have considered the space of

 natural images.

Conclusion

----------

 It is conventional wisdom that AI will change how we interact with

 computers. Unfortunately, many in the AI community greatly

 underestimate the depth of interface design, often regarding it as

 a simple problem, mostly about making things pretty or

 easy-to-use. In this view, interface design is a problem to be

 handed off to others, while the hard work is to train some machine

 learning system.

 This view is incorrect. At its deepest, interface design means

 developing the fundamental primitives human beings think and

 create with. This is a problem whose intellectual genesis goes

 back to the inventors of the alphabet, of cartography, and of

 musical notation, as well as modern giants such as Descartes,

 Playfair, Feynman, Engelbart, and Kay. It is one of the hardest,

 most important and most fundamental problems humanity grapples

 with.

 As discussed earlier, in one common view of AI our computers will

 continue to get better at solving problems, but human beings will

 remain largely unchanged. In a second common view, human beings

 will be modified at the hardware level, perhaps directly through

 neural interfaces, or indirectly through whole brain emulation.

 We’ve described a third view, in which AIs actually change

 humanity, helping us invent new cognitive technologies, which

 expand the range of human thought. Perhaps one day those

 cognitive technologies will, in turn, speed up the development of

 AI, in a virtuous feedback cycle:

 It would not be a Singularity in machines. Rather, it would be a

 Singularity in humanity’s range of thought. Of course, this loop

 is at present extremely speculative. The systems we’ve described

 can help develop more powerful ways of thinking, but there’s at

 most an indirect sense in which those ways of thinking are being

 used in turn to develop new AI systems.

 Of course, over the long run it’s possible that machines will

 exceed humans on all or most cognitive tasks. Even if that’s the

 case, cognitive transformation will still be a valuable end, worth

 pursuing in its own right. There is pleasure and value involved

 in learning to play chess or Go well, even if machines do it

 better. And in activities such as story-telling the benefit often

 isn’t so much the artifact produced as the process of construction

 itself, and the relationships forged. There is intrinsic value in

 personal change and growth, apart from instrumental benefits.

 The interface-oriented work we’ve discussed is outside the

 narrative used to judge most existing work in artificial

 intelligence. It doesn’t involve beating some benchmark for a

 classification or regression problem. It doesn’t involve

 impressive feats like beating human champions at games such as

 Go. Rather, it involves a much more subjective and

 difficult-to-measure criterion: is it helping humans think and

 create in new ways?

 This creates difficulties for doing this kind of work,

 particularly in a research setting. Where should one publish?

 What community does one belong to? What standards should be

 applied to judge such work? What distinguishes good work from

 bad?

 We believe that over the next few years a community will emerge

 which answers these questions. It will run workshops and

 conferences. It will publish work in venues such as Distill. Its

 standards will draw from many different communities: from the

 artistic and design and musical communities; from the mathematical

 community’s taste in abstraction and good definition; as well as

 from the existing AI and IA communities, including work on

 computational creativity and human-computer interaction. The

 long-term test of success will be the development of tools which

 are widely used by creators. Are artists using these tools to

 develop remarkable new styles? Are scientists in other fields

 using them to develop understanding in ways not otherwise

 possible? These are great aspirations, and require an approach

 that builds on conventional AI work, but also incorporates very

 different norms.

Augmenting Human Intellect: A Conceptual Framework

deeplearn.js font demo

Auto-encoding variational Bayes

Analyzing 50k fonts using deep neural networks

Autoencoding beyond pixels using a learned similarity metric

Sampling Generative Networks

Writing with the Machine

Automatic chemical design using a data-driven continuous representation of molecules

Generative visual manipulation on the natural image manifold

Generative adversarial nets

A Neural Representation of Sketch Drawings

Real-time human interaction with supervised learning algorithms for music composition and performance

TopoSketch: Drawing in Latent Space

Taking The Robots To Design School, Part 1

Hierarchical Variational Autoencoders for Music

Computational creativity: the final frontier?

Interactive machine learning: letting users build classifiers

Eccentric School of Painting Increased Its Vogue in the Current Art Exhibition — What Its Followers Attempt to Do

Genius: The Life and Science of Richard Feynman

Thought as a Technology

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Image-to-Image Translation with Conditional Adversarial Networks

Using Artificial Intelligence to Augment Human Intelligence.html


Weight Banding

Michael Petrov
Chelsea Voss
Ludwig Schubert
Nick Cammarata
Gabriel Goh
Chris Olah

Abstract: This article is part of the Circuits thread,  an experimental format collecting invited short articles and critical  commentary delving into the inner workings of neural networks.

![](Weight%20Banding_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

Introduction

------------

 Open up any ImageNet conv net and look at the weights in the last

layer. You’ll find a uniform spatial pattern to them, dramatically

unlike anything we see elsewhere in the network. No individual weight is

 unusual, but the uniformity is so striking that when we first

discovered it we thought it must be a bug. Just as different biological

tissue types jump out as distinct under a microscope, the weights in

this final layer jump out as distinct when visualized with NMF. We call

this phenomenon *weight banding*.

Microscope slides of different tissues

Muscle tissue

Epithelial tissue

Typical layer

Layer with weight banding

NMF of weights at different layers

 and small circuits. In contrast, weight banding is an example of what

we call a “structural phenomenon,” a larger-scale pattern in the

circuits and features of a neural network. Other examples of structural

 In the case of weight banding, we think of it as a structural

phenomenon because the pattern appears at the scale of an entire layer.

 In addition to describing weight banding, we’ll explore when and

why it occurs. We find that there appears to be a causal link between

whether a model uses global average pooling or fully connected layers at

 the end, suggesting that weight banding is part of an algorithm for

preserving information about larger scale structure in images.

Establishing causal links like this is a step towards closing the loop

between practical decisions in training neural networks and the

phenomena we observe inside them.

Where weight banding occurs

---------------------------

 In order to see the bands, we need to visualize the spatial

structure of the weights, as shown below. We typically do this using

 Visualizing Weights. For each neuron, we take the weights connecting it

 to the previous layer. We then use NMF to reduce the number of

dimensions corresponding to channels in the previous layer down to 3

factors, which we can map to RGB channels. Since which factor is which

is arbitrary, we use a heuristic to make the mapping consistent across

neurons. This reveals a very prominent pattern of horizontalThe

 stripes aren’t always perfectly horizontal - sometimes they exhibit a

slight preference for extra weight in the center of the central band, as

 seen in some examples below. stripes.

[2](#figure-2).

 These common networks have pooling operations before their fully

 connected layers and consistently show banding at their last

 convolutional layers.

86 weights hidden

InceptionV1

mixed 5b

470 weights hidden

ResNet50

block 4 unit 3

470 weights hidden

VGG19

conv5

 Interestingly, AlexNet does not exhibit this phenomenon.

[3](#figure-3).

 AlexNet does not have a pooling operation before its fully connected

 layers and does not show banding at its last convolutional

 layer.

 To make it easier to look for groups of similar weights, we

 sorted the neurons at each layer by similarity of their reduced

 forms.

160 weights hidden

AlexNet

conv5

 Unlike most modern vision models, AlexNet does not use global

average pooling. Instead, it has a fully connected layer directly

connected to its final convolutional layer, allowing it to treat

different positions differently. If one looks at the weights of this

fully connected layer, the weights strongly vary as a function of the

global y position.

 The horizontal stripes in weight banding mean that the filters

don’t care about horizontal position, but are strongly encoding relative

 vertical position. Our hypothesis is that weight banding is a learned

way to preserve spatial information as it gets lost through various

pooling operations.

 In the next section, we will construct our own simplified vision

network and investigate variations on its architecture in order to

understand exactly which conditions are necessary to produce weight

banding.

What affects banding

--------------------

 We’d like to understand which architectural decisions affect

weight banding. This will involve trying out different architectures and

 seeing whether weight banding persists.

 Since we will only want to change a single architectural parameter

 at a time, we will need a consistent baseline to apply our

modifications to. Ideally, this baseline would be as simple as possible.

 We created a simplified network architecture with 6 groups of

convolutions, separated by L2 pooling layers. At the end, it has a

global average pooling operation that reduces the input to 512 values

that are then fed to a fully connected layer with 1001 outputs.

[4](#figure-4). Our simplified vision network architecture.

 This simplified network reliably produces weight banding in its last layer

 (and usually in the two preceding layers as well).

416 weights hidden

simplified model (`5b`), baseline

 In the rest of this section, we’ll experiment with modifying this

architecture and its training settings and seeing if weight banding is

preserved.

### Rotating images 90 degrees

 To rule out bugs in training or some strange numerical problem, we decided

 to do a training run with the input rotated by 90 degrees. This sanity check

 yielded a very clear result showing *vertical* banding in the resulting

416 weights hidden

[6](#figure-6). simplified model (`5b`), 90º rotation

### Fully connected layer without global average pooling

 We remove the global average pooling step in our simplified model,

 allowing the fully connected layer to see all spatial positions at

once. This model did **not** exhibit weight banding, but

used 49x more parameters in the fully connected layer and overfit to the

 training set. This is pretty strong evidence that the use of aggressive

 pooling after the last convolutions in common models causes weight

banding. This result is also consistent with AlexNet not showing this

banding phenomenon (since it also does not have global average pooling).

416 weights hidden

### Average pooling along x-axis only

 We average out each row of the final convolutional layer, so that

vertical absolute position is preserved but horizontal absolute position

 is not.Since this model has 7x7 spatial

positions in the final convolutional layer, this modification increases

the number of parameters in the fully connected layer by 7x, but not the

[8](#figure-8).

416 weights hidden

 simplified model (`5a`), x-axis pooling

416 weights hidden

simplified model (`5b`), x-axis pooling

### Approaches where weight banding persisted

* Global average pooling with learned spatial masks. By applying

several different spatial masks and global average pooling, we can allow

 the model to preserve some spatial information. Intuitively, each mask

can select for a different subset of spatial positions.

 We tried experimental runs using each of 3, 5, or 16 different

masks.

 The masks that were learned corresponded to large-scale global

structure, but banding was still strongly present.

 `5b`.

* Adding a 7x7x512 mask with learned weights after `5b`. The hope was that a

 mask would help each `5b` neuron focus on the right parts of the 7x7 image

 without a convolution.

* Adding CoordConv channels to the inputs

 of `5a` and `5b`.

* Splitting the output of `5b` into 16 7x7x32 channel groups and feeding

 concatenated into the input of the final 1001-class fully connected layer.

 by VGG).

Confirming banding interventions in common architectures

--------------------------------------------------------

 In the previous section, we observed two interventions that

clearly affected weight banding: rotating the dataset by 90º and

removing the global average pooling before the fully connected layer.

 To confirm that these effects hold beyond our simplified model, we

 decided to make the same interventions to three

 common architectures (InceptionV1, ResNet50, VGG19) and train them

 from

 scratch.

With one exception, the effect holds in all three models.

#### InceptionV1

[9](#figure-9). Inception V1, layer `mixed_5c`, 5x5 convolution

86 weights hidden

baseline

86 weights hidden

90º rotation

86 weights hidden

global average pooling layer removed

#### ResNet50

[10](#figure-10). ResNet50, last 3x3 convolutional layer

470 weights hidden

baseline

470 weights hidden

90º rotation

470 weights hidden

global average pooling layer removed

#### VGG19

[11](#figure-11). VGG19, last 3x3 convolutional layer.

470 weights hidden

baseline

470 weights hidden

90º rotation

470 weights hidden

global average pooling layer removed

 The one exception is VGG19, where the removal of the pooling

operation before its set of fully connected layers did not eliminate

weight banding as expected; these weights look fairly similar to the

baseline. However, it clearly responds to rotation.

Conclusion

----------

 Once we really understand neural networks, one would expect us to

be able to leverage that understanding to design more effective neural

networks architectures. Early papers, like Zeiler et al,

 emphasized this quite strongly, but it’s unclear whether there have yet

 been any significant successes in doing this. This hints at significant

 limitations in our work. It may also be a missed opportunity: it seems

likely that if interpretability was useful in advancing neural network

capabilities, it would become more integrated into other research and

get attention from a wider range of researchers.

 It’s unclear whether weight banding is “good” or “bad.”On

 one hand, the 90º rotation experiment shows that weight banding is a

product of the dataset and is encoding useful information into the

weights. However, if spatial information could flow through the network

in a different, more efficient way, then perhaps the channels would be

able to focus on encoding relationships between features without needing

 to track spatial positions. We don’t have any

recommendation or action to take away from it. However, it is an example

 of a consistent link between architecture decisions and the resulting

trained weights. It has the right sort of flavor for something that

could inform architectural design, even if it isn’t particularly

actionable itself.

 More generally, weight banding is an example of a large-scale

structure. One of the major limitations of circuits has been how

small-scale it is. We’re hopeful that larger scale structures like

weight banding may help circuits form a higher-level story of neural

networks.

![](Weight%20Banding_files/multiple-pages.svg)

 an experimental format collecting invited short articles and critical

commentary delving into the inner workings of neural networks.

Muscle Tissue: Cardiac Muscle

Epithelial Tissues: Stratified Squamous Epithelium

Deconvolution and Checkerboard Artifacts

Going Deeper with Convolutions

Deep Residual Learning for Image Recognition

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Classification with Deep Convolutional Neural Networks

An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

Visualizing and Understanding Convolutional Networks