A Gentle Introduction to Graph Neural Networks

Benjamin Sanchez-Lengeling
Emily Reif
Adam Pearce
Alexander B. Wiltschko

Abstract: This article is one of two Distill publications about graph neural networks. Take a look at Understanding Convolutions on Graphs to understand how convolutions over images generalize naturally to convolutions over graphs.

Graphs are all around us; real world objects are often defined in

terms of their connections to other things. A set of objects, and the

connections between them, are naturally expressed as a *graph*.

Researchers have developed neural networks that operate on graph data

(called graph neural networks, or GNNs) for over a decade.

 Recent developments have increased their capabilities and expressive

power. We are starting to see practical applications in areas such as

This article explores and explains modern graph neural networks. We

divide this work into four parts. First, we look at what kind of data is

 most naturally phrased as a graph, and some common examples. Second, we

 explore what makes graphs different from other types of data, and some

of the specialized choices we have to make when using graphs. Third, we

build a modern GNN, walking through each of the parts of the model,

starting with historic modeling innovations in the field. We move

gradually from a bare-bones implementation to a state-of-the-art GNN

model. Fourth and finally, we provide a GNN playground where you can

play around with a real-word task and dataset to build a stronger

intuition of how each component of a GNN model contributes to the

predictions it makes.

Three types of attributes we might find in a graph, hover over to

highlight each attribute. Other types of graphs and attributes are

 They can also be undirected, where there is no notion of source or

destination nodes, and information flows both directions. Note that

having a single undirected edge is equivalent to having one directed

Graphs are very flexible data structures, and if this seems abstract

now, we will make it concrete with examples in the next section.

Graphs and where to find them

-----------------------------

You're probably already familiar with some types of graph data, such

as social networks. However, graphs are an extremely powerful and

general representation of data, we will show two types of data that you

might not think could be modeled as graphs: images and text. Although

counterintuitive, one can learn more about the symmetries and structure

of images and text by viewing them as graphs,, and build an intuition

that will help understand other less grid-like graph data, which we will

 discuss later.

### Images as graphs

We typically think of images as rectangular grids with image

channels, representing them as arrays (e.g., 244x244x3 floats). Another

way to think of images is as graphs with regular structure, where each

pixel represents a node and is connected via an edge to adjacent pixels.

 Each non-border pixel has exactly 8 neighbors, and the information

stored at each node is a 3-dimensional vector representing the RGB value

 of the pixel.

 with an entry if two nodes share an edge. Note that each of these three

 representations below are different views of the same piece of data.

### Text as graphs

We can digitize text by associating indices to each character, word,

or token, and representing text as a sequence of these indices. This

creates a simple directed graph, where each character or index is a node

 and is connected via an edge to the node that follows it.

→→→→→→→→→→→→→→→→→→→→GraphsareallaroundusGraphsareallaroundus

Edit the text above to see how the graph representation changes.

Of course, in practice, this is not usually how text and images are

encoded: these graph representations are redundant since all images and

all text will have very regular structures. For instance, images have a

banded structure in their adjacency matrix because all nodes (pixels)

are connected in a grid. The adjacency matrix for text is just a

diagonal line, because each word only connects to the prior word, and to

 the next one.

This representation (a sequence of character tokens) refers to the way

text is often represented in RNNs; other models, such as Transformers,

can be considered to view text as a fully connected graph where we learn

### Graph-valued data in the wild

Graphs are a useful tool to describe data you might already be

familiar with. Let's move on to data which is more heterogeneously

structured. In these examples, the number of neighbors to each node is

variable (as opposed to the fixed neighborhood size of images and text).

 This data is hard to phrase in any other way besides a graph.

**Molecules as graphs.** Molecules are the building

blocks of matter, and are built of atoms and electrons in 3D space. All

particles are interacting, but when a pair of atoms are stuck in a

stable distance from each other, we say they share a covalent bond.

Different pairs of atoms and bonds have different distances (e.g.

single-bonds, double-bonds). It's a very convenient and common

abstraction to describe this 3D object as a graph, where nodes are atoms

(Left) 3d representation of the Citronellal molecule (Center) Adjacency

matrix of the bonds in the molecule (Right) Graph representation of the

molecule.

(Left) 3d representation of the Caffeine molecule (Center) Adjacency

matrix of the bonds in the molecule (Right) Graph representation of the

molecule.

**Social networks as graphs.** Social networks are tools

 to study patterns in collective behaviour of people, institutions and

organizations. We can build a graph representing groups of people by

modelling individuals as nodes, and their relationships as edges.

(Left) Image of a scene from the play "Othello". (Center) Adjacency

matrix of the interaction between characters in the play. (Right) Graph

representation of these interactions.

(Left) Image of karate tournament. (Center) Adjacency matrix of the

interaction between people in a karate club. (Right) Graph

representation of these interactions.

**Citation networks as graphs.** Scientists routinely

cite other scientists' work when publishing papers. We can visualize

these networks of citations as a graph, where each paper is a node, and

each *directed* edge is a citation between one paper and another.

 Additionally, we can add information about each paper into each node,

such as a word embedding of the abstract. (see ,  , ).

**Other examples.** In computer vision, we sometimes

want to tag objects in visual scenes. We can then build graphs by

 can also be phrased as graphs, where the variables are nodes, and edges

 are operations that have these variables as input and output. You might

 see the term "dataflow graph" used in some of these contexts.

The structure of real-world graphs can vary greatly between different

 types of data — some graphs have many nodes with few connections

between them, or vice versa. Graph datasets can vary widely (both within

 a given dataset, and between datasets) in terms of the number of nodes,

 edges, and the connectivity of nodes.

 Edges per node (degree)

 Dataset

 Domain

 graphs

 nodes

 edges

 min

 mean

 max

 karate club

 Social network

 1

 34

 78

 4.5

 17

 qm9

 Small molecules

 134k

 ≤ 9

 ≤26

 1

 2

 5

 Cora

 Citation network

 1

 23,166

 91,500

 1

 7.8

 379

 Wikipedia links, English

 Knowledge graph

 1

 12M

 378M

 62.24

 1M

Summary statistics on graphs found in the real world. Numbers are

dependent on featurization decisions. More useful statistics and graphs

can be found in KONECT

What types of problems have graph structured data?

--------------------------------------------------

We have described some examples of graphs in the wild, but what tasks

 do we want to perform on this data? There are three general types of

prediction tasks on graphs: graph-level, node-level, and edge-level.

In a graph-level task, we predict a single property for a whole

graph. For a node-level task, we predict some property for each node in a

 graph. For an edge-level task, we want to predict the property or

presence of edges in a graph.

For the three levels of prediction problems described above

(graph-level, node-level, and edge-level), we will show that all of the

following problems can be solved with a single model class, the GNN. But

 first, let's take a tour through the three classes of graph prediction

problems in more detail, and provide concrete examples of each.

### Graph-level task

In a graph-level task, our goal is to predict the property of an

entire graph. For example, for a molecule represented as a graph, we

might want to predict what the molecule smells like, or whether it will

bind to a receptor implicated in a disease.

This is analogous to image classification problems with MNIST and

CIFAR, where we want to associate a label to an entire image. With text,

 a similar problem is sentiment analysis where we want to identify the

mood or emotion of an entire sentence at once.

### Node-level task

A classic example of a node-level prediction problem is Zach's karate club.

 The dataset is a single social network graph made up of individuals

that have sworn allegiance to one of two karate clubs after a political

rift. As the story goes, a feud between Mr. Hi (Instructor) and John H

(Administrator) creates a schism in the karate club. The nodes represent

 individual karate practitioners, and the edges represent interactions

between these members outside of karate. The prediction problem is to

classify whether a given member becomes loyal to either Mr. Hi or John

H, after the feud. In this case, distance between a node to either the

Instructor or Administrator is highly correlated to this label.

On the left we have the initial conditions of the problem, on the right

we have a possible solution, where each node has been classified based

on the alliance. The dataset can be used in other graph problems like

unsupervised learning.

 where we are trying to label the role of each pixel in an image. With

text, a similar task would be predicting the parts-of-speech of each

word in a sentence (e.g. noun, verb, adverb, etc).

### Edge-level task

The remaining prediction problem in graphs is *edge prediction*.

One example of edge-level inference is in image scene understanding.

Beyond identifying objects in an image, deep learning models can be used

 to predict the relationship between them. We can phrase this as an

edge-level classification: given nodes that represent the objects in the

 image, we wish to predict which of these nodes share an edge or what

the value of that edge is. If we wish to discover connections between

entities, we could consider the graph fully connected and based on their

 predicted value prune edges to arrive at a sparse graph.

![](A%20Gentle%20Introduction%20to%20Graph%20Neural%20Networks_files/merged.png)

In (b), above, the original image (a) has been segmented into five

entities: each of the fighters, the referee, the audience and the mat.

(C) shows the relationships between these entities.

On the left we have an initial graph built from the previous visual

scene. On the right is a possible edge-labeling of this graph when some

connections were pruned based on the model's output.

The challenges of using graphs in machine learning

--------------------------------------------------

So, how do we go about solving these different graph tasks with

neural networks? The first step is to think about how we will represent

graphs to be compatible with neural networks.

Machine learning models typically take rectangular or grid-like

arrays as input. So, it's not immediately intuitive how to represent

them in a format that is compatible with deep learning. Graphs have up

to four types of information that we will potentially want to use to

make predictions: nodes, edges, global-context and connectivity. The

first three are relatively straightforward: for example, with nodes we

However, representing a graph's connectivity is more complicated.

Perhaps the most obvious choice would be to use an adjacency matrix,

since this is easily tensorisable. However, this representation has a

few drawbacks. From the [example dataset table](#table), we

see the number of nodes in a graph can be on the order of millions, and

the number of edges per node can be highly variable. Often, this leads

to very sparse adjacency matrices, which are space-inefficient.

Another problem is that there are many adjacency matrices that can

encode the same connectivity, and there is no guarantee that these

different matrices would produce the same result in a deep neural

network (that is to say, they are not permutation invariant).

Learning permutation invariant operations is an area of recent research.

 from before can be described equivalently with these two adjacency

matrices. It can also be described with every other possible

permutation of the nodes.

Two adjacency matrices representing the same graph.

The example below shows every adjacency matrix that can describe this

 small graph of 4 nodes. This is already a significant number of

adjacency matrices–for larger examples like Othello, the number is

untenable.

All of these adjacency matrices represent the same graph. Click on an

edge to remove it on a "virtual edge" to add it and the matrices will

update accordingly.

One elegant and memory-efficient way of representing sparse matrices

 as a tuple (i,j) in the k-th entry of an adjacency list. Since we

expect the number of edges to be much lower than the number of entries

Nodes

[<PHONE>

Edges

[2100000001000000010000002110

Adjacency List

Global

0

Hover and click on the edges, nodes, and global graph marker to view and

 change attribute representations. On one side we have a small graph and

 on the other the information of the graph in a tensor representation.

It should be noted that the figure uses scalar values per

node/edge/global, but most practical tensor representations have vectors

Graph Neural Networks

---------------------

Now that the graph's description is in a matrix format that is

permutation invariant, we will describe using graph neural networks

(GNNs) to solve graph prediction tasks. **A GNN is an optimizable

transformation on all attributes of the graph (nodes, edges,

global-context) that preserves graph symmetries (permutation

 GNNs adopt a "graph-in, graph-out" architecture meaning that these

model types accept a graph as input, with information loaded into its

nodes, edges and global-context, and progressively transform these

embeddings, without changing the connectivity of the input graph.

### The simplest GNN

 (with vectors instead of scalars), we are now ready to build a GNN. We

will start with the simplest GNN architecture, one where we learn new

embeddings for all graph attributes (nodes, edges, global), but where we

 do not yet use the connectivity of the graph.

For simplicity, the previous diagrams used scalars to represent graph

attributes; in practice feature vectors, or embeddings, are much more

useful.

This GNN uses a separate multilayer perceptron (MLP) (or your

favorite differentiable model) on each component of a graph; we call

this a GNN layer. For each node vector, we apply the MLP and get back a

learned node-vector. We do the same for each edge, learning a per-edge

embedding, and also for the global-context vector, learning a single

embedding for the entire graph.

A single layer of a simple GNN. A graph is the input, and each component

 (V,E,U) gets updated by a MLP to produce a new graph. Each function

subscript indicates a separate function for a different graph attribute

at the n-th layer of a GNN model.

Because a GNN does not update the connectivity of the input graph, we

 can describe the output graph of a GNN with the same adjacency list and

 the same number of feature vectors as the input graph. But, the output

graph has updated embeddings, since the GNN has updated each of the

node, edge and global-context representations.

### GNN Predictions by Pooling Information

We will consider the case of binary classification, but this

framework can easily be extended to the multi-class or regression case.

If the task is to make binary predictions on nodes, and the graph

already contains node information, the approach is straightforward — for

 each node embedding, apply a linear classifier.

We could imagine a social network, where we wish to anonymize user data

(nodes) by not using them, and only using relational data (edges). One

 subsection. In the Karate club example, this would be just using the

number of meetings between people to determine the alliance to Mr. Hi or

 John H.

However, it is not always so simple. For instance, you might have

information in the graph stored in edges, but no information in nodes,

but still need to make predictions on nodes. We need a way to collect

information from edges and give them to nodes for prediction. We can do

this by *pooling*. Pooling proceeds in two steps:

2. The gathered embeddings are then *aggregated*, usually via a sum operation.

+Aggregate information from adjacent edges

Hover over a node (black node) to visualize which edges are gathered and

 aggregated to produce an embedding for that target node.

So If we only have edge-level features, and are trying to predict

binary node information, we can use pooling to route (or pass)

information to where it needs to go. The model looks like this.

 sub section. Nodes can be recognized as image entities, and we are

trying to predict if the entities share a relationship (binary edges).

If we only have node-level features, and need to predict a binary

global property, we need to gather all available node information

This is a common scenario for predicting molecular properties. For

example, we have atomic information, connectivity and we would like to

know the toxicity of a molecule (toxic/not toxic), or if it has a

particular odor (rose/not rose).

In our examples, the classification model *𝑐cc*

 can easily be replaced with any differentiable model, or adapted to

multi-class classification using a generalized linear model.

An end-to-end prediction task with a GNN model.

Now we've demonstrated that we can build a simple GNN model, and make

 binary predictions by routing information between different parts of

the graph. This pooling technique will serve as a building block for

constructing more sophisticated GNN models. If we have new graph

attributes, we just have to define how to pass information from one

attribute to another.

Note that in this simplest GNN formulation, we're not using the

connectivity of the graph at all inside the GNN layer. Each node is

processed independently, as is each edge, as well as the global context.

 We only use connectivity when pooling information for prediction.

### Passing messages between parts of the graph

We could make more sophisticated predictions by using pooling within

the GNN layer, in order to make our learned embeddings aware of graph

Message passing works in three steps:

2. Aggregate all messages via an aggregate function (like sum).

These steps are key for leveraging the connectivity of graphs. We

will build more elaborate variants of message passing in GNN layers that

 yield GNN models of increasing expressiveness and power.

This is reminiscent of standard convolution: in essence, message

passing and convolution are operations to aggregate and process the

information of an element's neighbors in order to update the element's

value. In graphs, the element is a node, and in images, the element is a

 pixel. However, the number of neighboring nodes in a graph can be

variable, unlike in an image where each pixel has a set number of

neighboring elements.

By stacking message passing GNN layers together, a node can

eventually incorporate information from across the entire graph: after

three layers, a node has information about the nodes three steps away

from it.

Schematic for a GCN architecture, which updates node representations of a

 graph by pooling neighboring nodes at a distance of one degree.

### Learning edge representations

Our dataset does not always contain all types of information (node,

edge, and global context).

When we want to make a prediction on nodes, but our dataset only has

edge information, we showed above how to use pooling to route

information from edges to nodes, but only at the final prediction step

of the model. We can share information between nodes and edges within

the GNN layer using message passing.

We can incorporate the information from neighboring edges in the same

 way we used neighboring node information earlier, by first pooling the

edge information, transforming it with an update function, and storing

it.

However, the node and edge information stored in a graph are not

necessarily the same size or shape, so it is not immediately clear how

to combine them. One way is to learn a linear mapping from the space of

edges to the space of nodes, and vice versa. Alternatively, one may

concatenate them together before the update function.

Architecture schematic for Message Passing layer. The first step

"prepares" a message composed of information from an edge and it's

connected nodes and then "passes" the message to the node.

Which graph attributes we update and in which order we update them is

 one design decision when constructing GNNs. We could choose whether to

update node embeddings before edge embeddings, or the other way around.

This is an open area of research with a variety of solutions– for

example we could update in a 'weave' fashion

 where we have four updated representations that get combined into new

node and edge representations: node to node (linear), edge to edge

(linear), node to edge (edge layer), edge to node (node layer).

### Adding global representations

There is one flaw with the networks we have described so far: nodes

that are far away from each other in the graph may never be able to

efficiently transfer information to one another, even if we apply

message passing several times. For one node, If we have k-layers,

information will propagate at most k-steps away. This can be a problem

for situations where the prediction task depends on nodes, or groups of

nodes, that are far apart. One solution would be to have all nodes be

able to pass information to each other.

Unfortunately for large graphs, this quickly becomes computationally

expensive (although this approach, called 'virtual edges', has been used

 for small graphs such as molecules).

 or context vector. This global context vector is connected to all other

 nodes and edges in the network, and can act as a bridge between them to

 pass information, building up a representation for the graph as a

whole. This creates a richer and more complex representation of the

graph than could have otherwise been learned.

Schematic of a Graph Nets architecture leveraging global representations.

In this view all graph attributes have learned representations, so we

 can leverage them during pooling by conditioning the information of our

 attribute of interest with respect to the rest. For example, for one

node we can consider information from neighboring nodes, connected edges

 and the global information. To condition the new node embedding on all

these possible sources of information, we can simply concatenate them.

Additionally we may also map them to the same space via a linear map and

Schematic for conditioning the information of one node based

 on three other embeddings (adjacent nodes, adjacent edges, global).

This step corresponds to the node operations in the Graph Nets Layer.

GNN playground

--------------

We've described a wide range of GNN components here, but how do they

actually differ in practice? This GNN playground allows you to see how

these different components and architectures contribute to a GNN's

ability to learn a real task.

 which is composed of molecules with associated odor percepts (labels).

Predicting the relation of a molecular structure (graph) to its smell is

 a 100 year-old problem straddling chemistry, physics, neuroscience, and

 machine learning.

To simplify the problem, we consider only a single binary label per

molecule, classifying if a molecular graph smells "pungent" or not, as

labeled by a professional perfumer. We say a molecule has a "pungent"

scent if it has a strong, striking smell. For example, garlic and

We represent each molecule as a graph, where atoms are nodes

containing a one-hot encoding for its atomic identity (Carbon, Nitrogen,

 Oxygen, Fluorine) and bonds are edges containing a one-hot encoding its

 bond type (single, double, triple or aromatic).

Our general modeling template for this problem will be built up using

 sequential GNN layers, followed by a linear model with a sigmoid

activation for classification. The design space for our GNN has many

levers that can customize the model:

1. The number of GNN layers, also called the *depth*.

2. The dimensionality of each attribute when updated. The update

function is a 1-layer MLP with a relu activation function and a layer

norm for normalization of activations.

3. The aggregation function used in pooling: max, mean or sum.

4. The graph attributes that get updated, or styles of message

passing: nodes, edges and global representation. We control these via

boolean toggles (on or off). A baseline model would be a

graph-independent GNN (all message-passing off) which aggregates all

data at the end into a single global attribute. Toggling on all

message-passing functions yields a GraphNets architecture.

To better understand how a GNN is learning a task-optimized

representation of a graph, we also look at the penultimate layer

activations of the GNN. These 'graph embeddings' are the outputs of the

GNN model right before prediction. Since we are using a generalized

linear model for prediction, a linear mapping is enough to allow us to

see how we are learning representations around the decision boundary.

Since these are high dimensional vectors, we reduce them to 2D via

principal component analysis (PCA).

A perfect model would visibility separate labeled data, but since we are

 reducing dimensionality and also have imperfect models, this boundary

might be harder to see.

Play around with different model architectures to build your

intuition. For example, see if you can edit the molecule on the left to

make the model prediction increase. Do the same edits have the same

effects for different model architectures?

 Task

For this **graph level prediction task**,

 each graph is a molecule. The task is to predict whether or not a

molecule will smell pungent. In the scatter plot, each point represents a

 graph.

 Model Options

Depth

1 layer

2 layers

3 layers

4 layers

 Aggregation function

Mean

Sum

Max

Node embedding size

25

50

100

Edge embedding size

5

10

20

Global embedding size

25

50

100

 Model AUC

0.75

Reset

Model Prediction

14% pungent

Ground Truth

not pungent

Carbon

Nitrogen

Oxygen

Sulphur

 –

    Single Bond

 =

    Double Bond

 ≡

   Triple Bond

 ≡ ≡

 Aromatic Bond

 Model Prediction

 Ground Truth

Pungent

  0%  100%

epoch: 50

Edit the molecule to see how the prediction changes, or

change the model params to load a different model. Select a different

molecule in the scatter plot.

### Some empirical GNN design lessons

When exploring the architecture choices above, you might have found

some models have better performance than others. Are there some clear

GNN design choices that will give us better performance? For example, do

 deeper GNN models perform better than shallower ones? or is there a

clear choice between aggregation functions? The answers are going to

With the following interactive figure, we explore the space of GNN

architectures and the performance of this task across a few major design

 choices: Style of message passing, the dimensionality of embeddings,

number of layers, and aggregation operation type.

Each point in the scatter plot represents a model: the x axis is the

number of trainable variables, and the y axis is the performance. Hover

over a point to see the GNN architecture parameters.

var spec = "BasicArchitectures.json";

}).catch(console.error);

Scatterplot of each model's performance vs its number of

trainable variables. Hover over a point to see the GNN architecture

parameters.

The first thing to notice is that, surprisingly, a higher number of

parameters does correlate with higher performance. GNNs are a very

parameter-efficient model type: for even a small number of parameters

(3k) we can already find models with high performance.

Next, we can look at the distributions of performance aggregated

based on the dimensionality of the learned representations for different

 graph attributes.

var spec = "ArchitectureNDim.json";

}).catch(console.error);

We can notice that models with higher dimensionality tend to have

better mean and lower bound performance but the same trend is not found

for the maximum. Some of the top-performing models can be found for

smaller dimensions. Since higher dimensionality is going to also involve

 a higher number of parameters, these observations go in hand with the

previous figure.

Next we can see the breakdown of performance based on the number of GNN layers.

var spec = "ArchitectureNLayers.json";

}).catch(console.error);

 Chart of number of layers vs model performance, and

scatterplot of model performance vs number of parameters. Each point is

colored by the number of layers. Hover over a point to see the GNN

architecture parameters.

The box plot shows a similar trend, while the mean performance tends

to increase with the number of layers, the best performing models do not

 have three or four layers, but two. Furthermore, the lower bound for

performance decreases with four layers. This effect has been observed

before, GNN with a higher number of layers will broadcast information at

 a higher distance and can risk having their node representations

'diluted' from many successive iterations .

Does our dataset have a preferred aggregation operation? Our

following figure breaks down performance in terms of aggregation type.

var spec = "ArchitectureAggregation.json";

}).catch(console.error);

Chart of aggregation type vs model performance, and

scatterplot of model performance vs number of parameters. Each point is

colored by aggregation type. Hover over a point to see the GNN

architecture parameters.

Overall it appears that sum has a very slight improvement on the mean

 performance, but max or mean can give equally good models. This is

The previous explorations have given mixed messages. We can find mean

 trends where more complexity gives better performance but we can find

clear counterexamples where models with fewer parameters, number of

layers, or dimensionality perform better. One trend that is much clearer

 is about the number of attributes that are passing information to each

other.

Here we break down performance based on the style of message passing.

 On both extremes, we consider models that do not communicate between

graph entities ("none") and models that have messaging passed between

nodes, edges, and globals.

var spec = "ArchitectureMessagePassing.json";

}).catch(console.error);

Chart of message passing vs model performance, and

scatterplot of model performance vs number of parameters. Each point is

colored by message passing. Hover over a point to see the GNN

architecture parameters

Overall we see that the more graph attributes are communicating, the

better the performance of the average model. Our task is centered on

global representations, so explicitly learning this attribute also tends

 to improve performance. Our node representations also seem to be more

useful than edge representations, which makes sense since more

information is loaded in these attributes.

There are many directions you could go from here to get better

performance. We wish two highlight two general directions, one related

to more sophisticated graph algorithms and another towards the graph

itself.

Up until now, our GNN is based on a neighborhood-based pooling

operation. There are some graph concepts that are harder to express in

this way, for example a linear graph path (a connected chain of nodes).

Designing new mechanisms in which graph information can be extracted,

executed and propagated in a GNN is one current research area , , , .

One of the frontiers of GNN research is not making new models and

architectures, but "how to construct graphs", to be more precise,

imbuing graphs with additional structure or relations that can be

leveraged. As we loosely saw, the more graph attributes are

communicating the more we tend to have better models. In this particular

 case, we could consider making molecular graphs more feature rich, by

adding additional spatial relationships between nodes, adding edges that

 are not bonds, or explicit learnable relationships between subgraphs.

See more in [Other types of graphs](#Other-types-of-graphs ).

Into the Weeds

--------------

While we only described graphs with vectorized information for each

attribute, graph structures are more flexible and can accommodate other

types of information. Fortunately, the message passing framework is

flexible enough that often adapting GNNs to more complex graph

structures is about defining how information is passed and updated by

new graph attributes.

For example, we can consider multi-edge graphs or *multigraphs*,

 where a pair of nodes can share multiple types of edges, this happens

when we want to model the interactions between nodes differently based

on their type. For example with a social network, we can specify edge

types based on the type of relationships (acquaintance, friend, family).

 A GNN can be adapted by having different types of message passing steps

 for each edge type.

We can also consider nested graphs, where for example a node represents a

 graph, also called a hypernode graph.

 Nested graphs are useful for representing hierarchical information. For

 example, we can consider a network of molecules, where a node

represents a molecule and an edge is shared between two molecules if we

have a way (reaction) of transforming one to the other  .

In this case, we can learn on a nested graph by having a GNN that learns

 representations at the molecule level and another at the reaction

network level, and alternate between them during training.

Another type of graph is a hypergraph,

 where an edge can be connected to multiple nodes instead of just two.

For a given graph, we can build a hypergraph by identifying communities

of nodes and assigning a hyper-edge that is connected to all nodes in a

community.

Schematic of more complex graphs. On the left we have an

example of a multigraph with three edge types, including a directed

edge. On the right we have a three-level hierarchical graph, the

intermediate level nodes are hypernodes.

### Sampling Graphs and Batching in GNNs

A common practice for training neural networks is to update network

parameters with gradients calculated on randomized constant size (batch

size) subsets of the training data (mini-batches). This practice

presents a challenge for graphs due to the variability in the number of

nodes and edges adjacent to each other, meaning that we cannot have a

constant batch size. The main idea for batching with graphs is to create

 subgraphs that preserve essential properties of the larger graph. This

graph sampling operation is highly dependent on context and involves

sub-selecting nodes and edges from a graph. These operations might make

sense in some contexts (citation networks) and in others, these might be

 too strong of an operation (molecules, where a subgraph simply

represents a new, smaller molecule). How to sample a graph is an open

research question.

If we care about preserving structure at a neighborhood level, one way

 Each neighborhood can be considered an individual graph and a GNN can

be trained on batches of these subgraphs. The loss can be masked to only

 consider the node-set since all neighboring nodes would have incomplete

 neighborhoods.

A more efficient strategy might be to first randomly sample a single

node, expand its neighborhood to distance k, and then pick the other

node within the expanded set. These operations can be terminated once a

certain amount of nodes, edges, or subgraphs are constructed.

If the context allows, we can build constant size neighborhoods by

picking an initial node-set and then sub-sampling a constant number of

nodes (e.g randomly, or via a random walk or Metropolis algorithm).

Four different ways of sampling the same graph. Choice of

sampling strategy depends highly on context since they will generate

different distributions of graph statistics (# nodes, #edges, etc.). For

 highly connected graphs, edges can be also subsampled.

Sampling a graph is particularly relevant when a graph is large

enough that it cannot be fit in memory. Inspiring new architectures and

### Inductive biases

When building a model to solve a problem on a specific kind of data,

we want to specialize our models to leverage the characteristics of that

 data. When this is done successfully, we often see better predictive

performance, lower training time, fewer parameters and better

generalization.

When labeling on images, for example, we want to take advantage of

the fact that a dog is still a dog whether it is in the top-left or

bottom-right corner of an image. Thus, most image models use

convolutions, which are translation invariant. For text, the order of

the tokens is highly important, so recurrent neural networks process

data sequentially. Further, the presence of one token (e.g. the word

'not') can affect the meaning of the rest of a sentence, and so we need

components that can 'attend' to other parts of the text, which

transformer models like BERT and GPT-3 can do. These are some examples

of inductive biases, where we are identifying symmetries or regularities

 in the data and adding modelling components that take advantage of

these properties.

In the case of graphs, we care about how each graph component (edge,

node, global) is related to each other so we seek models that have a

relational inductive bias. A

model should preserve explicit relationships between entities (adjacency

 matrix) and preserve graph symmetries (permutation invariance). We

expect problems where the interaction between entities is important will

 benefit from a graph structure. Concretely, this means designing

transformation on sets: the order of operation on nodes or edges should

not matter and the operation should work on a variable number of

inputs.

### Comparing aggregation operations

Pooling information from neighboring nodes and edges is a critical

step in any reasonably powerful GNN architecture. Because each node has a

 variable number of neighbors, and because we want a differentiable

method of aggregating this information, we want to use a smooth

aggregation operation that is invariant to node ordering and the number

of nodes provided.

 A desirable property of an aggregation operation is that similar inputs

 provide similar aggregated outputs, and vice-versa. Some very simple

candidate permutation-invariant operations are sum, mean, and max.

Summary statistics like variance also work. All of these take a variable

 number of inputs, and provide an output that is the same, no matter the

 input ordering. Let's explore the difference between these operations.

24?Max4Mean3Sum644?44833?Max3Mean3Sum63-1?333

There is no operation that is uniformly the best choice. The mean

operation can be useful when nodes have a highly-variable number of

neighbors or you need a normalized view of the features of a local

neighborhood. The max operation can be useful when you want to highlight

 single salient features in local neighborhoods. Sum provides a balance

between these two, by providing a snapshot of the local distribution of

features, but because it is not normalized, can also highlight outliers.

 In practice, sum is commonly used.

 take into account several aggregation operations by concatenating them

and adding a scaling function that depends on the degree of connectivity

 of the entity to aggregate. Meanwhile, domain specific aggregation

operations can also be designed. One example lies with the "Tetrahedral

Chirality" aggregation operators .

### GCN as subgraph function approximators

Another way to see GCN (and MPNN) of k-layers with a 1-degree

neighbor lookup is as a neural network that operates on learned

embeddings of subgraphs of size k.

When focusing on one node, after k-layers, the updated node

representation has a limited viewpoint of all neighbors up to

k-distance, essentially a subgraph representation. Same is true for edge

 representations.

So a GCN is collecting all possible subgraphs of size k and learning

vector representations from the vantage point of one node or edge. The

number of possible subgraphs can grow combinatorially, so enumerating

these subgraphs from the beginning vs building them dynamically as in a

GCN, might be prohibitive.

### Edges and the Graph Dual

One thing to note is that edge predictions and node predictions,

while seemingly different, often reduce to the same problem: an edge

To obtain 𝐺GG's

 dual, we can convert nodes to edges (and edges to nodes). A graph and

its dual contain the same information, just expressed in a different

way. Sometimes this property makes solving problems easier in one

representation than another, like frequencies in Fourier space. In

We've talked a lot about graph convolutions and message passing, and

of course, this raises the question of how do we implement these

operations in practice? For this section, we explore some of the

properties of matrix multiplication, message passing, and its connection

 to traversing a graph.

 It should be noted that this message passing is not updating the

representation of the node features, just pooling neighboring node

 is zero. As long as we have an operation to gather values based on an

index, we should be able to just retrieve positive entries.

Additionally, this matrix multiply-free approach frees us from using

summation as an aggregation operation.

We can imagine that applying this operation multiple times allows us

to propagate information at greater distances. In this sense, matrix

multiplication is a form of traversing over a graph. This relationship

There are deeper connections on how we can view matrices as graphs to explore .

### Graph Attention Networks

 For example, when we consider the sum-aggregation of a node and its

1-degree neighboring nodes we could also consider using a weighted

sum.The challenge then is to associate weights in a permutation

invariant fashion. One approach is to consider a scalar scoring function

 In this case, the scoring function can be interpreted as a function

that measures how relevant a neighboring node is in relation to the

center node. Weights can be normalized, for example with a softmax

function to focus most of the weight on a neighbor most relevant for a

node in relation to a task. This concept is the basis of Graph Attention

 Networks (GAT)  and Set Transformers.

 Permutation invariance is preserved, because scoring works on pairs of

nodes. A common scoring function is the inner product and nodes are

often transformed before scoring into query and key vectors via a linear

 map to increase the expressivity of the scoring mechanism. Additionally

 for interpretability, the scoring weights can be used as a measure of

the importance of an edge in relation to a task.

Schematic of attention over one node with respect to it's

adjacent nodes. For each edge an interaction score is computed,

normalized and used to weight node embeddings.

Additionally, transformers can be viewed as GNNs with an attention mechanism .

 Under this view, the transformer models several elements (i.g.

character tokens) as nodes in a fully connected graph and the attention

mechanism is assigning edge embeddings to each node-pair which are used

to compute attention weights. The difference lies in the assumed pattern

 of connectivity between entities, a GNN is assuming a sparse pattern

and the Transformer is modelling all connections.

### Graph explanations and attributions

When deploying GNN in the wild we might care about model

interpretability for building credibility, debugging or scientific

discovery. The graph concepts that we care to explain vary from context

to context. For example, with molecules we might care about the presence

 or absence of particular subgraphs,

 while in a citation network we might care about the degree of

connectedness of an article. Due to the variety of graph concepts, there

 assign ranked importance values to parts of a graph that are relevant

for a task. Because realistic and challenging graph problems can be

generated synthetically, GNNs can serve as a rigorous and repeatable

testbed for evaluating attribution techniques .

Schematic of some explanability techniques on graphs.

Attributions assign ranked values to graph attributes. Rankings can be

used as a basis to extract connected subgraphs that might be relevant to

 a task.

### Generative modelling

Besides learning predictive models on graphs, we might also care

about learning a generative model for graphs. With a generative model we

 can generate new graphs by sampling from a learned distribution or by

completing a graph given a starting point. A relevant application is in

the design of new drugs, where novel molecular graphs with specific

properties are desired as candidates to treat a disease.

A key challenge with graph generative models lies in modelling the

 term can be avoided by only predicting known edges and a subset of the

edges that are not present. The graphVAE learns to model positive

patterns of connectivity and some patterns of non-connectivity in the

adjacency matrix.

Another approach is to build a graph sequentially, by starting with a

 graph and applying discrete actions such as addition or subtraction of

nodes and edges iteratively. To avoid estimating a gradient for discrete

 actions we can use a policy gradient. This has been done via an

Final thoughts

--------------

Graphs are a powerful and rich structured data type that have

strengths and challenges that are very different from those of images

and text. In this article, we have outlined some of the milestones that

researchers have come up with in building neural network based models

that process graphs. We have walked through some of the important design

 choices that must be made when using these architectures, and hopefully

 the GNN playground can give an intuition on what the empirical results

of these design choices are. The success of GNNs in recent years creates

 a great opportunity for a wide range of new problems, and we are

excited to see what the field will bring.

Understanding Convolutions on Graphs

The Graph Neural Network Model

A Deep Learning Approach to Antibiotic Discovery

Learning to simulate complex physics with graph networks

Fake News Detection on Social Media using Geometric Deep Learning

Traffic prediction with advanced Graph Neural Networks

Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in {Real-Time}

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Distributed Representations of Words and Phrases and their Compositionality

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Glove: Global Vectors for Word Representation

Learning to Represent Programs with Graphs

Deep Learning for Symbolic Mathematics

KONECT

An Information Flow Model for Conflict and Fission in Small Groups

Learning Latent Permutations with Gumbel-Sinkhorn Networks

Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs

Neural Message Passing for Quantum Chemistry

Relational inductive biases, deep learning, and graph networks

Deep Sets

Molecular graph convolutions: moving beyond fingerprints

Feature-wise transformations

Leffingwell Odor Dataset

Machine Learning for Scent: Learning Generalizable Perceptual Representations of Small Molecules

Benchmarking Graph Neural Networks

Design Space for Graph Neural Networks

Principal Neighbourhood Aggregation for Graph Nets

Graph Traversal with Tensor Functionals: A Meta-Algorithm for Scalable Learning

Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels

Representation Learning on Graphs with Jumping Knowledge Networks

Neural Execution of Graph Algorithms

Graph Theory

A nested-graph model for the representation and manipulation of complex objects

Modeling polypharmacy side effects with graph convolutional networks

Machine learning in chemical reaction space

Graphs and Hypergraphs

HyperGCN: A New Method of Training Graph Convolutional Networks on Hypergraphs

Hierarchical Message-Passing Graph Neural Networks

Little Ball of Fur

Sampling from large graphs

Metropolis Algorithms for Representative Subgraph Sampling

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

GraphSAINT: Graph Sampling Based Inductive Learning Method

How Powerful are Graph Neural Networks?

Rep the Set: Neural Networks for Learning Set Representations

Message Passing Networks for Molecules with Tetrahedral Chirality

N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules

Dual-Primal Graph Convolutional Networks

Viewing matrices & probability as graphs

Graphs and Matrices

Modern Graph Theory

Attention Is All You Need

Graph Attention Networks

Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

Transformers are Graph Neural Networks

Using Attribution to Decode Dataset Bias in Neural Network Models for Chemistry

GNNExplainer: Generating Explanations for Graph Neural Networks

Explainability Methods for Graph Convolutional Neural Networks

Evaluating Attribution for Graph Neural Networks

Variational Graph Auto-Encoders

GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models

Optimization of Molecules via Deep Reinforcement Learning

Self-Referencing Embedded Strings (SELFIES): A 100% robust molecular string representation

GraphGen: A Scalable Approach to Domain-agnostic Labeled Graph Generation


Growing Neural Cellular Automata

Alexander Mordvintsev
Ettore Randazzo
Eyvind Niklasson
Michael Levin

Abstract: This article is part of the     Differentiable Self-organizing Systems Thread,     an experimental format collecting invited short articles delving into     differentiable self-organizing systems, interspersed with critical     commentary from several experts in adjacent fields.

### Contents

[Model](#model)

[Experiments](#experiment-1)

* [Learning to Grow](#experiment-1)

* [What persists, exists](#experiment-2)

* [Learning to regenerate](#experiment-3)

* [Rotating the perceptive field](#experiment-4)

[Related Work](#related-work)

[Discussion](#discussion)

![](Growing%20Neural%20Cellular%20Automata_files/multiple-pages.svg)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Self-classifying MNIST Digits](https://distill.pub/2020/selforg/mnist/)

 Most multicellular organisms begin their life as a single egg cell - a

 single cell whose progeny reliably self-assemble into highly complex

 anatomies with many organs and tissues in precisely the same arrangement

 each time. The ability to build their own bodies is probably the most

 fundamental skill every living creature possesses. Morphogenesis (the

 process of an organism's shape development) is one of the most striking

 examples of a phenomenon called *self-organisation*. Cells, the tiny

 building blocks of bodies, communicate with their neighbors to decide the

 shape of organs and body plans, where to grow each organ, how to

 interconnect them, and when to eventually stop. Understanding the interplay

 of the emergence of complex outcomes from simple rules and

 homeostatic

 Self-regulatory feedback loops trying maintain the body in a stable state

 or preserve its correct overall morphology under external

 perturbations

 feedback loops is an active area of research

 . What is clear

 is that evolution has learned to exploit the laws of physics and computation

 to implement the highly robust morphogenetic software that runs on

 genome-encoded cellular hardware.

 This process is extremely robust to perturbations. Even when the organism is

 fully developed, some species still have the capability to repair damage - a

 process known as regeneration. Some creatures, such as salamanders, can

 fully regenerate vital organs, limbs, eyes, or even parts of the brain!

 Morphogenesis is a surprisingly adaptive process. Sometimes even a very

 atypical development process can result in a viable organism - for example,

 when an early mammalian embryo is cut in two, each half will form a complete

 individual - monozygotic twins!

 The biggest puzzle in this field is the question of how the cell collective

 knows what to build and when to stop. The sciences of genomics and stem cell

 biology are only part of the puzzle, as they explain the distribution of

 specific components in each cell, and the establishment of different types

 of cells. While we know of many genes that are *required* for the

 process of regeneration, we still do not know the algorithm that is

 *sufficient* for cells to know how to build or remodel complex organs

 to a very specific anatomical end-goal. Thus, one major lynch-pin of future

 work in biomedicine is the discovery of the process by which large-scale

 anatomy is specified within cell collectives, and how we can rewrite this

 information to have rational control of growth and form. It is also becoming

 clear that the software of life possesses numerous modules or subroutines,

 such as "build an eye here", which can be activated with simple signal

 triggers. Discovery of such subroutines and a

 mapping out of the developmental logic is a new field at the intersection of

 developmental biology and computer science. An important next step is to try

 to formulate computational models of this process, both to enrich the

 conceptual toolkit of biologists and to help translate the discoveries of

 biology into better robotics and computational technology.

 Imagine if we could design systems of the same plasticity and robustness as

 biological life: structures and machines that could grow and repair

 themselves. Such technology would transform the current efforts in

 regenerative medicine, where scientists and clinicians seek to discover the

 inputs or stimuli that could cause cells in the body to build structures on

 demand as needed. To help crack the puzzle of the morphogenetic code, and

 also exploit the insights of biology to create self-repairing systems in

 real life, we try to replicate some of the desired properties in an

 *in silico* experiment.

Model

-----

 Those in engineering disciplines and researchers often use many kinds of

 simulations incorporating local interaction, including systems of partial

 derivative equation (PDEs), particle systems, and various kinds of Cellular

 Automata (CA). We will focus on Cellular Automata models as a roadmap for

 the effort of identifying cell-level rules which give rise to complex,

 regenerative behavior of the collective. CAs typically consist of a grid of

 cells being iteratively updated, with the same set of rules being applied to

 each cell at every step. The new state of a cell depends only on the states

 of the few cells in its immediate neighborhood. Despite their apparent

 simplicity, CAs often demonstrate rich, interesting behaviours, and have a

 long history of being applied to modeling biological phenomena.

 Let's try to develop a cellular automata update rule that, starting from a

 single cell, will produce a predefined multicellular pattern on a 2D grid.

 This is our analogous toy model of organism development. To design the CA,

 we must specify the possible cell states, and their update function. Typical

 CA models represent cell states with a set of discrete values, although

 variants using vectors of continuous values exist. The use of continuous

 values has the virtue of allowing the update rule to be a differentiable

 function of the cell's neighbourhood's states. The rules that guide

 individual cell behavior based on the local environment are analogous to the

 low-level hardware specification encoded by the genome of an organism.

 Running our model for a set amount of steps from a starting configuration

 will reveal the patterning behavior that is enabled by such hardware.

 So - what is so special about differentiable update rules? They will allow

 us to use the powerful language of loss functions to express our wishes, and

 the extensive existing machinery around gradient-based numerical

 optimization to fulfill them. The art of stacking together differentiable

 functions, and optimizing their parameters to perform various tasks has a

 long history. In recent years it has flourished under various names, such as

 (Deep) Neural Networks, Deep Learning or Differentiable Programming.

A single update step of the model.

### Cell State

 We will represent each cell state as a vector of 16 real values (see the

 figure above). The first three channels represent the cell color visible to

 and an αlphaα equal to 1.0 for foreground pixels, and 0.0 for background.

 The alpha channel (αlphaα) has a special meaning: it demarcates living

 cells, those belonging to the pattern being grown. In particular, cells

 cells are "dead" or empty and have their state vector values explicitly set

 can become mature if their alpha passes the 0.1 threshold.

![](Growing%20Neural%20Cellular%20Automata_files/alive2.svg)

 Hidden channels don't have a predefined meaning, and it's up to the update

 rule to decide what to use them for. They can be interpreted as

 concentrations of some chemicals, electric potentials or some other

 signaling mechanism that are used by cells to orchestrate the growth. In

 terms of our biological analogy - all our cells share the same genome

 (update rule) and are only differentiated by the information encoded the

 chemical signalling they receive, emit, and store internally (their state

 vectors).

### Cellular Automaton rule

 Now it's time to define the update rule. Our CA runs on a regular 2D grid of

 16-dimensional vectors, essentially a 3D array of shape [height, width, 16].

 We want to apply the same operation to each cell, and the result of this

 operation can only depend on the small (3x3) neighborhood of the cell. This

 is heavily reminiscent of the convolution operation, one of the cornerstones

 of signal processing and differential programming. Convolution is a linear

 operation, but it can be combined with other per-cell operations to produce

 a complex update rule, capable of learning the desired behaviour. Our cell

 update rule can be split into the following phases, applied in order:

**Perception.** This step defines what each cell perceives of

 the environment surrounding it. We implement this via a 3x3 convolution with

 a fixed kernel. One may argue that defining this kernel is superfluous -

 after all we could simply have the cell learn the requisite perception

 kernel coefficients. Our choice of fixed operations are motivated by the

 fact that real life cells often rely only on chemical gradients to guide the

 organism development. Thus, we are using classical Sobel filters to estimate

 the partial derivatives of cell state channels in the x⃗ec{x}x⃗ and

 y⃗ec{y}y⃗ directions, forming a 2D gradient vector in each direction, for

 each state channel. We concatenate those gradients with the cells own

 rather *percepted vector,* for each cell.

def perceive(state\_grid):

sobel\_x = [[-1, 0, +1],

[-2, 0, +2],

[-1, 0, +1]]

sobel\_y = transpose(sobel\_x)

# Convolve sobel filters with states

# in x, y and channel dimension.

grad\_x = conv2d(sobel\_x, state\_grid)

grad\_y = conv2d(sobel\_y, state\_grid)

# Concatenate the cell's state channels,

# the gradients of channels in x and

# the gradient of channels in y.

perception\_grid = concat(

state\_grid, grad\_x, grad\_y, axis=2)

return perception\_grid

**Update rule.** Each cell now applies a series of operations

 to the perception vector, consisting of typical differentiable programming

 building blocks, such as 1x1-convolutions and ReLU nonlinearities, which we

 call the cell's "update rule". Recall that the update rule is learned, but

 every cell runs the same update rule. The network parametrizing this update

 rule consists of approximately 8,000 parameters. Inspired by residual neural

 networks, the update rule outputs an incremental update to the cell's state,

 which applied to the cell before the next time step. The update rule is

 designed to exhibit "do-nothing" initial behaviour - implemented by

 initializing the weights of the final convolutional layer in the update rule

 with zero. We also forego applying a ReLU to the output of the last layer of

 the update rule as the incremental updates to the cell state must

 necessarily be able to both add or subtract from the state.

def update(perception\_vector):

# The following pseudocode operates on

# a single cell's perception vector.

# Our reference implementation uses 1D

# convolutions for performance reasons.

x = dense(perception\_vector, output\_len=128)

x = relu(x)

ds = dense(x, output\_len=16, weights\_init=0.0)

return ds

**Stochastic cell update.** Typical cellular automata update

 all cells simultaneously. This implies the existence of a global clock,

 synchronizing all cells. Relying on global synchronisation is not something

 one expects from a self-organising system. We relax this requirement by

 assuming that each cell performs an update independently, waiting for a

 random time interval between updates. To model this behaviour we apply a

 random per-cell mask to update vectors, setting all update values to zero

 with some predefined probability (we use 0.5 during training). This

 operation can be also seen as an application of per-cell dropout to update

 vectors.

def stochastic\_update(state\_grid, ds\_grid):

# Zero out a random fraction of the updates.

rand\_mask = cast(random(64, 64) < 0.5, float32)

ds\_grid = ds\_grid \* rand\_mask

return state\_grid + ds\_grid

**Living cell masking.** We want to model the growth process

 that starts with a single cell, and don't want empty cells to participate in

 computations or carry any hidden state. We enforce this by explicitly

 setting all channels of empty cells to zeros. A cell is considered empty if

 there is no "mature" (alpha>0.1) cell in its 3x3 neightborhood.

def alive\_masking(state\_grid):

# Take the alpha channel as the measure of "life".

alive = max\_pool(state\_grid[:, :, 3], (3,3)) > 0.1

state\_grid = state\_grid \* cast(alive, float32)

return state\_grid

Experiment 1: Learning to Grow

------------------------------

Training regime for learning a target pattern.

 In our first experiment, we simply train the CA to achieve a target image

 after a random number of updates. This approach is quite naive and will run

 into issues. But the challenges it surfaces will help us refine future

 attempts.

 We initialize the grid with zeros, except a single seed cell in the center,

 which will have all channels except RGB

 We set RGB channels of the seed to zero because we want it to be visible

 on the white background.

 set to one. Once the grid is initialized, we iteratively apply the update

 rule. We sample a random number of CA steps from the [64, 96]

 This should be a sufficient number of steps to grow the pattern of the

 size we work with (40x40), even considering the stochastic nature of our

 update rule.

 range for each training step, as we want the pattern to be stable across a

 number of iterations. At the last step we apply pixel-wise L2 loss between

 RGBA channels in the grid and the target pattern. This loss can be

 differentiably optimized

 We observed training instabilities, that were manifesting themselves as

 sudden jumps of the loss value in the later stages of the training. We

 managed to mitigate them by applying per-variable L2 normalization to

 parameter gradients. This may have the effect similar to the weight

 normalization . Other training

 parameters are available in the accompanying source code.

 with respect to the update rule parameters by backpropagation-through-time,

 the standard method of training recurrent neural networks.

 Once the optimisation converges, we can run simulations to see how our

 learned CAs grow patterns starting from the seed cell. Let's see what

 happens when we run it for longer than the number of steps used during

 training. The animation below shows the behaviour of a few different models,

 trained to generate different emoji patterns.

 Your browser does not support the video tag.

 Many of the patterns exhibit instability for longer time periods.

 We can see that different training runs can lead to models with drastically

 different long term behaviours. Some tend to die out, some don't seem to

 know how to stop growing, but some happen to be almost stable! How can we

 steer the training towards producing persistent patterns all the time?

Experiment 2: What persists, exists

-----------------------------------

 One way of understanding why the previous experiment was unstable is to draw

 a parallel to dynamical systems. We can consider every cell to be a

 dynamical system, with each cell sharing the same dynamics, and all cells

 being locally coupled amongst themselves. When we train our cell update

 model we are adjusting these dynamics. Our goal is to find dynamics that

 satisfy a number of properties. Initially, we wanted the system to evolve

 from the seed pattern to the target pattern - a trajectory which we achieved

 in Experiment 1. Now, we want to avoid the instability we observed - which

 in our dynamical system metaphor consists of making the target pattern an

 attractor.

 One strategy to achieve this is letting the CA iterate for much longer time

 and periodically applying the loss against the target, training the system

 by backpropagation through these longer time intervals. Intuitively we claim

 that with longer time intervals and several applications of loss, the model

 is more likely to create an attractor for the target shape, as we

 iteratively mold the dynamics to return to the target pattern from wherever

 the system has decided to venture. However, longer time periods

 substantially increase the training time and more importantly, the memory

 requirements, given that the entire episode's intermediate activations must

 be stored in memory for a backwards-pass to occur.

 Instead, we propose a "sample pool" based strategy to a similar effect. We

 define a pool of seed states to start the iterations from, initially filled

 with the single black pixel seed state. We then sample a batch from this

 pool which we use in our training step. To prevent the equivalent of

 "catastrophic forgetting" we replace one sample in this batch with the

 original, single-pixel seed state. After concluding the training step , we

 replace samples in the pool that were sampled for the batch with the output

 states from the training step over this batch. The animation below shows a

 random sample of the entries in the pool every 20 training steps.

def pool\_training():

# Set alpha and hidden channels to (1.0).

seed = zeros(64, 64, 16)

seed[64//2, 64//2, 3:] = 1.0

target = targets['lizard']

pool = [seed] \* 1024

for i in range(training\_iterations):

idxs, batch = pool.sample(32)

# Sort by loss, descending.

batch = sort\_desc(batch, loss(batch))

# Replace the highest-loss sample with the seed.

batch[0] = seed

# Perform training.

outputs, loss = train(batch, target)

# Place outputs back in the pool.

pool[idxs] = outputs

 Your browser does not support the video tag.

 A random sample of the patterns in the pool during training, sampled

 every 20 training steps.

 Early on in the training process, the random dynamics in the system allow

 the model to end up in various incomplete and incorrect states. As these

 states are sampled from the pool, we refine the dynamics to be able to

 recover from such states. Finally, as the model becomes more robust at going

 from a seed state to the target state, the samples in the pool reflect this

 and are more likely to be very close to the target pattern, allowing the

 training to refine these almost completed patterns further.

 Essentially, we use the previous final states as new starting points to

 force our CA to learn how to persist or even improve an already formed

 pattern, in addition to being able to grow it from a seed. This makes it

 possible to add a periodical loss for significantly longer time intervals

 than otherwise possible, encouraging the generation of an attractor as the

 target shape in our coupled system. We also noticed that reseeding the

 highest loss sample in the batch, instead of a random one, makes training

 more stable at the initial stages, as it helps to clean up the low quality

 states from the pool.

 Here is what a typical training progress of a CA rule looks like. The cell

 rule learns to stabilize the pattern in parallel to refining its features.

 Your browser does not support the video tag.

 CA behaviour at training steps 100, 500, 1000, 4000.

Experiment 3: Learning to regenerate

------------------------------------

 In addition to being able to grow their own bodies, living creatures are

 great at maintaining them. Not only does worn out skin get replaced with new

 skin, but very heavy damage to complex vital organs can be regenerated in

 some species. Is there a chance that some of the models we trained above

 have regenerative capabilities?

 Your browser does not support the video tag.

 Patterns exhibit some regenerative properties upon being damaged, but

 not full re-growth.

 The animation above shows three different models trained using the same

 settings. We let each of the models develop a pattern over 100 steps, then

 damage the final state in five different ways: by removing different halves

 of the formed pattern, and by cutting out a square from the center. Once

 again, we see that these models show quite different out-of-training mode

 behaviour. For example "the lizard" develops quite strong regenerative

 capabilities, without being explicitly trained for it!

 Since we trained our coupled system of cells to generate an attractor

 towards a target shape from a single cell, it was likely that these systems,

 once damaged, would generalize towards non-self-destructive reactions.

 That's because the systems were trained to grow, stabilize, and never

 entirely self-destruct. Some of these systems might naturally gravitate

 towards regenerative capabilities, but nothing stops them from developing

 different behaviors such as explosive mitoses (uncontrolled growth),

 unresponsiveness to damage (overstabilization), or even self destruction,

 especially for the more severe types of damage.

 If we want our model to show more consistent and accurate regenerative

 capabilities, we can try to increase the basin of attraction for our target

 pattern - increase the space of cell configurations that naturally gravitate

 towards our target shape. We will do this by damaging a few pool-sampled

 states before each training step. The system now has to be capable of

 regenerating from states damaged by randomly placed erasing circles. Our

 hope is that this will generalize to regenerational capabilities from

 various types of damage.

 Your browser does not support the video tag.

 Damaging samples in the pool encourages the learning of robust

 regenerative qualities. Row 1 are samples from the pool, Row 2 are their

 respective states after iterating the model.

 The animation above shows training progress, which includes sample damage.

 We sample 8 states from the pool. Then we replace the highest-loss sample

 (top-left-most in the above) with the seed state, and damage the three

 lowest-loss (top-right-most) states by setting a random circular region

 within the pattern to zeros. The bottom row shows states after iteration

 from the respective top-most starting state. As in Experiment 2, the

 resulting states get injected back into the pool.

 Your browser does not support the video tag.

 Patterns exposed to damage during training exhibit astounding

 regenerative capabilities.

 As we can see from the animation above, models that were exposed to damage

 during training are much more robust, including to types of damage not

 experienced in the training process (for instance rectangular damage as

 above).

Experiment 4: Rotating the perceptive field

-------------------------------------------

 As previously described, we model the cell's perception of its neighbouring

 cells by estimating the gradients of state channels in x⃗ec{x}x⃗ and

 y⃗ec{y}y⃗ using Sobel filters. A convenient analogy is that each agent has

 two sensors (chemosensory receptors, for instance) pointing in orthogonal

 directions that can sense the gradients in the concentration of certain

 chemicals along the axis of the sensor. What happens if we rotate those

 sensors? We can do this by rotating the Sobel kernels.

 -\sin 	heta \ \sin 	heta & \cos 	heta \end{bmatrix} \* egin{bmatrix}

 This simple modification of the perceptive field produces rotated versions

 of the pattern for an angle of choosing without retraining as seen below.

![](Growing%20Neural%20Cellular%20Automata_files/rotation.png)

 Rotating the axis along which the perception step computes gradients

 brings about rotated versions of the pattern.

 In a perfect world, not quantized by individual cells in a pixel-lattice,

 this would not be too surprising, as, after all, one would expect the

 angle - a simple change of frame of reference. However, it is important to

 note that things are not as simple in a pixel based model. Rotating pixel

 based graphics involves computing a mapping that's not necessarily bijective

 and classically involves interpolating between pixels to achieve the desired

 result. This is because a single pixel, when rotated, will now likely

 overlap several pixels. The successful growth of patterns as above suggests

 a certain robustness to the underlying conditions outside of those

 experienced during training.

Related Work

------------

### CA and PDEs

 There exists an extensive body of literature that describes the various

 flavours of cellular automata and PDE systems, and their applications to

 modelling physical, biological or even social systems. Although it would be

 impossible to present a just overview of this field in a few lines, we will

 describe some prominent examples that inspired this work. Alan Turing

 introduced his famous Turing patterns back in 1952

 , suggesting how

 reaction-diffusion systems can be a valid model for chemical behaviors

 during morphogenesis. A particularly inspiring reaction-diffusion model that

 stood the test of time is the Gray-Scott model

 , which shows an extreme variety of

 behaviors controlled by just a few variables.

 Ever since von Neumann introduced CAs

  as models for self-replication they

 have captivated researchers' minds, who observed extremely complex

 behaviours emerging from very simple rules. Likewise, the a broader audience

 outside of academia were seduced by CA's life-like behaviours thanks to

 Conway's Game of Life . Perhaps

 motivated in part by the proof that something as simple as the Rule 110 is

 Turing complete, Wolfram's "*A New Kind of Science"*

 asks for a paradigm shift centered

 around the extensive usage of elementary computer programs such as CA as

 tools for understanding the world.

 More recently, several researchers generalized Conway's Game of life to work

 on more continuous domains. We were particularly inspired by Rafler's

 SmoothLife  and Chan's Lenia

 , the latter of

 which also discovers and classifies entire species of "lifeforms".

 A number of researchers have used evolutionary algorithms to find CA rules

 that reproduce predefined simple patterns

 .

 For example, J. Miller  proposed an

 experiment similar to ours, using evolutionary algorithms to design a CA

 rule that could build and regenerate the French flag, starting from a seed

 cell.

### Neural Networks and Self-Organisation

 The close relation between Convolutional Neural Networks and Cellular

 Automata has already been observed by a number of researchers

 . The

 connection is so strong it allowed us to build Neural CA models using

 components readily available in popular ML frameworks. Thus, using a

 different jargon, our Neural CA could potentially be named "Recurrent

 Residual Convolutional Networks with 'per-pixel' Dropout".

 The Neural GPU

  offers

 a computational architecture very similar to ours, but applied in the

 context of learning multiplication and a sorting algorithm.

 Looking more broadly, we think that the concept of self-organisation is

 finding its way into mainstream machine learning with popularisation of

 Graph Neural Network  models.

 Typically, GNNs run a repeated computation across vertices of a (possibly

 dynamic) graph. Vertices communicate locally through graph edges, and

 aggregate global information required to perform the task over multiple

 rounds of message exchanges, just as atoms can be thought of as

 communicating with each other to produce the emergent properties of a

 molecule , or even points of a point

 cloud talk to their neighbors to figure out their global shape

 .

 Self-organization also appeared in fascinating contemporary work using more

 traditional dynamic graph networks, where the authors evolved

 Self-Assembling Agents to solve a variety of virtual tasks

 .

### Swarm Robotics

 One of the most remarkable demonstrations of the power of self-organisation

 is when it is applied to swarm modeling. Back in 1987, Reynolds' Boids

  simulated the flocking behaviour of birds with

 just a tiny set of handcrafted rules. Nowadays, we can embed tiny robots

 with programs and test their collective behavior on physical agents, as

 demonstrated by work such as Mergeable Nervous Systems

  and Kilobots

 . To the best of our knowledge, programs

 embedded into swarm robots are currently designed by humans. We hope our

 work can serve as an inspiration for the field and encourage the design of

 collective behaviors through differentiable modeling.

Discussion

----------

### Embryogenetic Modeling

 Your browser does not support the video tag.

 Regeneration-capable 2-headed planarian, the creature that inspired this

 work

 This article describes a toy embryogenesis and regeneration model. This is a

 major direction for future work, with many applications in biology and

 beyond. In addition to the implications for understanding the evolution and

 control of regeneration, and harnessing this understanding for biomedical

 repair, there is the field of bioengineering. As the field transitions from

 synthetic biology of single cell collectives to a true synthetic morphology

 of novel living machines , it

 will be essential to develop strategies for programming system-level

 capabilities, such as anatomical homeostasis (regenerative repair). It has

 long been known that regenerative organisms can restore a specific

 anatomical pattern; however, more recently it's been found that the target

 morphology is not hard coded by the DNA, but is maintained by a

 physiological circuit that stores a setpoint for this anatomical homeostasis

 . Techniques are

 now available for re-writing this setpoint, resulting for example

  in 2-headed flatworms

 that, when cut into pieces in plain water (with no more manipulations)

 result in subsequent generations of 2-headed regenerated worms (as shown

 above). It is essential to begin to develop models of the computational

 processes that store the system-level target state for swarm behavior

 , so that efficient strategies can be developed for rationally editing this

 information structure, resulting in desired large-scale outcomes (thus

 defeating the inverse problem that holds back regenerative medicine and many

 other advances).

### Engineering and machine learning

 The models described in this article run on the powerful GPU of a modern

 computer or a smartphone. Yet, let's speculate about what a "more physical"

 implementation of such a system could look like. We can imagine it as a grid

 of tiny independent computers, simulating individual cells. Each of those

 computers would require approximately 10Kb of ROM to store the "cell

 genome": neural network weights and the control code, and about 256 bytes of

 RAM for the cell state and intermediate activations. The cells must be able

 to communicate their 16-value state vectors to neighbors. Each cell would

 also require an RGB-diode to display the color of the pixel it represents. A

 single cell update would require about 10k multiply-add operations and does

 not have to be synchronised across the grid. We propose that cells might

 wait for random time intervals between updates. The system described above

 is uniform and decentralised. Yet, our method provides a way to program it

 to reach the predefined global state, and recover this state in case of

 multi-element failures and restarts. We therefore conjecture this kind of

 modeling may be used for designing reliable, self-organising agents. On the

 more theoretical machine learning front, we show an instance of a

 decentralized model able to accomplish remarkably complex tasks. We believe

 this direction to be opposite to the more traditional global modeling used

 in the majority of contemporary work in the deep learning field, and we hope

 this work to be an inspiration to explore more decentralized learning

 modeling.

![](Growing%20Neural%20Cellular%20Automata_files/multiple-pages.svg)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Self-classifying MNIST Digits](https://distill.pub/2020/selforg/mnist/)

Top-down models in biology: explanation and control of complex living systems above the molecular level

Re-membering
 the body: applications of computational neuroscience to the top-down
control of regeneration of limbs and other complex organs

Transmembrane voltage potential controls embryonic eye patterning in Xenopus laevis

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

The chemical basis of morphogenesis

Complex Patterns in a Simple System

Theory of Self-Reproducing Automata

MATHEMATICAL GAMES

A New Kind of Science

Generalization of Conway's "Game of Life" to a continuous domain - SmoothLife

Lenia: Biology of Artificial Life

Intrinsically Motivated Exploration for Automated Discovery of Patterns in Morphogenetic Systems

Evolving Self-organizing Cellular Automata Based on Neural Network Genotypes

CA-NEAT: Evolved Compositional Pattern Producing Networks for Cellular Automata Morphogenesis and Replication

Evolving a Self-Repairing, Self-Regulating, French Flag Organism

Learning Cellular Automaton Dynamics with Neural Networks

Cellular automata as convolutional neural networks

Neural GPUs Learn Algorithms

Improving the Neural GPU Architecture for Algorithm Learning

A Comprehensive Survey on Graph Neural Networks

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Dynamic Graph CNN for Learning on Point Clouds

Learning to Control Self- Assembling Morphologies: A Study of Generalization via Modularity

Flocks, Herds and Schools: A Distributed Behavioral Model

Mergeable nervous systems for robots

Kilobot: A low cost scalable robot system for collective behaviors

What Bodies Think About: Bioelectric Computation Outside the Nervous System

A scalable pipeline for designing reconfigurable organisms

Perspective: The promise of multi-cellular engineered living systems

Physiological inputs regulate species-specific anatomy during embryogenesis and regeneration

Long-range neural and gap junction protein-mediated cues control polarity during planarian regeneration

Long-Term, Stochastic Editing of Regenerative Anatomy via Targeting Endogenous Bioelectric Gradients

Pattern Regeneration in Coupled Networks

Bioelectrical
 control of positional information in development and regeneration: A
review of conceptual and computational advances

Modeling Cell Migration in a Simulated Bioelectrical Signaling Network for Anatomical Regeneration

Investigating the effects of noise on a cell-to-cell communication mechanism for structure regeneration

Social Intelligence

Inceptionism: Going deeper into neural networks