Understanding Convolutions on Graphs

Ameya Daigavane
Balaraman Ravindran
Gaurav Aggarwal

Abstract: This article is one of two Distill publications about graph neural networks.         Take a look at         A Gentle Introduction to Graph Neural Networks          for a companion view on many things graph and neural network related.

### Contents

[Introduction](#introduction)

[The Challenges of Computation on Graphs](#challenges)

* [Lack of Consistent Structure](#lack-of-consistent-structure)

* [Node-Order Equivariance](#node-order)

* [Scalability](#scalability)

[Problem Setting and Notation](#problem-and-notation)

[Extending Convolutions to Graphs](#extending)

[Polynomial Filters on Graphs](#polynomial-filters)

[Modern Graph Neural Networks](#modern-gnns)

[Interactive Graph Neural Networks](#interactive)

[From Local to Global Convolutions](#from-local-to-global)

* [Spectral Convolutions](#spectral)

* [Global Propagation via Graph Embeddings](#graph-embeddings)

[Learning GNN Parameters](#learning)

[Conclusions and Further Reading](#further-reading)

* [GNNs in Practice](#practical-techniques)

* [Different Kinds of Graphs](#different-kinds-of-graphs)

* [Pooling](#pooling)

[Supplementary Material](#supplementary)

* [Reproducing Experiments](#experiments-notebooks)

* [Recreating Visualizations](#visualizations-notebooks)

*This article is one of two Distill publications about graph neural networks.

 Take a look at

 for a companion view on many things graph and neural network related.*

 Many systems and interactions - social networks, molecules,

organizations, citations, physical models, transactions - can be

represented quite naturally as graphs.

 How can we reason about and make predictions within these

systems?

 One idea is to look at tools that have worked well in other

domains: neural networks have shown immense predictive power in a

variety of learning tasks.

 However, neural networks have been traditionally used to operate

 on fixed-size and/or regular-structured inputs (such as sentences,

images and video).

 This makes them unable to elegantly process graph-structured

data.

 By extracting and utilizing features from the underlying graph,

 GNNs can make more informed predictions about entities in these interactions,

 as compared to models that consider individual entities in isolation.

 GNNs are not the only tools available to model graph-structured data:

 graph kernels

 and random-walk methods

 were some of the most popular ones.

 Today, however, GNNs have largely replaced these techniques

 because of their inherent flexibility to model the underlying systems

 better.

 In this article, we will illustrate

 the challenges of computing over graphs,

 describe the origin and design of graph neural networks,

 and explore the most popular GNN variants in recent times.

 Particularly, we will see that many of these variants

 are composed of similar building blocks.

 First, let's discuss some of the complications that graphs come with.

 The Challenges of Computation on Graphs

-----------------------------------------

###

 Lack of Consistent Structure

 Consider the task of predicting whether a given chemical molecule is toxic  :

**Left:** A non-toxic 1,2,6-trigalloyl-glucose molecule.

**Right:** A toxic caramboxin molecule.

 Looking at a few examples, the following issues quickly become apparent:

* Molecules may have different numbers of atoms.

* The atoms in a molecule may be of different types.

* Each of these atoms may have different number of connections.

* These connections can have different strengths.

 Representing graphs in a format that can be computed over is non-trivial,

###

 Node-Order Equivariance

 Representing the graph as one vector requires us to fix an order on the nodes.

 But what do we do when the nodes have no inherent order?

 **Above:**

 As a result, we would like our algorithms to be node-order equivariant:

 they should not depend on the ordering of the nodes of the graph.

 If we permute the nodes in some way, the resulting representations of

###

 Scalability

 Operating on data this large is not easy.

 Luckily, most naturally occuring graphs are 'sparse':

 they tend to have their number of edges linear in their number of vertices.

 We will see that this allows the use of clever methods

 to efficiently compute representations of nodes within the graph.

 in comparison to the size of the graphs they operate on.

 Problem Setting and Notation

------------------------------

 There are many useful problems that can be formulated over graphs:

* **Node Classification:** Classifying individual nodes.

* **Graph Classification:** Classifying entire graphs.

* **Node Clustering:** Grouping together similar nodes based on connectivity.

* **Link Prediction:** Predicting missing links.

* **Influence Maximization:** Identifying influential nodes.

 Examples of problems that can be defined over graphs.

 This list is not exhaustive!

 Generally, however, GNNs compute node representations in an iterative process.

 For example, the 'node features' for a pixel in a color image

 would be the red, green and blue channel (RGB) values at that pixel.

 These kinds of graphs are called 'homogeneous'.

 Many of the same ideas we will see here

 apply to other kinds of graphs:

 Sometimes we will need to denote a graph property by a matrix MMM,

 Extending Convolutions to Graphs

----------------------------------

 they depend on the absolute positions of pixels.

 where the neighbourhood structure differs from node to node.

 The curious reader may wonder if performing some sort of padding and ordering

 This has been attempted with some success ,

 but the techniques we will look at here are more general and powerful.

image/svg+xml

Convolution in CNNs

1

7

6

7

1

6

4

5

6

3

 Convolutions in CNNs are inherently localized.

image/svg+xml

4

6

1

2

5

4

1

7

3

6

1

7

6

Localized Convolution in GNNs

2

 GNNs can perform localized convolutions mimicking CNNs.

 Hover over a node to see its immediate neighbourhood highlighted on the left.

 The structure of this neighbourhood changes from node to node.

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/neighbourhoods-for-cnns-and-gnns.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 much like how CNNs compute localized filters over neighbouring pixels.

 Finally, we will discuss alternative methods

 Polynomial Filters on Graphs

------------------------------

###

 The Graph Laplacian

 Given a graph GGG, let us fix an arbitrary ordering of the nnn nodes of GGG.

Dv=∑uAvu.

 D\_v = \sum\_u A\_{vu}.

 Dv=u∑Avu.

 The degree of node vvv is the number of edges incident at vvv.

 in the matrix AAA. We will use this notation throughout this section.

 Then, the graph Laplacian LLL is the square n×nn \times nn×n matrix defined as:

 L=D−A.

 L = D - A.

 L=D−A.

![](Understanding%20Convolutions%20on%20Graphs_files/laplacian.svg)

 Zeros in LLL are not displayed above.

 The graph Laplacian gets its name from being the discrete analog of the

 [Laplacian operator](https://mathworld.wolfram.com/Laplacian.html)

 from calculus.

 Although it encodes precisely the same information as the adjacency matrix AAA

 ,

 the graph Laplacian has many interesting properties of its own.

 The graph Laplacian shows up in many mathematical problems involving graphs:

 [spectral clustering](https://arxiv.org/abs/<PHONE>),

 and

 We will see some of these properties

 in [a later section](#spectral),

 but will instead point readers to

 for greater insight into the graph Laplacian.

###

 Polynomials of the Laplacian

 Now that we have understood what the graph Laplacian is,

 we can build polynomials  of the form:

 pw(L)=w0In+w1L+w2L2+…+wdLd=∑i=0dwiLi.

 pw(L)=w0In+w1L+w2L2+…+wdLd=i=0∑dwiLi.

 Each polynomial of this form can alternately be represented by

 its vector of coefficients w=[w0,…,wd]w = [w\_0, \ldots, w\_d]w=[w0,…,wd].

 These polynomials can be thought of as the equivalent of 'filters' in CNNs,

 and the coefficients www as the weights of the 'filters'.

 each of the xvx\_vxv for v∈Vv \in Vv∈V is just a real number.

 Using the previously chosen ordering of the nodes,

 we can stack all of the node features xvx\_vxv

 to get a vector x∈Rnx \in \mathbb{R}^nx∈Rn.

 Once we have constructed the feature vector xxx,

 we can define its convolution with a polynomial filter pwp\_wpw as:

 x′=pw(L) x

 x' = p\_w(L) \ x

 x′=pw(L) x

 To understand how the coefficients www affect the convolution,

 let us begin by considering the 'simplest' polynomial:

 when w0=1w\_0 = 1w0=1 and all of the other coefficients are 000.

 In this case, x′x'x′ is just xxx:

 x′=pw(L) x=∑i=0dwiLix=w0Inx=x.

 x' = p\_w(L) \ x = \sum\_{i = 0}^d w\_i L^ix = w\_0 I\_n x = x.

 x′=pw(L) x=i=0∑dwiLix=w0Inx=x.

 Now, if we increase the degree, and consider the case where

 instead w1=1w\_1 = 1w1=1 and and all of the other coefficients are 000.

 Then, x′x'x′ is just LxLxLx, and so:

 xv′=(Lx)v=Lvx=∑u∈GLvuxu=∑u∈G(Dvu−Avu)xu=Dv xv−∑u∈N(v)xu

 \begin{aligned}

 x'\_v = (Lx)\_v &= L\_v x \\

 &= \sum\_{u \in G} L\_{vu} x\_u \\

 &= \sum\_{u \in G} (D\_{vu} - A\_{vu}) x\_u \\

 &= D\_v \ x\_v - \sum\_{u \in \mathcal{N}(v)} x\_u

 \end{aligned}

 xv′=(Lx)v=Lvx=u∈G∑Lvuxu=u∈G∑(Dvu−Avu)xu=Dv xv−u∈N(v)∑xu

 We see that the features at each node vvv are combined

 with the features of its immediate neighbours u∈N(v)u \in \mathcal{N}(v)u∈N(v).

 For readers familiar with

 this is the exact same idea. When xxx is an image,

 At this point, a natural question to ask is:

 Indeed, it is not too hard to show that:

 This is Lemma 5.2 from .

distG(v,u)>i⟹Lvui=0.

 \text{dist}\_G(v, u) > i \quad \Longrightarrow \quad L\_{vu}^i = 0.

 distG(v,u)>i⟹Lvui=0.

 \begin{aligned}

 x'\_v = (p\_w(L)x)\_v &= (p\_w(L))\_v x \\

 &= \sum\_{i = 0}^d w\_i L\_v^i x \\

 &= \sum\_{i = 0}^d w\_i \sum\_{u \in G} L\_{vu}^i x\_u \\

 \end{aligned}

 the resulting pixel in x′x'x′.

 Note that even after adjusting for position,

 #button-container {

 text-align: center;

 }

Reset Grid

Randomize Grid

−2−1012Color Scale

Input Grid

x∈{0,1}25x \in \{0, 1\}^{25}x∈{0,1}25

pw(L)p\_w(L)pw(L)

Convolutional Kernel at Highlighted Pixel

Output Grid

x′∈R25x' \in \mathbb{R}^{25}x′∈R25

 #poly-main-div {

 position: relative;

 height: 220px;

 }

 #poly-svg {

 position: absolute;

 left: 0px;

 top: 0px;

 }

 #figcaptions {

 position: relative;

 }

 #figcaptions figcaption {

 position: absolute;

 }

 #orig-nat-caption-1 {

 left: 170px;

 top: 140px;

 }

 #orig-nat-caption-2 {

 left: 170px;

 top: 160px;

 }

 #poly-caption {

 left: 460px;

 top: 90px;

 }

 #kernel-caption {

 left: 360px;

 top: 230px;

 }

 #upd-nat-caption-1 {

 left: 725px;

 top: 140px;

 }

 #upd-nat-caption-2 {

 left: 740px;

 top: 160px;

 }

 #poly\_conv\_sliders form {

 margin-left: 5%;

 margin-right: 5%;

 }

 #poly\_conv\_sliders input {

 width: 80%;

 }

 #poly\_conv\_sliders output {

 width: 3em;

 margin-left: 0 !important;

 }

 #poly\_conv\_sliders {

 display: flex;

 width: 60%;

 margin: 0 auto;

 text-align: center;

 align-items: center;

 justify-content: center;

 }

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

Convolve

pw(L)=∑i=02wiLi=1I + 0.1L + 0L2.

 p\_w(L)

 = \sum\_{i = 0}^2 w\_i L^i

 = 1I \ + \ 0.1L \ + \ 0L^2.

pw(L)=i=0∑2wiLi=1I + 0.1L + 0L2.

w0{w\_{0}}w0

1w1{w\_{1}}w1

0.1w2{w\_{2}}w2

0

Choice of Laplacian

Unnormalized L\text{Unnormalized} \ LUnnormalized L

Normalized L~\text{Normalized} \ \tilde{L}Normalized L~

 #button-container {

 text-align: center;

 }

Reset Coefficients

poly\_input\_slider\_watch = undefined

highlight\_selected\_cell = ƒ()

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Hover over a pixel in the input grid (left, representing xxx)

 to highlight it and see the equivalent convolutional kernel

 for that pixel under the arrow.

 The result x′x'x′ of the convolution is shown on the right:

 note that different convolutional kernels are applied at different pixels,

 depending on their location.

 Use the sliders at the bottom to change the coefficients www.

 To reset all coefficients www to 000, press 'Reset Coefficients.'

###

 ChebNet

pw(L)=∑i=1dwiTi(L~)

 p\_w(L) = \sum\_{i = 1}^d w\_i T\_i(\tilde{L})

 pw(L)=i=1∑dwiTi(L~)

 where TiT\_iTi is the degree-iii

L~=2Lλmax(L)−In.

 \tilde{L} = \frac{2L}{\lambda\_{\max}(L)} - I\_n.

 L~=λmax(L)2L−In.

 What is the motivation behind these choices?

 This prevents the entries of powers of L~\tilde{L}L~ from blowing up.

 in order to show the result x′x'x′ on the same color scale.

 We won't talk about this in more depth here,

 but will advise interested readers to take a look at  as a definitive resource.

###

 Polynomial Filters are Node-Order Equivariant

 Clearly, this sum does not depend on the order of the neighbours.

 A similar proof follows for higher degree polynomials:

 the entries in the powers of LLL are equivariant to the ordering of the nodes.

**Details for the Interested Reader**

 As above, let's assume an arbitrary node-order over the nnn nodes of our graph.

 We can represent any permutation by a

 [permutation matrix](https://en.wikipedia.org/wiki/Permutation_matrix) PPP.

 PPP will always be an orthogonal 0−10-10−1 matrix:

 PPT=PTP=In.

 PP^T = P^TP = I\_n.

 PPT=PTP=In.

 f(Px)=Pf(x).

 f(Px) = P f(x).

 f(Px)=Pf(x).

 When switching to the new node-order using the permutation PPP,

 the quantities below transform in the following way:

 x→PxL→PLPTLi→PLiPT

 \begin{aligned}

 x &\to Px \\

 L &\to PLP^T \\

 L^i &\to PL^iP^T

 \end{aligned}

 xLLi→Px→PLPT→PLiPT

 and so, for the case of polynomial filters, we can see that:

 f(Px)=∑i=0dwi(PLiPT)(Px)=P∑i=0dwiLix′=Pf(x).

 \begin{aligned}

 f(Px) & = \sum\_{i = 0}^d w\_i (PL^iP^T) (Px) \\

 & = P \sum\_{i = 0}^d w\_i L^i x' \\

 & = P f(x).

 \end{aligned}

 f(Px)=i=0∑dwi(PLiPT)(Px)=Pi=0∑dwiLix′=Pf(x).

 as needed.

###

 Embedding Computation

 We now describe how we can build a graph neural network

 by stacking ChebNet (or any polynomial filter) layers

 one after the other with non-linearities,

 much like a standard CNN.

 In particular, if we have KKK different polynomial filter layers,

 the kthk^{\text{th}}kth of which has its own learnable weights w(k)w^{(k)}w(k),

 we would perform the following computation:

 Start with the original features.

h(0)=x\quad {\color{#FE6100} h^{(0)}} = xh(0)=x

 Then iterate, for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

p(k)=pw(k)(L)g(k)=p(k)×h(k−1)h(k)=σ(g(k))

 \begin{aligned}

 \quad {p^{(k)}} &= p\_{\color{#4D9DB5} w^{(k)}}(L) \\

 \\

 \quad {g^{(k)}} &= p^{(k)} \times {\color{#FE6100} h^{(k - 1)}} \\

 \\

 \quad {\color{#FE6100} h^{(k)}} &= \sigma \left({g^{(k)}} \right)

 \end{aligned}

 p(k)g(k)h(k)=pw(k)(L)=p(k)×h(k−1)=σ(g(k))

 Color Codes:

 * Computed node embeddings.

* Learnable parameters.

 #spectral-conv div {

 margin: 0 auto 10px auto;

 display: block;

 }

 #cheb-conv {

 width: 800px;

 height: 310px;

 display: block;

 margin-top: 0;

 margin-bottom: 0;

 margin-left: auto;

 margin-right: auto;

 position: relative;

 }

 span.green {

 color: #177245;

 }

 span.blue {

 color: #0047AB;

 }

 /\* Legend \*/

 .legend {

 list-style: none;

 margin-top: 5px;

 }

 .legend li {

 margin-bottom: 5px;

 }

 .legend span.box {

 border: 1px solid #ccc;

 float: left;

 width: 12px;

 height: 12px;

 margin: 2px;

 margin-right: 5px;

 }

 .legend .main { background-color: #FE6100; }

 .legend .param { background-color: #4D9DB5; }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/updated-chebnet-equations.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Note that these networks

 reuse the same filter weights across different nodes,

 exactly mimicking weight-sharing in Convolutional Neural Networks (CNNs)

 which reuse weights for convolutional filters across a grid.

 Modern Graph Neural Networks

------------------------------

 ChebNet was a breakthrough in learning localized filters over graphs,

 focussing on a particular vertex vvv:

(Lx)v=Lvx=∑u∈GLvuxu=∑u∈G(Dvu−Avu)xu=Dv xv−∑u∈N(v)xu

 \begin{aligned}

 (Lx)\_v &= L\_v x \\

 &= \sum\_{u \in G} L\_{vu} x\_u \\

 &= \sum\_{u \in G} (D\_{vu} - A\_{vu}) x\_u \\

 &= D\_v \ x\_v - \sum\_{u \in \mathcal{N}(v)} x\_u

 \end{aligned}

 (Lx)v=Lvx=u∈G∑Lvuxu=u∈G∑(Dvu−Avu)xu=Dv xv−u∈N(v)∑xu

 As we noted before, this is a 111-hop localized convolution.

 But more importantly, we can think of this convolution as arising of two steps:

* Aggregating over immediate neighbour features xux\_uxu.

* Combining with the node's own feature xvx\_vxv.

**Key Idea:**

 What if we consider different kinds of 'aggregation' and 'combination' steps,

 beyond what are possible using polynomial filters?

 By ensuring that the aggregation is node-order equivariant,

 the overall convolution becomes node-order equivariant.

 after each step, every node receives some 'information' from its neighbours.

###

 Embedding Computation

 Message-passing forms the backbone of many GNN architectures today.

 We describe the most popular ones in depth below:

* Graph Convolutional Networks (GCN)

* Graph Attention Networks (GAT)

* Graph Sample and Aggregate (GraphSAGE)

* Graph Isomorphism Network (GIN)

* GCN

* GAT

* GraphSAGE

* GIN

hv(0)\color{#FE6100} h\_v^{(0)}hv(0)

===

xvx\_vxv

 Node vvv's initial embedding.

 ... is just node vvv's original features.

 for all v∈V.v \in V.v∈V.

 and for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

hv(k)\color{#FE6100} h\_v^{(k)}hv(k)

===

f(k)(W(k)⋅∑u∈N(v)hu(k−1)∣N(v)∣+B(k)⋅hv(k−1)){\color{#4D9DB5}

 f^{(k)}}\left({\color{#4D9DB5} W^{(k)}} \cdot \frac{\sum\limits\_{u

\in \mathcal{N}(v)} {\color{#A95AA1} h\_u^{(k - 1)}}}{|\mathcal{N}(v)|} +

 {\color{#4D9DB5} B^{(k)}} \cdot {\color{#FE6100} h\_v^{(k -

1)}}\right)f(k)⎝⎜⎜⎜⎛W(k)⋅∣N(v)∣u∈N(v)∑hu(k−1)+B(k)⋅hv(k−1)⎠⎟⎟⎟⎞

 Node vvv's embedding at step kkk.

 Mean of vvv's neighbour's embeddings at step k−1k - 1k−1.

 Node vvv's embedding at step k−1k - 1k−1.

 for all v∈V.v \in V.v∈V.

 Color Codes:

 * Embedding of node vvv.

* Embedding of a neighbour of node vvv.

* (Potentially) Learnable parameters.

hv(0)\color{#FE6100} h\_v^{(0)}hv(0)

===

xvx\_vxv

 Node vvv's initial embedding.

 ... is just node vvv's original features.

 for all v∈V.v \in V.v∈V.

 and for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

hv(k)\color{#FE6100} h\_v^{(k)}hv(k)

===

f(k)(W(k)⋅[∑u∈N(v)αvu(k−1)hu(k−1)+αvv(k−1)hv(k−1)]){\color{#4D9DB5}

 f^{(k)}}\left({\color{#4D9DB5} W^{(k)}} \cdot \left[\sum\limits\_{u

\in \mathcal{N}(v)} \alpha\_{vu}^{(k - 1)} {\color{#A95AA1} h\_u^{(k -

1)}} + \alpha\_{vv}^{(k - 1)} {\color{#FE6100} h\_v^{(k -

 Node vvv's embedding at step kkk.

 Weighted mean of vvv's neighbour's embeddings at step k−1k - 1k−1.

 Node vvv's embedding at step k−1k - 1k−1.

 for all v∈V.v \in V.v∈V.

αvu(k)\alpha\_{vu}^{(k)}αvu(k)

===

A(k)(hv(k),hu(k))∑w∈N(v)A(k)(hv(k),hw(k))\frac{{\color{#4D9DB5}

 A^{(k)}}({\color{#FE6100} h\_v^{(k)}}, {\color{#A95AA1}

h\_u^{(k)}})}{\sum\limits\_{w \in \mathcal{N}(v)}{\color{#4D9DB5}

 for all (v,u)∈E.(v, u) \in E.(v,u)∈E.

 Color Codes:

 * Embedding of node vvv.

* Embedding of a neighbour of node vvv.

* (Potentially) Learnable parameters.

hv(0)\color{#FE6100} h\_v^{(0)}hv(0)

===

xvx\_vxv

 Node vvv's initial embedding.

 ... is just node vvv's original features.

 for all v∈V.v \in V.v∈V.

 and for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

hv(k)\color{#FE6100} h\_v^{(k)}hv(k)

===

f(k)(∑u∈N(v)hu(k−1)+(1+ϵ(k))⋅hv(k−1)){\color{#4D9DB5}

 f^{(k)}}\left(\sum\limits\_{u \in \mathcal{N}(v)} {\color{#A95AA1}

h\_u^{(k - 1)}} + (1 + {\color{#4D9DB5} \epsilon^{(k)}}) \cdot

 Node vvv's embedding at step kkk.

 Sum of vvv's neighbour's embeddings at step k−1k - 1k−1.

 Node vvv's embedding at step k−1k - 1k−1.

 for all v∈V.v \in V.v∈V.

 Color Codes:

 * Embedding of node vvv.

* Embedding of a neighbour of node vvv.

* (Potentially) Learnable parameters.

hv(0)\color{#FE6100} h\_v^{(0)}hv(0)

===

xvx\_vxv

 Node vvv's initial embedding.

 ... is just node vvv's original features.

 for all v∈V.v \in V.v∈V.

 and for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

hv(k)\color{#FE6100} h\_v^{(k)}hv(k)

===

f(k)(W(k)⋅[AGGu∈N(v)({hu(k−1)}), hv(k−1)]){\color{#4D9DB5}

 f^{(k)}}\left({\color{#4D9DB5} W^{(k)}} \cdot \left[\underset{{u \in

\mathcal{N}(v)}}{\color{#4D9DB5} \text{AGG}}(\{ {\color{#A95AA1}

 Node vvv's embedding at step kkk.

 Aggregation of vvv's neighbour's embeddings at step k−1k - 1k−1 ...

 ... concatenated with ...

 ... Node vvv's embedding at step k−1k - 1k−1.

 for all v∈V.v \in V.v∈V.

 Color Codes:

 * Embedding of node vvv.

* Embedding of a neighbour of node vvv.

* (Potentially) Learnable parameters.

Predictions can be made at each node by using the final computed embedding:

yv^=PREDICT(hv(K))

\hat{y\_v} = \text{PREDICT}({\color{#FE6100} h\_v^{(K)}})

yv^=PREDICT(hv(K))

The variant we discuss here is the 2-parameter model from the original paper ,

f(W⋅∑u∈N(v)hu∣N(v)∣+B⋅hv)

{\color{#4D9DB5} f}\left({\color{#4D9DB5} W} \cdot \sum\_{u \in

\mathcal{N}(v)} \frac{\color{#A95AA1} h\_u}{|\mathcal{N}(v)|} +

instead of the normalization defined in the original paper:

f(W⋅∑u∈N(v)hu∣N(u)∣∣N(v)∣+B⋅hv)

{\color{#4D9DB5} f}\left({\color{#4D9DB5} W} \cdot \sum\_{u \in

\mathcal{N}(v)} \frac{\color{#A95AA1}

h\_u}{\sqrt{|\mathcal{N}(u)||\mathcal{N}(v)|}} + {\color{#4D9DB5} B}

\cdot {\color{#FE6100} h\_v} \right)f⎝⎜⎛W⋅u∈N(v)∑∣N(u)∣∣N(v)∣hu+B⋅hv⎠⎟⎞

 for ease of exposition.

interactive\_list = undefined

 #gnn-models ul.list {

 display: grid;

 grid-template-columns: 1fr 1fr 1fr 1fr;

 grid-column-gap: 4px;

 text-align: center;

 padding: 0;

 list-style: none;

 }

 #gnn-models ul.list li {

 margin: 0;

 background: #f3f3f3;

 border-bottom: 3px solid #e3e3e3;

 cursor: pointer;

 line-height: 2em;

 border-top-left-radius: 3px;

 border-top-right-radius: 3px;

 }

 #gnn-models ul.list li:hover {

 border-bottom-color: #D9D9D9;

 }

 #gnn-models ul.list li.selected {

 background: #D9D9D9;

 border-bottom-color: #D9D9D9;

 }

 #gnn-models div {

 margin: 0 auto 10px auto;

 display: none;

 }

 #gnn-models {

 display: block;

 margin-left: auto;

 margin-right: auto;

 position: relative;

 }

 #gnn-models[data-show='gcn'] .gnn-model-gcn,

 #gnn-models[data-show='gin'] .gnn-model-gin,

 #gnn-models[data-show='graphsage'] .gnn-model-graphsage,

 #gnn-models[data-show='gat'] .gnn-model-gat {

 display: block;

 }

 /\* Legend \*/

 .legend {

 list-style: none;

 margin-top: 5px;

 }

 .legend li {

 margin-bottom: 0;

 }

 .legend span.box {

 border: 1px solid #ccc;

 float: left;

 width: 12px;

 height: 12px;

 margin: 2px;

 margin-right: 5px;

 }

 .legend .main { background-color: #FE6100; }

 .legend .neigh { background-color: #A95AA1; }

 .legend .param { background-color: #4D9DB5; }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/interactive-gnn-equations.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

###

 Thoughts

  demonstrates that aggregation functions indeed can be compared on how well

 they can uniquely preserve node neighbourhood features;

 Here, we've talk about GNNs where the computation only occurs at the nodes.

 More recent GNN models

 such as Message-Passing Neural Networks

 and Graph Networks

 perform computation over the edges as well;

 they compute edge embeddings together with node embeddings.

 This is an even more general framework -

 but the same 'message passing' ideas from this section apply.

 Interactive Graph Neural Networks

-----------------------------------

 Below is an interactive visualization of these GNN models on small graphs.

 but the same equations hold when the node features are vectors.

* GCN

* GAT

* GraphSAGE

* GIN

 #button-container {

 text-align: center;

 }

Reset

Undo Last Update

Update All Nodes

Randomize Graph

 #button-container {

 text-align: center;

 }

Reset

Undo Last Update

Update All Nodes

Randomize Graph

 #button-container {

 text-align: center;

 }

Reset

Undo Last Update

Update All Nodes

Randomize Graph

 #button-container {

 text-align: center;

 }

Reset

Undo Last Update

Update All Nodes

Randomize Graph

 #main-div {

 display: flex;

 flex-direction: row;

 align-items: center;

 justify-content: center;

 }

 #labels-div form {

 margin-top: 2%;

 margin-bottom: 2%;

 }

 #labels-div {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 width: 65%;

 }

 #iteration {

 font-weight: bold;

 margin-bottom: 20%;

 }

 #sliders {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 Initial Graph

 Parameters for Next Update

W(1)W^{(1)}W(1)

1

B(1)B^{(1)}B(1)

1

A6B2C-10D1E3

 #main-div {

 display: flex;

 flex-direction: row;

 align-items: center;

 justify-content: center;

 }

 #labels-div {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 width: 65%;

 }

 #iteration {

 font-weight: bold;

 margin-bottom: 20%;

 }

 #labels-div form {

 margin-top: 2%;

 margin-bottom: 2%;

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 Initial Graph

 Parameters for Next Update

W(1)W^{(1)}W(1)

1

AW(1)A\_W^{(1)}AW(1)

1

A6B2C-10D1E3

 #main-div {

 display: flex;

 flex-direction: row;

 align-items: center;

 justify-content: center;

 }

 #labels-div form {

 margin-top: 2%;

 margin-bottom: 2%;

 }

 #labels-div {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 width: 65%;

 }

 #iteration {

 font-weight: bold;

 margin-bottom: 20%;

 }

 #sliders {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 Initial Graph

 Parameters for Next Update

W(1)W^{(1)}W(1)

1

B(1)B^{(1)}B(1)

1

Us(1)U\_s^{(1)}Us(1)

1

Uh(1)U\_h^{(1)}Uh(1)

1

A6B2C-10D1E3

 #main-div {

 display: flex;

 flex-direction: row;

 align-items: center;

 justify-content: center;

 }

 #labels-div form {

 margin-top: 2%;

 margin-bottom: 2%;

 }

 #labels-div {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 width: 65%;

 }

 #iteration {

 font-weight: bold;

 margin-bottom: 20%;

 }

 #sliders {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 Initial Graph

 Parameters for Next Update

ϵ(1){\epsilon}^{(1)}ϵ(1)

0

A6B2C-10D1E3

 **Next Update (Iteration 1):**

 Equation for Node AAA:

 \begin{aligned}

 {\color{#FE6100} h\_{A}^{(1)}}

 &= {f} \left(

 {\color{#4D9DB5} B^{(1)}} \times {\color{#FE6100} h\_{A}^{(0)}} \right) \\

 &= f \left(

 {\color{#4D9DB5} 1} \times {\color{#FE6100} 6} \right) \\

 &= f \left(

 -3.5 +

 6 \right) \\

 &= f \left(2.5 \right) \\

 &= \text{ReLU} \left(2.5\right)

 = 2.5.

 **Next Update (Iteration 1):**

 Equation for Node AAA:

 \begin{aligned}

 {\color{#FE6100} h\_{A}^{(1)}}

 &= f \left({\color{#4D9DB5} W^{(1)}} \times

\left(\alpha\_{AC} \times {\color{#A95AA1} h\_{C}^{(0)}} + \alpha\_{AE}

\times {\color{#A95AA1} h\_{E}^{(0)}} + \alpha\_{AA} \times

{\color{#FE6100} h\_{A}^{(0)}} \right) \right) \\

 &= f \left({\color{#4D9DB5} W^{(1)}} \times \left(0

\times {\color{#A95AA1} h\_{C}^{(0)}} + 0.05 \times {\color{#A95AA1}

h\_{E}^{(0)}} + 0.95 \times {\color{#FE6100} h\_{A}^{(0)}} \right) \right)

 \\

 &= f \left({\color{#4D9DB5} W^{(1)}} \times \big(0

\times {\color{#A95AA1} -10} + 0.05 \times {\color{#A95AA1} 3} + 0.95

\times {\color{#FE6100} 6}\big) \right) \\

 &= f \left({\color{#4D9DB5} W^{(1)}} \times \big(0 +

0.14 + 5.72 \big) \right) \\

 &= f \left({\color{#4D9DB5} 1} \times \left(5.86

\right) \right) \\

 &= f \left(5.86\right) \\

 &= \text{ReLU} \left(5.86\right)

 = 5.86.

 with attention weights αA\alpha\_{A}αA computed as:

 \begin{aligned}

 \alpha\_{AC} = \frac{\exp{(e\_{AC})}}{\exp{(e\_{AC})} +

\exp{(e\_{AE})} + \exp{(e\_{AA})}} &\approx 0. \\ \alpha\_{AE} =

\frac{\exp{(e\_{AE})}}{\exp{(e\_{AC})} + \exp{(e\_{AE})} + \exp{(e\_{AA})}}

&\approx 0.05. \\ \alpha\_{AA} = \frac{\exp{(e\_{AA})}}{\exp{(e\_{AC})}

 + \exp{(e\_{AE})} + \exp{(e\_{AA})}} &\approx 0.95.

 \end{aligned}

 where the unnormalized attention weights eAe\_{A}eA are given by:

 \begin{aligned}

 e\_{AC} &= \text{ReLU}\left({\color{#4D9DB5} A\_{W}^{(1)}}

\left({\color{#4D9DB5} W^{(1)}}{\color{#FE6100} h\_{A}^{(0)}} +

{\color{#4D9DB5} W^{(1)}}{\color{#A95AA1} h\_{C}^{(0)}} \right)\right) =

 \text{ReLU}\left({\color{#4D9DB5} 1}\left({\color{#4D9DB5} 1}\times

{\color{#FE6100} 6} + {\color{#4D9DB5} 1} \times {\color{#A95AA1}

-10}\right)\right) = \text{ReLU}\left(-4\right) = 0. \\ e\_{AE} &=

\text{ReLU}\left({\color{#4D9DB5} A\_{W}^{(1)}} \left({\color{#4D9DB5}

W^{(1)}}{\color{#FE6100} h\_{A}^{(0)}} + {\color{#4D9DB5}

W^{(1)}}{\color{#A95AA1} h\_{E}^{(0)}} \right)\right) =

\text{ReLU}\left({\color{#4D9DB5} 1}\left({\color{#4D9DB5} 1}\times

{\color{#FE6100} 6} + {\color{#4D9DB5} 1} \times {\color{#A95AA1}

3}\right)\right) = \text{ReLU}\left(9\right) = 9. \\ e\_{AA} &=

\text{ReLU}\left({\color{#4D9DB5} A\_{W}^{(1)}} \left({\color{#4D9DB5}

W^{(1)}}{\color{#FE6100} h\_{A}^{(0)}} + {\color{#4D9DB5}

W^{(1)}}{\color{#FE6100} h\_{A}^{(0)}} \right)\right) =

\text{ReLU}\left({\color{#4D9DB5} 1}\left({\color{#4D9DB5} 1}\times

{\color{#FE6100} 6} + {\color{#4D9DB5} 1} \times {\color{#FE6100}

6}\right)\right) = \text{ReLU}\left(12\right) = 12. \\

 \end{aligned}

 We have omitted the superscripts on the attention weights for clarity.

 **Next Update (Iteration 1):**

 Equation for Node AAA:

 hA(1)=f(W(1)×RNN[hC(0),hE(0)]+B(1)×hA(0))=f(1×3+1×6)=f(3+6)=f(9)=ReLU(9)=9.

 \begin{aligned}

 {\color{#FE6100} h\_{A}^{(1)}}

 &= {f} \left(

 {\color{#4D9DB5} W^{(1)}} \times

\text{RNN}\left[{\color{#A95AA1} h\_{C}^{(0)}} , {\color{#A95AA1}

h\_{E}^{(0)}}\right] +

 {\color{#4D9DB5} B^{(1)}} \times {\color{#FE6100}

h\_{A}^{(0)}} \right) \\

 &= f \left(

 {\color{#4D9DB5} 1} \times {\color{#A95AA1} 3} +

 {\color{#4D9DB5} 1} \times {\color{#FE6100} 6}

\right) \\

 &= f \left(

 3 +

 6 \right) \\

 &= f \left(9 \right) \\

 &= \text{ReLU} \left(9\right)

 = 9.

 \begin{aligned}

 \qquad s\_0 &= 0. \\

 s\_{1}&= \text{ReLU}\left({\color{#4D9DB5} U\_s} s\_{0} +

{\color{#4D9DB5} U\_h} {\color{#A95AA1} h\_{C}^{(0)}}\right) =

\text{ReLU}\left(

 {\color{#4D9DB5} 1}

 \times 0 +

 {\color{#4D9DB5} 1}

 \times {\color{#A95AA1} -10} \right) =

\text{ReLU}\left(-10\right) = 0. \\ s\_{2}&=

\text{ReLU}\left({\color{#4D9DB5} U\_s} s\_{1} + {\color{#4D9DB5} U\_h}

{\color{#A95AA1} h\_{E}^{(0)}}\right) = \text{ReLU}\left(

 {\color{#4D9DB5} 1}

 \times 0 +

 {\color{#4D9DB5} 1}

 \times {\color{#A95AA1} 3} \right) = \text{ReLU}\left(3\right) =

3.

 \end{aligned}

 Concisely, the initial RNN\text{RNN}RNN state is s0=0s\_0 = 0s0=0, and

 the RNN\text{RNN}RNN state update equation is:

 st=ReLU(Usst−1+Uhhut(0))=ReLU(1st−1+1hut(0)).

 \begin{aligned}

 s\_{t} &= \text{ReLU}\left({\color{#4D9DB5} U\_s} s\_{t - 1} +

{\color{#4D9DB5} U\_h} h\_{u\_t}^{(0)}\right) \\ &=

\text{ReLU}\left({\color{#4D9DB5} 1} s\_{t - 1} + {\color{#4D9DB5} 1}

 h\_{u\_t}^{(0)}\right).

 \end{aligned}

 st=ReLU(Usst−1+Uhhut(0))=ReLU(1st−1+1hut(0)).

 RNN\text{RNN}RNN to learn the same final aggregated value.

 This tries to make the RNN\text{RNN}RNN node-order invariant.

 **Next Update (Iteration 1):**

 Equation for Node AAA:

 \begin{aligned}

 {\color{#FE6100} h\_{A}^{(1)}}

 &= {f} \left(

 \left({\color{#A95AA1} h\_{C}^{(0)}} + {\color{#A95AA1} h\_{E}^{(0)}}\right) +

 &= f \left(

 \left({\color{#A95AA1} -10} + {\color{#A95AA1} 3}\right) +

 \left(1 + {\color{#4D9DB5} 0}\right)

 \times {\color{#FE6100} 6} \right) \\

 &= f \left(

 \left(-7\right) +

 6 \right) \\

 &= f \left(-1 \right) \\

 &= \text{ReLU} \left(-1\right)

 = 0.

 div.eqn {

 margin-left: 2%;

 }

 #gnn-models-viz ul {

 display: grid;

 grid-template-columns: 1fr 1fr 1fr 1fr;

 grid-column-gap: 4px;

 text-align: center;

 padding: 0;

 justify-content: center;

 align-content: center;

 list-style: none;

 max-width: 100%;

 }

 #gnn-models-viz ul li {

 margin: 0;

 background: #f3f3f3;

 border-bottom: 3px solid #e3e3e3;

 cursor: pointer;

 line-height: 2em;

 border-top-left-radius: 3px;

 border-top-right-radius: 3px;

 }

 #gnn-models-viz ul li:hover {

 border-bottom-color: #D9D9D9;

 }

 #gnn-models-viz ul li.selected {

 background: #D9D9D9;

 border-bottom-color: #D9D9D9;

 }

 #gnn-models-viz {

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/interactive-gnn-visualizations.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Choose a GNN model using the tabs at the top. Click on a node

to see the update equation at that node for the next iteration.

 Use the sliders on the left to change the weights for the

current iteration, and watch how the update equation changes.

 This ideology is followed by many popular Graph Neural Network libraries,

 From Local to Global Convolutions

-----------------------------------

 The methods we've seen so far perform 'local' convolutions:

 While performing enough steps of message-passing will eventually ensure that

 information from all nodes in the graph is passed,

 one may wonder if there are more direct ways to perform 'global' convolutions.

 in the context of neural networks by ,

 much before any of the GNN models we looked at above.

###

 Spectral Convolutions

 As before, we will focus on the case where nodes have one-dimensional features.

 'feature vector' x∈Rnx \in \mathbb{R}^nx∈Rn.

**Key Idea:**

 Given a feature vector xxx,

 the Laplacian LLL allows us to quantify how smooth xxx is, with respect to GGG.

 How?

 if we look at the following quantity involving LLL:

RL(x)=xTLxxTx=∑(i,j)∈E(xi−xj)2∑ixi2=∑(i,j)∈E(xi−xj)2.

 R\_L(x) = \frac{x^T L x}{x^T x} = \frac{\sum\_{(i, j) \in E}

(x\_i - x\_j)^2}{\sum\_i x\_i^2} = \sum\_{(i, j) \in E} (x\_i - x\_j)^2.

 RL(x)=xTxxTLx=∑ixi2∑(i,j)∈E(xi−xj)2=(i,j)∈E∑(xi−xj)2.

 we immediately see that feature vectors xxx that assign similar values to

 An eigenvalue λ\lambdaλ of a matrix AAA is a value

 For a friendly introduction to eigenvectors,

 please see [this tutorial](http://www.sosmath.com/matrix/eigen0/eigen0.html).

 uk1Tuk2={1 if k1=k2.0 if k1≠k2.

 u\_{k\_1}^T u\_{k\_2} =

 \begin{cases}

 1 \quad \text{ if } {k\_1} = {k\_2}. \\

 0 \quad \text{ if } {k\_1} \neq {k\_2}.

 \end{cases}

 uk1Tuk2={1 if k1=k2.0 if k1≠k2.

argminx, x⊥{u1,…,ui−1}RL(x)=ui.minx, x⊥{u1,…,ui−1}RL(x)=λi.

 \qquad

 \qquad

 \qquad

 \min\_{x, \ x \perp \{u\_1, \ldots, u\_{i - 1}\}} R\_L(x) = \lambda\_i.

 x, x⊥{u1,…,ui−1}argminRL(x)=ui.x, x⊥{u1,…,ui−1}minRL(x)=λi.

 The set of eigenvalues of LLL are called its 'spectrum', hence the name!

 We denote the 'spectral' decomposition of LLL as:

 L=UΛUT.

 L = U \Lambda U^T.

 L=UΛUT.

 where Λ\LambdaΛ is the diagonal matrix of sorted eigenvalues,

 Λ=[λ1⋱λn]U=[u1 ⋯ un].

 \Lambda = \begin{bmatrix}

 \lambda\_{1} & & \\

 & \ddots & \\

 & & \lambda\_{n}

 \end{bmatrix}

 \qquad

 \qquad

 \qquad

 \qquad

 U = \begin{bmatrix} \\ u\_1 \ \cdots \ u\_n \\ \end{bmatrix}.

 Λ=⎣⎡λ1⋱λn⎦⎤U=⎣⎡u1 ⋯ un⎦⎤.

 As these nnn eigenvectors form a basis for Rn\mathbb{R}^nRn,

 x=∑i=1nxi^ui=Ux^.

 x = \sum\_{i = 1}^n \hat{x\_i} u\_i = U \hat{x}.

 x=i=1∑nxi^ui=Ux^.

 We call x^\hat{x}x^ as the spectral representation of the feature vector xxx.

 The orthonormality condition allows us to state:

 x=Ux^⟺UTx=x^.

 x = U \hat{x} \quad \Longleftrightarrow \quad U^T x = \hat{x}.

 x=Ux^⟺UTx=x^.

 This pair of equations allows us to interconvert

 for any vector x∈Rnx \in \mathbb{R}^nx∈Rn.

###

 Spectral Representations of Natural Images

 connected by edges to adjacent pixels.

 vector, indicating the values for the red, green and blue (RGB) channels.

 To shed some light on what the spectral representation actually encodes,

* We first collect all pixel values across a channel into a feature vector xxx.

* Then, we obtain its spectral representation x^\hat{x}x^.

 x^=UTx

 \hat{x} = U^T x

 x^=UTx

* We truncate this to the first mmm components to get x^m\hat{x}\_mx^m.

 x^m=Truncatem(x^)

 \hat{x}\_m = \text{Truncate}\_m(\hat{x})

 x^m=Truncatem(x^)

 xm=Ux^m

 x\_m = U \hat{x}\_m

 xm=Ux^m

 Finally, we stack the resulting channels back together to get back an image.

 We can now see how the resulting image changes with choices of mmm.

 as we can reconstruct each channel exactly.

Sample Image

 Chicken

 Fish

 Frog

 Spider

 Original Image

 xxx

Keep First 200 Spectral Components

 Transformed Image

 x′x'x′

Number of Spectral Components (m)

200

updateArrowCaption = undefined

drawArrow = undefined

drawCurrBaseImg = undefined

drawCurrUpdImg = undefined

 #specDecompBody {

 display: flex;

 justify-content: center;

 }

 #svgArrow {

 margin-top: 83px;

 margin-left: 2em;

 margin-right: 2em;

 }

 #specDecompAll {

 display: flex;

 flex-direction: column;

 text-align: center;

 height: 350px;

 justify-content: space-evenly;

 }

 #specDecompInputsRadio input {

 margin-top: 0.5em;

 vertical-align: unset !important;

 }

 #origCaption {

 margin-top: 1em;

 }

 #updCaption {

 margin-top: 1em;

 }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/spectral-decompositions-of-natural-images.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Use the radio buttons at the top to chose one of the four sample images.

 Each of these images has been taken from the ImageNet

 dataset and downsampled to 505050 pixels wide and 404040 pixels tall.

 images get progressively blurrier as the number of components decrease.

 As mmm decreases, we see that the output image xmx\_mxm gets blurrier.

 We see that we do not need to keep all nnn components;

 We can relate this to the Fourier decomposition of images:

 To complement the visualization above,

 We change the coefficients of the first 101010 out of 646464 eigenvectors

 in the spectral representation

 and see how the resulting image changes:

Spectral Representation

x^=[x^1 x^2 x^3 x^4 x^5 x^6 x^7 x^8 x^9 x^10]\hat{x}

 = \left[

 \hat{x}\_{1} \ \hat{x}\_{2} \ \hat{x}\_{3} \ \hat{x}\_{4} \ \hat{x}\_{5} \

 \hat{x}\_{6} \ \hat{x}\_{7} \ \hat{x}\_{8} \ \hat{x}\_{9} \ \hat{x}\_{10}

 \right]x^=[x^1 x^2 x^3 x^4 x^5 x^6 x^7 x^8 x^9 x^10]

RL(x)=1.77R\_L(x) = 1.77RL(x)=1.77

Natural Representation

x∈R64x \in \mathbb{R}^{64}x∈R64

 #spec-main-div {

 position: relative;

 height: 400px;

 }

 #spec-svg {

 position: absolute;

 left: 0px;

 top: 0px;

 }

 #figcaptions {

 position: relative;

 }

 #figcaptions figcaption {

 position: absolute;

 }

 #spec-caption-1 {

 left: 660px;

 top: 400px;

 }

 #spec-caption-2 {

 left: 620px;

 top: 420px;

 }

 #quot-caption-div {

 position: relative;

 width: 200px;

 left: 170px;

 top: 350px;

 }

 #figcaptions #quot-caption-1 {

 position: relative;

 text-align: center;

 font: inherit;

 color: inherit;

 }

 #nat-caption-1 {

 left: 205px;

 top: 400px;

 }

 #nat-caption-2 {

 left: 255px;

 top: 420px;

 }

 #spec-sliders form {

 margin-left: 2%;

 margin-right: 2%;

 }

 #spec-sliders input {

 width: 100%;

 }

 #spec-sliders output {

 width: 3em;

 margin-left: 0 !important;

 }

 #spec-sliders {

 display: flex;

 margin: 0 auto;

 text-align: center;

 }

 #subgrid-main-div {

 display: flex;

 margin: 0 auto;

 }

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

−0.4−0.20.00.20.4Color Scale

x^1{\hat{x}\_{1}}x^1

0x^2{\hat{x}\_{2}}x^2

-0.3x^3{\hat{x}\_{3}}x^3

-0.3x^4{\hat{x}\_{4}}x^4

0.6x^5{\hat{x}\_{5}}x^5

-0.7x^6{\hat{x}\_{6}}x^6

0.4x^7{\hat{x}\_{7}}x^7

0.4x^8{\hat{x}\_{8}}x^8

0.4x^9{\hat{x}\_{9}}x^9

0.4x^10{\hat{x}\_{10}}x^10

0.7

 #button-container {

 text-align: center;

 }

Reset Coefficients

spec\_input\_slider\_watch = undefined

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/interactive-spectral-conversions.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Move the sliders to change the spectral representation x^\hat{x}x^ (right),

 and see how xxx itself changes on the image (left).

 Note how the first eigenvectors are much 'smoother' than the later ones,

 and the many patterns we can make with only 101010 eigenvectors.

 and the smoothness correspondingly decreases as we consider later eigenvectors.

 For any image xxx, we can think of

 the initial entries of the spectral representation x^\hat{x}x^

### Embedding Computation

 We now have the background to understand spectral convolutions

 As before, the model we describe below has KKK layers:

 each layer kkk has learnable parameters w^(k)\hat{w}^{(k)}w^(k),

 called the 'filter weights'.

 eigenvectors used to compute the spectral representations.

 We had shown in the previous section that we can take m≪nm \ll nm≪n

 and still not lose out on significant amounts of information.

 than just direct convolution in the natural domain.

 using spectral representations automatically enforces an inductive bias for

 neighbouring nodes to get similar representations.

 Assuming one-dimensional node features for now,

 the output of each layer is a vector of node representations h(k)h^{(k)}h(k),

 where each node's representation corresponds to a row

 of the vector.

h(k)=[h1(k)⋮hn(k)]

 h(k)=⎣⎢⎢⎢⎡h1(k)⋮hn(k)⎦⎥⎥⎥⎤

 for each k=0,1,2,…k = 0, 1, 2, \ldots k=0,1,2,… upto K KK.

 allowing us to compute UmU\_mUm.

 Start with the original features.

h(0)=x\quad {\color{#FE6100} h^{(0)}} = xh(0)=x

 Then iterate, for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

h^(k−1)=UmTh(k−1)g^(k)=w^(k)⊙h^(k−1)g(k)=Umg^(k)h(k)=σ(g(k))

 \begin{aligned}

 {\hat{h}^{(k - 1)}} &= U\_m^T {\color{#FE6100} h^{(k - 1)}} \\

 \\

 {\hat{g}^{(k)}} &=

 {\color{#4D9DB5} \hat{w}^{(k)}} \odot {\color{#FE6100} \hat{h}^{(k - 1)}} \\

 \\

 {g^{(k)}} &= U\_m {\hat{g}^{(k)}} \\

 \\

 {\color{#FE6100} h^{(k)}} &= \sigma \left({g^{(k)}} \right)

 \end{aligned}

 h^(k−1)g^(k)g(k)h(k)=UmTh(k−1)=w^(k)⊙h^(k−1)=Umg^(k)=σ(g(k))

⊙\odot⊙ represents element-wise multiplication.

 Color Codes:

 * Computed node embeddings.

* Learnable parameters.

 #spectral-conv div {

 margin: 0 auto 10px auto;

 display: block;

 }

 #spectral-conv {

 width: 800px;

 height: 380px;

 display: block;

 margin-top: 0;

 margin-bottom: 0;

 margin-left: auto;

 margin-right: auto;

 position: relative;

 }

 #spectral-conv-init {

 width: 800px;

 height: 100px;

 display: block;

 margin-top: 0;

 margin-left: auto;

 margin-right: auto;

 position: relative;

 }

 span.green {

 color: #177245;

 }

 span.blue {

 color: #0047AB;

 }

 /\* Legend \*/

 .legend {

 list-style: none;

 margin-top: 5px;

 }

 .legend li {

 margin-bottom: 5px;

 }

 .legend span.box {

 border: 1px solid #ccc;

 float: left;

 width: 12px;

 height: 12px;

 margin: 2px;

 margin-right: 5px;

 }

 .legend .main { background-color: #FE6100; }

 .legend .param { background-color: #4D9DB5; }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/spectral-convolutions-equation.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 if (name === "spec\_figure") return Inspector.into(".spec\_figure")();

 if (name === "style") return Inspector.into(".spec\_figure\_style")();

 });

 }, 200);

 see  for details.

###

 Spectral Convolutions are Node-Order Equivariant

 as for Laplacian polynomial filters.

**Details for the Interested Reader**

 As in [our proof before](#poly-filters-equivariance),

 let's fix an arbitrary node-order.

 Then, any other node-order can be represented by a

 permutation of this original node-order.

 We can associate this permutation with its permutation matrix PPP.

 Under this new node-order,

 the quantities below transform in the following way:

 x→PxA→PAPTL→PLPTUm→PUm

 \begin{aligned}

 x &\to Px \\

 A &\to PAP^T \\

 L &\to PLP^T \\

 U\_m &\to PU\_m

 \end{aligned}

 xALUm→Px→PAPT→PLPT→PUm

 which implies that, in the embedding computation:

 x^→(PUm)T(Px)=UmTx=x^w^→(PUm)T(Pw)=UmTw=w^g^→g^g→(PUm)g^=P(Umg^)=Pg

 \begin{aligned}

 \hat{x} &\to \left(PU\_m\right)^T (Px) = U\_m^T x = \hat{x} \\

 \hat{w} &\to \left(PU\_m\right)^T (Pw) = U\_m^T w = \hat{w} \\

 \hat{g} &\to \hat{g} \\

 g &\to (PU\_m)\hat{g} = P(U\_m\hat{g}) = Pg

 \end{aligned}

 x^w^g^g→(PUm)T(Px)=UmTx=x^→(PUm)T(Pw)=UmTw=w^→g^→(PUm)g^=P(Umg^)=Pg

 Hence, as σ\sigmaσ is applied elementwise:

 f(Px)=σ(Pg)=Pσ(g)=Pf(x)

 f(Px) = \sigma(Pg) = P \sigma(g) = P f(x)

 f(Px)=σ(Pg)=Pσ(g)=Pf(x)

 as required.

 are unchanged by permutations of the nodes.

 Formally, they are what we would call node-order invariant.

 The theory of spectral convolutions is mathematically well-grounded;

 however, there are some key disadvantages that we must talk about:

 because of the repeated

 multiplications with UmU\_mUm and UmTU\_m^TUmT.

* The learned filters are specific to the input graphs,

 as they are represented in terms

 of the spectral decomposition of input graph Laplacian LLL.

 This means they do not transfer well to new graphs

 which have significantly different structure (and hence, significantly

 different eigenvalues) .

 While spectral convolutions have largely been superseded by

 'local' convolutions for the reasons discussed above,

 there is still much merit to understanding the ideas behind them.

 Indeed, a recently proposed GNN model called Directional Graph Networks

 actually uses the Laplacian eigenvectors

 and their mathematical properties

 extensively.

###

 Global Propagation via Graph Embeddings

 A simpler way to incorporate graph-level information

 is to compute embeddings of the entire graph by pooling node

 (and possibly edge) embeddings,

 and then using the graph embedding to update node embeddings,

 following an iterative scheme similar to what we have looked at here.

 This is an approach used by Graph Networks

 .

 We will briefly discuss how graph-level embeddings

 can be constructed in [Pooling](#pooling).

 However, such approaches tend to ignore the underlying

 topology of the graph that spectral convolutions can capture.

 Learning GNN Parameters

-------------------------

 once a suitable loss function L\mathcal{L}L is defined:

 such as categorical cross-entropy when multiple classes are present:

 L(yv,yv^)=−∑cyvclogyvc^.

 \mathcal{L}(y\_v, \hat{y\_v}) = -\sum\_{c} y\_{vc} \log{\hat{y\_{vc}}}.

 L(yv,yv^)=−c∑yvclogyvc^.

 LG=∑v∈Lab(G)L(yv,yv^)∣Lab(G)∣

 LG=∣Lab(G)∣v∈Lab(G)∑L(yv,yv^)

 where, we only compute losses over labelled nodes Lab(G)\text{Lab}(G)Lab(G).

* **Graph Classification**: By aggregating node representations,

 one can construct a vector representation of the entire graph.

 See [Pooling](#pooling) for how representations of graphs can be constructed.

* **Link Prediction**: By sampling pairs of adjacent and non-adjacent nodes,

 L(yv,yu,evu)=−evulog(pvu)−(1−evu)log(1−pvu)pvu=σ(yvTyu)

 \begin{aligned}

 p\_{vu} &= \sigma(y\_v^Ty\_u)

 \end{aligned}

 L(yv,yu,evu)pvu=−evulog(pvu)−(1−evu)log(1−pvu)=σ(yvTyu)

* **Node Clustering**: By simply clustering the learned node representations.

 The broad success of pre-training for natural language processing models

 such as ELMo  and BERT

 has sparked interest in similar techniques for GNNs

 .

 The key idea in each of these papers is to train GNNs to predict

 local (eg. node degrees, clustering coefficient, masked node attributes)

 mimicking random-walk approaches such as node2vec  and DeepWalk :

LG=∑v∑u∈NR(v)logexpzvTzu∑u′expzu′Tzu.

 LG=v∑u∈NR(v)∑logu′∑expzu′TzuexpzvTzu.

 techniques such as Noise Contrastive Estimation  are especially useful.

 Conclusion and Further Reading

--------------------------------

 While we have looked at many techniques and ideas in this article,

 the field of Graph Neural Networks is extremely vast.

 while still communicating the key ideas and design principles behind GNNs.

 We recommend the interested reader take a look at

  for a more comprehensive survey.

###

 GNNs in Practice

 but we can still represent many GNN update equations using

 For example, the GCN variant discussed here can be represented as:

 h(k)=D−1A⋅h(k−1)W(k)T+h(k−1)B(k)T.

 h^{(k)} = D^{-1} A \cdot h^{(k - 1)} {W^{(k)}}^T + h^{(k - 1)} {B^{(k)}}^T.

 h(k)=D−1A⋅h(k−1)W(k)T+h(k−1)B(k)T.

 such as GPUs.

 Regularization techniques for standard neural networks,

 such as Dropout ,

 can be applied in a straightforward manner to the parameters

 (for example, zero out entire rows of W(k)W^{(k)}W(k) above).

 However, there are graph-specific techniques such as DropEdge

 that removes entire edges at random from the graph,

 that also boost the performance of many GNN models.

###

 Different Kinds of Graphs

 However, there are some simple variants of spatial convolutions for:

* Temporal graphs: Aggregate across previous and/or future node features.

 see  for more discussion.

###

 Pooling

 This article discusses how GNNs compute useful representations

 of nodes.

 But what if we wanted to compute representations of graphs for

 graph-level tasks (for example, predicting the toxicity of a molecule)?

 hG=PREDICTG(AGGv∈G({hv}))

 hG=PREDICTG(AGGv∈G({hv}))

* SortPool: Sort vertices

of the graph to get a fixed-size node-order invariant representation of

the graph, and then apply any standard neural network architecture.

* DiffPool: Learn to

cluster vertices, build a coarser graph over clusters instead of nodes,

then apply a GNN over the coarser graph. Repeat until only one cluster

is left.

* SAGPool: Apply a GNN to

learn node scores, then keep only the nodes with the top scores,

throwing away the rest. Repeat until only one node is left.

 Supplementary Material

------------------------

###

 Reproducing Experiments

 The experiments from

 can be reproduced using the following

###

 Recreating Visualizations

 To aid in the creation of future interactive articles,

 we have created ObservableHQ

 notebooks for each of the interactive visualizations here:

 which pulls together the following standalone notebooks:

Graph Kernels

Node2vec: Scalable Feature Learning for Networks

DeepWalk: Online Learning of Social Representations

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Neural Message Passing for Quantum Chemistry

Learning Convolutional Neural Networks for Graphs

A Tutorial on Spectral Clustering

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Wavelets on Graphs via Spectral Graph Theory

Chebyshev Polynomials

Semi-Supervised Classification with Graph Convolutional Networks

Graph Attention Networks

Inductive Representation Learning on Large Graphs

How Powerful are Graph Neural Networks?

Relational inductive biases, deep learning, and graph networks

Spectral Networks and Locally Connected Networks on Graphs

ImageNet: A Large-Scale Hierarchical Image Database

On the Transferability of Spectral Graph Filters

Directional Graph Networks

Deep contextualized word representations

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Strategies for Pre-training Graph Neural Networks

Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes

When Does Self-Supervision Help Graph Convolutional Networks?

Self-supervised Learning on Graphs: Deep Insights and New Direction

Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

Learning word embeddings efficiently with noise-contrastive estimation

A Comprehensive Survey on Graph Neural Networks

Graph Neural Networks: A Review of Methods and Applications

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

An End-to-End Deep Learning Architecture for Graph Classification

Hierarchical Graph Representation Learning with Differentiable Pooling

Self-Attention Graph Pooling


Self-classifying MNIST Digits

Ettore Randazzo
Alexander Mordvintsev
Eyvind Niklasson
Michael Levin
Sam Greydanus

Abstract: This article is part of the     Differentiable Self-organizing Systems Thread,     an experimental format collecting invited short articles delving into     differentiable self-organizing systems, interspersed with critical     commentary from several experts in adjacent fields.

### Contents

[Model](#model)

[Experiments](#experiment-1)

* [Self-classify, persist & mutate](#experiment-1)

* [Stabilizing classification](#experiment-2)

[Robustness](#robustness)

[Related Work](#related-work)

[Discussion](#discussion)

![](Self-classifying%20MNIST%20Digits_files/multiple-pages.svg)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Growing Neural Cellular Automata](https://distill.pub/2020/growing-ca/)

[Self-Organising Textures](https://distill.pub/selforg/2021/textures/)

Growing Neural Cellular Automata

 demonstrated how simple cellular automata (CAs) can learn to

self-organise into complex shapes while being resistant to

perturbations. Such a computational model approximates a solution to an

open question in biology, namely, how do cells cooperate to create a

complex multicellular anatomy and work to regenerate it upon damage? The

 model parameterizing the cells' rules is parameter-efficient,

end-to-end differentiable, and illustrates a new approach to modeling

the regulation of anatomical homeostasis. In this work, we use a version

 of this model to show how CAs can be applied to a common task in

Our question is closely related to another unsolved problem in

developmental and regenerative biology: how cell groups decide whether

an organ or tissue pattern is correct, or whether current anatomy needs

to be remodeled (anatomical surveillance and repair toward a specific

target morphology). For example, when scientists surgically transplanted

 a salamander tail to its flank, it slowly remodeled into a limb - the

organ that belongs at this location .

 Similarly, tadpoles with craniofacial organs in the wrong positions

usually become normal frogs because they remodel their faces, placing

the eye, mouth, nostrils, etc. in their correct locations. Cell groups

move around and stop when the correct frog-specific anatomical

configuration has been achieved .

 All of these examples illustrate the ability of biological systems to

determine their current anatomical structure and decide whether it

matches a species-specific target morphology .

 Despite the recent progress in molecular biology of genes necessary for

 this process, there is still a fundamental knowledge gap concerning the

 algorithms sufficient for cell collectives to measure and classify

their own large-scale morphology. More broadly, it is important to

create computational models of swarm intelligence that explicitly define

 and distinguish the dynamics of the basal cognition of single cells

versus cell collectives .

### The self-classifying MNIST task

Suppose a population of agents is arranged on a grid. They do not

know where they are in the grid and they can only communicate with their

 immediate neighbors. They can also observe whether a neighbor is

missing. Now suppose these agents are arranged to form the shape of a

digit. Given that all the agents operate under the same rules, can they

form a communication protocol such that, after a number of iterations of

 if some agents were to be removed and added to form a new digit from a

preexisting one, would they be able to know which the new digit is?

Because digits are not rotationally invariant (i.e. 6 is a rotation

of 9), we presume the agents must be made aware of their orientation

with respect to the grid. Therefore, while they do not know where they

are, they do know where up, down, left and right are. The biological

analogy here is a situation where the remodeling structures exist in the

 context of a larger body and a set of morphogen gradients or tissue

polarity that indicate directional information with respect to the three

 major body axes. Given these preliminaries, we introduce the

self-classifying MNIST task.

Each sample of the MNIST dataset

consists of a 28x28 image with a single monochrome channel that is

classically displayed in greyscale. The label is an integer in [0,9][0,9][0,9].

Our goal is for all cells that make up the digit to correctly output

the label of the digit. To convey this structural information to the

cells, we make a distinction between alive and dead cells by rescaling

the values of the image to [0, 1]. Then we treat a cell as alive if its

value in the MNIST sample is larger than 0.1. The intuition here is that

 we are placing living cells in a cookie cutter and asking them to

identify the global shape of the cookie cutter. We visualize the label

output by assigning a color to each cell, as you can see above. We use

the same mapping between colors and labels throughout the article.

Please note that there is a slider in the interactive demo controls

which you can use to adjust the color palette if you have trouble

differentiating between the default colors.

Model

-----

### Target labels

The work in Growing CA used RGB images as targets, and optimized the

first three state channels to approximate those images. For our

experiments, we treat the last ten channels of our cells as a

pseudo-distribution over each possible label (digit). During inference,

we simply pick the label corresponding to the channel with the highest

output value.

### Alive cells and cell states

In Growing CA we assigned a cell's state to be "dead" or "alive"

based on the strength of its alpha channel and the activity of its

neighbors. This is similar to the rules of Conway's Game of Life .

 In the Growing CA model, "alive" cells are cells which update their

state and dead cells are "frozen" and do not undergo updates. In

contrast to biological life, what we call "dead" cells aren't dead in

the sense of being non-existent or decayed, but rather frozen: they are

visible to their neighbors and maintain their state throughout the

simulation. In this work, meanwhile, we use input pixel values to

determine whether cells are alive or dead and perform computations with

 It is important to note that the values of MNIST pixels are exposed to

the cell update rule as an immutable channel of the cell state. In other

 words, we make cells aware of their own pixel intensities as well as

those of their neighbors. Given 19 mutable cell state channels (nine

general purpose state channels for communication and ten output state

channels for digit classification) and an immutable pixel channel, each

cell perceives 19 + 1 state channels and only outputs state updates for

the 19 mutable state channels.

**A note on digit topology.**Keen

readers may notice that our model requires each digit to be a single

connected component in order for classification to be possible, since

any disconnected components will be unable to propagate information

between themselves. We made this design decision in order to stay true

to our core biological analogy, which involves a group of cells that is

trying to identify its global shape. Even though the vast majority of

samples from MNIST are fully connected, some aren't. We do not expect

our models to classify non-connected minor components correctly, but we

do not remove them This choice complicates

 comparison between the MNIST train/test accuracies of neural network

classifiers vs. CAs. However, such a comparison is not in scope of this

article..

### Perception

**A note on model size.** Like the Growing CA model, our

 MNIST CA is small by the standards of deep learning - it has less than

25k parameters. Since this work aims to demonstrate a novel approach to

classification, we do not attempt to maximise the validation accuracy of

 the model by increasing the number of parameters or any other tuning.

We suspect that, as with other deep neural network models, one would

observe a positive correlation between accuracy and model size.

Experiment 1: Self-classify, persist and mutate

-----------------------------------------------

In our first experiment, we use the same training paradigm as was

discussed in Growing CA. We train with a pool of initial samples to

allow the model to learn to persist and then perturb the converged

states. However, our perturbation is different. Previously, we destroyed

 the states of cells at random in order to make the CAs resistant to

destructive perturbations (analogous to traumatic tissue loss). In this

context, perturbation has a slightly different role to play. Here we aim

 to build a CA model that not only has regenerative properties, but also

Biologically, this corresponds to a teratogenic influence during

development, or alternatively, a case of an incorrect or incomplete

remodeling event such as metamorphosis or rescaling. The distinction

between training our model from scratch and training it to accommodate

perturbations is subtle but important. An important feature of life is

the ability to react adaptively to external perturbations that are not

accounted for in the normal developmental sequence of events. If our

virtual cells simply learned to recognize a digit and then entered some

dormant state and did not react to any further changes, we would be

missing this key property of living organisms. One could imagine a

trivial solution in the absence of perturbations, where a single wave of

 information is passed from the boundaries of the digit inwards and then

 back out, in such a way that all cells could agree on a correct

classification. By introducing perturbations to new digits, the cells

have to be in constant communication and achieve a "dynamic homeostasis"

 - continually "kept on their toes" in anticipation of new or further

communication from their neighbours.

In our model, we achieve this dynamic homeostasis by randomly

mutating the underlying digit at training time. Starting from a certain

digit and after some time evolution, we sample a new digit, erase all

cell states that are not present in both digits and bring alive the

cells that were not present in the original digit but are present in the

 new digit. This kind of mutation teaches CAs to learn to process new

information and adapt to changing conditions. It also exposes the cells

to training states where all of the cells that remain after a

perturbation are misclassifying the new digit and must recover from this

 catastrophic mutation. This in turn forces our CAs to learn to change

their own classifications to adapt to changing global structure.

We use a pixel-wise (cell-wise) cross entropy loss on the last ten

channels of each pixel, applying it after letting the CA evolve for 20

steps.

 Your browser does not support the video tag.

A first attempt at having the neural CAs classify digits. Each digit is a

 separate evolution of the neural CA, with the visualisations collated.

Halfway through, the underlying digit is swapped for a new one - a

"mutation".

The video above shows the CA classifying a batch of digits for 200

steps. We then mutate the digits and let the system evolve and classify

for a further 200 steps.

The results look promising overall and we can see how our CAs are

able to recover from mutations. However, astute observers may notice

that often not all cells agree with each other. Often, the majority of

the digit is classified correctly, but some outlier cells are still

convinced they are part of a different digit, often switching back and

forth in an oscillating pattern, causing a flickering effect in the

visualization. This is not ideal, since we would like the population of

cells to reach stable, total, agreement. The next experiment

troubleshoots this undesired behaviour.

Experiment 2: Stabilizing classification

----------------------------------------

 which we define as the mean percentage of cells that have a correct

output. We track this metric both before and after mutation.

Average accuracy across the cells in a digit over time.

In the figure above, we show the mean percentage of correctly

classified pixels in the test set over the course of 400 steps. At step

200, we randomly mutate the digit. Accordingly, we see a brief drop in

accuracy as the cells re-organise and eventually come to agreement on

what the new digit is.

We immediately notice an interesting phenomenon: the cell accuracy

appears to decrease over time after the cells have come to an agreement.

 However, the graph does not necessarily reflect the qualitative issue

of unstable labels that we set out to solve. The slow decay in accuracy

may be a reflection of the lack of total agreement, but doesn't capture

the stark instability issue.

Average total agreement among cells across the test set in MNIST, over time.

This metric does a better job of capturing the issues we are seeing.

The total agreement starts at zero and then spikes up to roughly 78%,

only to lose more than 10% agreement over the next 100 steps. Again,

behaviour after mutation does not appear to be significantly different.

Our model is not only unstable in the short term, exhibiting flickering,

 but is also unstable over longer timescales. As time goes on, cells are

 becoming less sure of themselves. Let's inspect the inner states of the

 CA to see why this is happening.

The figure above shows the time evolution of the average magnitude of

 the state values of active cells (solid line), and the average

magnitude of the residual updates (dotted line). Two important things

are happening here: 1) the average magnitude of each cell's internal

states is growing monotonically on this timescale; 2) the average

magnitude of the residual updates is staying roughly constant. We

theorize that, unlike 1), a successful CA model should stabilize the

magnitude of its internal states once cells have reached an agreement.

In order for this to happen, its residual updates should approach zero

over time, unlike what we observed in 2).

**Using an** **L2L\_2L2** **loss.** One

 problem with cross entropy loss is that it tends to push raw logit

values indefinitely higher. Another problem is that two sets of logits

can have vastly different values but essentially the same prediction

over classes. As such, training the CA with cross-entropy loss neither

requires nor encourages a shared reference range for logit values,

making it difficult for the cells to effectively communicate and

stabilize. Finally, we theorize that large magnitudes in the

classification channels may in turn lead the remaining

(non-classification) state channels to transition to a high magnitude

regime. More specifically, we believe that *cross-entropy loss causes

unbounded growth in classification logits, which prevents residual

updates from approaching zero, which means that neighboring cells

 loss and use one-hot vectors as targets. Intuitively, this solution

should be more stable since the raw state channels for classification

are never pushed out of the range [0,1][0, 1][0,1]

 and a properly classified digit in a cell will have exactly one

classification channel set to 1 and the rest to 0. In summary, an L2L\_2L2

 loss should decrease the magnitude of all the internal state channels

while keeping the classification targets in a reasonable range.

 Here we add noise to the residual updates by sampling from a normal

 Your browser does not support the video tag.

The video above shows a batch of runs with the augmentations in

place. Qualitatively, the result looks much better as there is less

flickering and more total agreement. Let's check the quantitative

metrics to see if they, too, show improvement.

Comparison of average accuracy and total agreement

 when using cross-entropy and when using L2L\_2L2 loss.

| Model | Top accuracy | Accuracy at 200  | Top agreement | Agreement at 200 |

| --- | --- | --- | --- | --- |

| CE | **96.2 at 80** | **95.3** | 77.9 at 80 | 66.2 |

| L2L\_2L2 | 95.0 at 95 | 94.7 | 85.5 at 175 | 85.2 |

| L2L\_2L2 + Noise | 95.4 at 65 | **95.3** | **88.2 at 190** | **88.1** |

The figure and table above show that cross-entropy achieves the

highest accuracy of all models at roughly 80 steps. However, the

### Internal states

 loss stabilizes the magnitude of the states, and how residual updates

quickly decay to small values as the system nears agreement.

 Your browser does not support the video tag.

Visualisation of internal state channel values during mutations. Note

the accelerated timeline after a few seconds showing the relative

stability of the channel values.

To further validate our results, we can visualize the dynamics of the

 internal states of the final model. For visualization purposes, we have

 as most state values are less than one but a few are much larger. The

states converge to stable configurations quickly and the state channels

exhibit spatial continuity with the neighbouring states. More

specifically, we don't see any stark discontinuities in state values of

neighbouring pixels. Applying a mutation causes the CA to readapt to the

 new shape and form a new classification in just a few steps, after

which its internal values are stable.

Robustness

----------

Recall that during training we used random digit mutations to ensure

that the resulting CA would be responsive to external changes. This

allowed us to learn a dynamical system of agents which interact to

produce stable behavior at the population level, even when perturbed to

form a different digit from the original. Biologically, this model helps

 us understand the mutation insensitivity of some large-scale anatomical

 control mechanisms. For example, planaria continuously accumulate

mutations over millions of years of somatic inheritance but still always

 regenerate the correct morphology in nature (and exhibit no genetic

strains with new morphologies) .

This robustness to change was critically important to our interactive

 demo, since the cells needed to reclassify drawings as the user changed

 them. For example, when the user converted a six to an eight, the cells

 needed to quickly re-classify themselves to an eight. We encourage the

reader to play with the interactive demo and experience this for

themselves. In this section, we want to showcase a few behaviours we

found interesting.

 Your browser does not support the video tag.

The video above shows how the CA is able to interactively adjust to

our own writing and to change classification when the drawing is

updated.

### Robustness to out-of-distribution shapes

In the field of machine learning, researchers take great interest in

how their models perform on out-of-distribution data. In the

experimental sections of this article, we evaluated our model on the

test set of MNIST. In this section, we go further and examine how the

model reacts to digits drawn by us and not sampled from MNIST at all. We

 vary the shapes of the digits until the model is no longer capable of

classifying them correctly. Every classification model inherently

contains certain inductive biases that render them more or less robust

to generalizing to out-of-distribution data. Our model can be seen as a

recurrent convolutional model and thus we expect it to exhibit some of

the key properties of traditional convolutional models such as

translation invariance. However, we strongly believe that the

self-organising nature of this model introduces a novel inductive bias

which may have interesting properties of its own. Biology offers

examples of "repairing to novel configurations": 2-headed planaria, once

 created, regenerate to this new configuration which was not present in

the evolutionary "training set" .

 Your browser does not support the video tag.

Demonstration of some of the failure cases of the CA.

Above, we can see that our CA fails to classify some variants of 1

and 9. This is likely because MNIST training data is not sufficiently

representative of all writing styles. We hypothesize that more varied

and extensive datasets would improve performance. The model often

oscillates between two attractors (of competing digit labels) in these

situations. This is interesting because this behavior could not arise

from static classifiers such as traditional convolutional neural

networks.

 Your browser does not support the video tag.

By construction, our CA is translation invariant. But perhaps

surprisingly, we noticed that our model is also scale-invariant for

out-of-distribution digit sizes up to a certain point. Alas, it does not

 generalize well enough to classify digits of arbitrary lengths and

widths.

 Your browser does not support the video tag.

Demonstration of the behaviour of the model with chimeric configurations.

It is also interesting to see how our CA classifies "chimeric

digits", which are shapes composed of multiple digits. First, when

creating a 3-5 chimera, the classification of 3 appears to dominate that

 of the 5. Second, when creating a 8-9 chimera, the CAs reach an

oscillating attractor where sections of the two digits are correctly

classified. Third, when creating a 6-9 chimera, the CAs converge to an

oscillating attractor but the 6 is misclassified as a 4.

These phenomena are important in biology as scientists begin to develop

predictive models for the morphogenetic outcome of chimeric cell

collectives. We still do not have a framework for knowing in advance

what anatomical structures will form from a combination of, for example

leg-and-tail blastema cells in an axolotl, heads of planaria housing

stem cells from species with different head shapes, or composite embryos

 consisting of, for example, frog and axolotl blastomeres .

 Likewise, designing information signals that induce the emergence of

desired tissue patterns from a chimeric cellular collective, in vitro or

 in vivo, remains an open problem.

Related Work

------------

This article is follow-up work to Growing Neural Cellular Automata ,

 and it is meant to be read after the latter. In this article, we

purposefully skim over details of the original model and we refer the

**MNIST and CA.** Since CAs are easy to apply to two

dimensional grids, many researchers wondered if they could use them to

somehow classify the MNIST dataset. We are aware of work that combines

 To the best of our knowledge, we are the first to train end-to-end

differentiable Neural CAs for classification purposes and we are the

first to introduce the self-classifying variant of MNIST wherein each

pixel in the digit needs to coordinate locally in order to reach a

global agreement about its label.

Discussion

----------

This article serves as a proof-of-concept for how simple

self-organising systems such as CA can be used for classification when

trained end-to-end through backpropagation.

Our model adapts to writing and erasing and is surprisingly robust to

 certain ranges of digit stretching and brush widths. We hypothesize

that self-organising models with constrained capacity may be inherently

robust and have good generalisation properties. We encourage future work

 to test this hypothesis.

From a biological perspective, our work shows we can teach things to a

 collective of cells that they could not learn individually (by training

 or engineering a single cell). Training cells in unison (while

communicating with each other) allows them to learn more complex

behaviour than any attempt to train them one by one, which has important

 implications for strategies in regenerative medicine. The current focus

 on editing individual cells at the genetic or molecular signaling level

 faces fundamental barriers when trying to induce desired complex,

system-level outcomes (such as regenerating or remodeling whole organs).

 The inverse problem of determining which cell-level rules (e.g.,

genetic information) must be changed to achieve a global outcome is very

 difficult. In contrast and complement to this approach, we show the

first component of a roadmap toward developing effective strategies for

communication with cellular collectives. Future advances in this field

may be able to induce desired outcomes by using stimuli at the system's

input layer (experience), not hardware rewiring, to re-specify outcomes

at the tissue, organ, or whole-body level .

![](Self-classifying%20MNIST%20Digits_files/multiple-pages.svg)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Growing Neural Cellular Automata](https://distill.pub/2020/growing-ca/)

[Self-Organising Textures](https://distill.pub/selforg/2021/textures/)

Growing Neural Cellular Automata

The transformation of a tail into limb after xenoplastic transplantation

Normalized
 shape and location of perturbed craniofacial structures in the Xenopus
tadpole reveal an innate ability to achieve correct morphology

Top-down models in biology: explanation and control of complex living systems above the molecular level

On Having No Head: Cognition throughout Biological Systems

The biogenic approach to cognition

Gradient-based learning applied to document recognition

MATHEMATICAL GAMES

Auto-Encoding Variational Bayes

Practical Variational Inference for Neural Networks

Noisy Networks for Exploration

Planarian regeneration as a model of anatomical homeostasis: Recent progress in biophysical and computational approaches

Long-range neural and gap junction protein-mediated cues control polarity during planarian regeneration

Bioelectrical Mechanisms for Programming Growth and Form: Taming Physiological Networks for Soft Body Robotics

Interspecies chimeras

Reservoir Computing Hardware with Cellular Automata

Energy-Efficient Pattern Recognition Hardware With Elementary Cellular Automata

Asynchronous network of cellular automaton-based neurons for efficient implementation of Boltzmann machines

An Approach to Searching for Two-Dimensional Cellular Automata for Recognition of Handwritten Digits

Biologically inspired cellular automata learning and prediction model for handwritten pattern recognition

Pattern Classification with Rejection Using Cellular Automata-Based Filtering

The body electric 2.0: recent advances in developmental bioelectricity for regenerative and synthetic bioengineering

Re-membering
 the body: applications of computational neuroscience to the top-down
control of regeneration of limbs and other complex organs