### Contents

[Introduction](#introduction)
[The Challenges of Computation on Graphs](#challenges)
* [Lack of Consistent Structure](#lack-of-consistent-structure)
* [Node-Order Equivariance](#node-order)
* [Scalability](#scalability)

[Problem Setting and Notation](#problem-and-notation)
[Extending Convolutions to Graphs](#extending)
[Polynomial Filters on Graphs](#polynomial-filters)
[Modern Graph Neural Networks](#modern-gnns)
[Interactive Graph Neural Networks](#interactive)
[From Local to Global Convolutions](#from-local-to-global)
* [Spectral Convolutions](#spectral)
* [Global Propagation via Graph Embeddings](#graph-embeddings)

[Learning GNN Parameters](#learning)

[Conclusions and Further Reading](#further-reading)
* [GNNs in Practice](#practical-techniques)
* [Different Kinds of Graphs](#different-kinds-of-graphs)
* [Pooling](#pooling)

[Supplementary Material](#supplementary)
* [Reproducing Experiments](#experiments-notebooks)
* [Recreating Visualizations](#visualizations-notebooks)

*This article is one of two Distill publications about graph neural networks.
 Take a look at
 [A Gentle Introduction to Graph Neural Networks](https://distill.pub/2021/gnn-intro/)

 for a companion view on many things graph and neural network related.*

 Many systems and interactions - social networks, molecules, organizations, citations, physical models, transactions - can be represented quite naturally as graphs.
 How can we reason about and make predictions within these systems?

 One idea is to look at tools that have worked well in other domains: neural networks have shown immense predictive power in a variety of learning tasks.
 However, neural networks have been traditionally used to operate on fixed-size and/or regular-structured inputs (such as sentences, images and video).
 This makes them unable to elegantly process graph-structured data.

![Neural networks generally operate on fixed-size input vectors. How do we input a graph to a neural network?](images/standard-neural-networks.svg "How do we input a graph to a neural network?")

 Graph neural networks (GNNs) are a family of neural networks that can operate naturally on graph-structured data.
 By extracting and utilizing features from the underlying graph,
 GNNs can make more informed predictions about entities in these interactions,
 as compared to models that consider individual entities in isolation.

 GNNs are not the only tools available to model graph-structured data:
 graph kernels
 and random-walk methods
 were some of the most popular ones.
 Today, however, GNNs have largely replaced these techniques
 because of their inherent flexibility to model the underlying systems
 better.

 In this article, we will illustrate
 the challenges of computing over graphs,
 describe the origin and design of graph neural networks,
 and explore the most popular GNN variants in recent times.
 Particularly, we will see that many of these variants
 are composed of similar building blocks.

 First, let’s discuss some of the complications that graphs come with.

 The Challenges of Computation on Graphs
-----------------------------------------

###
 Lack of Consistent Structure

 Graphs are extremely flexible mathematical models; but this means they lack consistent structure across instances.
 Consider the task of predicting whether a given chemical molecule is toxic  :

![The molecular structure of non-toxic 1,2,6-trigalloyl-glucose.](images/1,2,6-trigalloyl-glucose-molecule.svg)
![The molecular structure of toxic caramboxin.](images/caramboxin-molecule.svg)

**Left:** A non-toxic 1,2,6-trigalloyl-glucose molecule.

**Right:** A toxic caramboxin molecule.

 Looking at a few examples, the following issues quickly become apparent:

* Molecules may have different numbers of atoms.
* The atoms in a molecule may be of different types.
* Each of these atoms may have different number of connections.
* These connections can have different strengths.

 Representing graphs in a format that can be computed over is non-trivial,
 and the final representation chosen often depends significantly on the actual problem.

###
 Node-Order Equivariance

 Extending the point above: graphs often have no inherent ordering present amongst the nodes.
 Compare this to images, where every pixel is uniquely determined by its absolute position within the image!

![Representing the graph as one vector requires us to fix an order on the nodes. But what do we do when the nodes have no inherent order?](images/node-order-alternatives.svg)

 Representing the graph as one vector requires us to fix an order on the nodes.
 But what do we do when the nodes have no inherent order?
 **Above:**
 The same graph labelled in two different ways. The alphabets indicate the ordering of the nodes.

 As a result, we would like our algorithms to be node-order equivariant:
 they should not depend on the ordering of the nodes of the graph.
 If we permute the nodes in some way, the resulting representations of
 the nodes as computed by our algorithms should also be permuted in the same way.

###
 Scalability

 Graphs can be really large! Think about social networks like Facebook and Twitter, which have over a billion users.
 Operating on data this large is not easy.

 Luckily, most naturally occuring graphs are ‘sparse’:
 they tend to have their number of edges linear in their number of vertices.
 We will see that this allows the use of clever methods
 to efficiently compute representations of nodes within the graph.
 Further, the methods that we look at here will have significantly fewer parameters
 in comparison to the size of the graphs they operate on.

 Problem Setting and Notation
------------------------------

 There are many useful problems that can be formulated over graphs:

* **Node Classification:** Classifying individual nodes.
* **Graph Classification:** Classifying entire graphs.
* **Node Clustering:** Grouping together similar nodes based on connectivity.
* **Link Prediction:** Predicting missing links.
* **Influence Maximization:** Identifying influential nodes.

![Examples of problems that can be defined over graphs.](images/graph-tasks.svg)

 Examples of problems that can be defined over graphs.
 This list is not exhaustive!

 A common precursor in solving many of these problems is **node representation learning**:
 learning to map individual nodes to fixed-size real-valued vectors (called ‘representations’ or ‘embeddings’).

 In [Learning GNN Parameters](#learning), we will see how the learnt embeddings can be used for these tasks.

 Different GNN variants are distinguished by the way these representations are computed.
 Generally, however, GNNs compute node representations in an iterative process.
 We will use the notation hv(k)h\_v^{(k)}hv(k) to indicate the representation of node vvv after the kthk^{\text{th}}kth iteration.
 Each iteration can be thought of as the equivalent of a ‘layer’ in standard neural networks.

 We will define a graph GGG as a set of nodes, VVV, with a set of edges EEE connecting them.
 Nodes can have individual features as part of the input: we will denote by xvx\_vxv the individual feature for node v∈Vv \in Vv∈V.
 For example, the ‘node features’ for a pixel in a color image
 would be the red, green and blue channel (RGB) values at that pixel.

 For ease of exposition, we will assume GGG is undirected, and all nodes are of the same type.
 These kinds of graphs are called ‘homogeneous’.
 Many of the same ideas we will see here
 apply to other kinds of graphs:
 we will discuss this later in [Different Kinds of Graphs](#different-kinds-of-graphs).

 Sometimes we will need to denote a graph property by a matrix MMM,
 where each row MvM\_vMv represents a property corresponding to a particular vertex vvv.

 Extending Convolutions to Graphs
----------------------------------

 Convolutional Neural Networks have been seen to be quite powerful in extracting features from images.
 However, images themselves can be seen as graphs with a very regular grid-like structure,
 where the individual pixels are nodes, and the RGB channel values at each pixel as the node features.

 A natural idea, then, is to consider generalizing convolutions to arbitrary graphs. Recall, however, the challenges
 listed out in the [previous section](#challenges): in particular, ordinary convolutions are not node-order invariant, because
 they depend on the absolute positions of pixels.
 It is initially unclear as how to generalize convolutions over grids to convolutions over general graphs,
 where the neighbourhood structure differs from node to node.

 The curious reader may wonder if performing some sort of padding and ordering
 could be done to ensure the consistency of neighbourhood structure across nodes.
 This has been attempted with some success ,
 but the techniques we will look at here are more general and powerful.

 Convolutions in CNNs are inherently localized.
 Neighbours participating in the convolution at the center pixel are highlighted in gray.

 GNNs can perform localized convolutions mimicking CNNs.
 Hover over a node to see its immediate neighbourhood highlighted on the left.
 The structure of this neighbourhood changes from node to node.

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";
 import define from "./notebooks/neighbourhoods-for-cnns-and-gnns.js";
 setTimeout(() => {
 new Runtime().module(define, name => {
 if (name === "cnn\_svg") return new Inspector(document.querySelector("#observablehq-cnn\_svg-35509536"));
 if (name === "svg") return new Inspector(document.querySelector("#observablehq-svg-35509536"));
 return ["adjust\_dimensions","reset\_nodes","highlight\_nodes","get\_node\_position","remove\_old\_arrows","draw\_arrows\_to\_updated\_node","add\_interactivity","on\_selected\_node\_change","updated\_node\_position"].includes(name);
 });
 }, 200);

 We begin by introducing the idea of constructing polynomial filters over node neighbourhoods,
 much like how CNNs compute localized filters over neighbouring pixels.
 Then, we will see how more recent approaches extend on this idea with more powerful mechanisms.
 Finally, we will discuss alternative methods
 that can use ‘global’ graph-level information for computing node representations.

 Polynomial Filters on Graphs
------------------------------

###
 The Graph Laplacian

 Given a graph GGG, let us fix an arbitrary ordering of the nnn nodes of GGG.
 We denote the 0−10-10−1 adjacency matrix of GGG by AAA, we can construct the diagonal degree matrix DDD of GGG as:

Dv=∑uAvu.
 D\_v = \sum\_u A\_{vu}.
 Dv=u∑Avu.

 The degree of node vvv is the number of edges incident at vvv.

 where AvuA\_{vu}Avu denotes the entry in the row corresponding to vvv and the column corresponding to uuu
 in the matrix AAA. We will use this notation throughout this section.

 Then, the graph Laplacian LLL is the square n×nn \times nn×n matrix defined as:
 L=D−A.
 L = D - A.
 L=D−A.

![](images/laplacian.svg)

 The Laplacian LLL for an undirected graph GGG, with the row corresponding to node C\textsf{C}C highlighted.
 Zeros in LLL are not displayed above.
 The Laplacian LLL depends only on the structure of the graph GGG, not on any node features.

 The graph Laplacian gets its name from being the discrete analog of the
 [Laplacian operator](https://mathworld.wolfram.com/Laplacian.html)
 from calculus.

 Although it encodes precisely the same information as the adjacency matrix AAA

 In the sense that given either of the matrices AAA or LLL, you can construct the other.
 ,
 the graph Laplacian has many interesting properties of its own.

 The graph Laplacian shows up in many mathematical problems involving graphs:
 [random walks](https://people.math.sc.edu/lu/talks/nankai_2014/spec_nankai_2.pdf),
 [spectral clustering](https://arxiv.org/abs/0711.0189),
 and
 [diffusion](https://www.math.fsu.edu/~bertram/lectures/Diffusion.pdf), to name a few.

 We will see some of these properties
 in [a later section](#spectral),
 but will instead point readers to
 [this tutorial](https://csustan.csustan.edu/~tom/Clustering/GraphLaplacian-tutorial.pdf)
 for greater insight into the graph Laplacian.

###
 Polynomials of the Laplacian

 Now that we have understood what the graph Laplacian is,
 we can build polynomials  of the form:
 pw(L)=w0In+w1L+w2L2+…+wdLd=∑i=0dwiLi.
 p\_w(L) = w\_0 I\_n + w\_1 L + w\_2 L^2 + \ldots + w\_d L^d = \sum\_{i = 0}^d w\_i L^i.
 pw(L)=w0In+w1L+w2L2+…+wdLd=i=0∑dwiLi.
 Each polynomial of this form can alternately be represented by
 its vector of coefficients w=[w0,…,wd]w = [w\_0, \ldots, w\_d]w=[w0,…,wd].
 Note that for every www, pw(L)p\_w(L)pw(L) is an n×nn \times nn×n matrix, just like LLL.

 These polynomials can be thought of as the equivalent of ‘filters’ in CNNs,
 and the coefficients www as the weights of the ‘filters’.

 For ease of exposition, we will focus on the case where nodes have one-dimensional features:
 each of the xvx\_vxv for v∈Vv \in Vv∈V is just a real number.
 The same ideas hold when each of the xvx\_vxv are higher-dimensional vectors, as well.

 Using the previously chosen ordering of the nodes,
 we can stack all of the node features xvx\_vxv
 to get a vector x∈Rnx \in \mathbb{R}^nx∈Rn.

![Fixing a node order and collecting all node features into a single vector.](images/node-order-vector.svg)

 Fixing a node order (indicated by the alphabets) and collecting all node features into a single vector xxx.

 Once we have constructed the feature vector xxx,
 we can define its convolution with a polynomial filter pwp\_wpw as:
 x′=pw(L) x
 x’ = p\_w(L) \ x
 x′=pw(L) x
 To understand how the coefficients www affect the convolution,
 let us begin by considering the ‘simplest’ polynomial:
 when w0=1w\_0 = 1w0=1 and all of the other coefficients are 000.
 In this case, x′x’x′ is just xxx:
 x′=pw(L) x=∑i=0dwiLix=w0Inx=x.
 x’ = p\_w(L) \ x = \sum\_{i = 0}^d w\_i L^ix = w\_0 I\_n x = x.
 x′=pw(L) x=i=0∑dwiLix=w0Inx=x.
 Now, if we increase the degree, and consider the case where
 instead w1=1w\_1 = 1w1=1 and and all of the other coefficients are 000.
 Then, x′x’x′ is just LxLxLx, and so:
 xv′=(Lx)v=Lvx=∑u∈GLvuxu=∑u∈G(Dvu−Avu)xu=Dv xv−∑u∈N(v)xu
 \begin{aligned}
 x’\_v = (Lx)\_v &= L\_v x \\
 &= \sum\_{u \in G} L\_{vu} x\_u \\
 &= \sum\_{u \in G} (D\_{vu} - A\_{vu}) x\_u \\
 &= D\_v \ x\_v - \sum\_{u \in \mathcal{N}(v)} x\_u
 \end{aligned}
 xv′=(Lx)v=Lvx=u∈G∑Lvuxu=u∈G∑(Dvu−Avu)xu=Dv xv−u∈N(v)∑xu
 We see that the features at each node vvv are combined
 with the features of its immediate neighbours u∈N(v)u \in \mathcal{N}(v)u∈N(v).

 For readers familiar with
 [Laplacian filtering of images](https://docs.opencv.org/3.4/d5/db5/tutorial_laplace_operator.html),
 this is the exact same idea. When xxx is an image,
 x′=Lxx’ = Lxx′=Lx is exactly the result of applying a ‘Laplacian filter’ to xxx.

 At this point, a natural question to ask is:
 How does the degree ddd of the polynomial influence the behaviour of the convolution?
 Indeed, it is not too hard to show that:
 This is Lemma 5.2 from .
distG(v,u)>i⟹Lvui=0.
 \text{dist}\_G(v, u) > i \quad \Longrightarrow \quad L\_{vu}^i = 0.
 distG(v,u)>i⟹Lvui=0.

 This implies, when we convolve xxx with pw(L)p\_w(L)pw(L) of degree ddd to get x′x’x′:
 xv′=(pw(L)x)v=(pw(L))vx=∑i=0dwiLvix=∑i=0dwi∑u∈GLvuixu=∑i=0dwi∑u∈GdistG(v,u)≤iLvuixu.
 \begin{aligned}
 x’\_v = (p\_w(L)x)\_v &= (p\_w(L))\_v x \\
 &= \sum\_{i = 0}^d w\_i L\_v^i x \\
 &= \sum\_{i = 0}^d w\_i \sum\_{u \in G} L\_{vu}^i x\_u \\
 &= \sum\_{i = 0}^d w\_i \sum\_{u \in G \atop \text{dist}\_G(v, u) \leq i} L\_{vu}^i x\_u.
 \end{aligned}
 xv′=(pw(L)x)v=(pw(L))vx=i=0∑dwiLvix=i=0∑dwiu∈G∑Lvuixu=i=0∑dwidistG(v,u)≤iu∈G∑Lvuixu.

 Effectively, the convolution at node vvv occurs only with nodes uuu which are not more than ddd hops away.
 Thus, these polynomial filters are localized. The degree of the localization is governed completely by ddd.

 To help you understand these ‘polynomial-based’ convolutions better, we have created the visualization below.
 Vary the polynomial coefficients and the input grid xxx to see how the result x′x’x′ of the convolution changes.
 The grid under the arrow shows the equivalent convolutional kernel applied at the highlighted pixel in xxx to get
 the resulting pixel in x′x’x′.
 The kernel corresponds to the row of pw(L)p\_w(L)pw(L) for the highlighted pixel.
 Note that even after adjusting for position,
 this kernel is different for different pixels, depending on their position within the grid.

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";
 import define from "./notebooks/cleaner-interactive-graph-polynomial-convolutions.js";
 setTimeout(() => {
 new Runtime().module(define, name => {
 if (name === "grid\_buttons\_display") return new Inspector(document.querySelector("#observablehq-grid\_buttons\_display-05850d43"));
 if (name === "poly\_color\_scale") return new Inspector(document.querySelector("#observablehq-poly\_color\_scale-05850d43"));
 if (name === "poly\_figcaptions") return new Inspector(document.querySelector("#observablehq-poly\_figcaptions-05850d43"));
 if (name === "poly\_conv\_main\_div") return new Inspector(document.querySelector("#observablehq-poly\_conv\_main\_div-05850d43"));
 if (name === "viewof laplacian\_type") return new Inspector(document.querySelector("#observablehq-viewof-laplacian\_type-05850d43"));
 if (name === "polynomial\_display") return new Inspector(document.querySelector("#observablehq-polynomial\_display-05850d43"));
 if (name === "poly\_conv\_sliders") return new Inspector(document.querySelector("#observablehq-poly\_conv\_sliders-05850d43"));
 if (name === "highlight\_selected\_cell") return new Inspector(document.querySelector("#observablehq-highlight\_selected\_cell-05850d43"));
 if (name === "reset\_coeffs\_button\_display") return new Inspector(document.querySelector("#observablehq-reset\_coeffs\_button\_display-05850d43"));
 if (name === "poly\_input\_slider\_watch") return new Inspector(document.querySelector("#observablehq-poly\_input\_slider\_watch-05850d43"));
 return ["svg","draw\_bottom\_line","draw\_arrow","draw\_original\_img","draw\_convolutional\_kernel","draw\_updated\_img","draw\_static\_graph\_orig","draw\_static\_graph\_upd","draw\_dyn\_graph\_orig","draw\_dyn\_graph\_upd"].includes(name);
 });
 }, 200);

 Hover over a pixel in the input grid (left, representing xxx)
 to highlight it and see the equivalent convolutional kernel
 for that pixel under the arrow.
 The result x′x’x′ of the convolution is shown on the right:
 note that different convolutional kernels are applied at different pixels,
 depending on their location.

 Click on the input grid to toggle pixel values between 000 (white) and 111 (blue).
 To randomize the input grid, press ‘Randomize Grid’. To reset all pixels to 000, press ‘Reset Grid’.
 Use the sliders at the bottom to change the coefficients www.
 To reset all coefficients www to 000, press ‘Reset Coefficients.’

###
 ChebNet

 ChebNet  refines this idea of polynomial filters by looking at polynomial filters of the form:

pw(L)=∑i=1dwiTi(L~)
 p\_w(L) = \sum\_{i = 1}^d w\_i T\_i(\tilde{L})
 pw(L)=i=1∑dwiTi(L~)

 where TiT\_iTi is the degree-iii
[Chebyshev polynomial of the first kind](https://en.wikipedia.org/wiki/Chebyshev_polynomials) and
 L~\tilde{L}L~ is the normalized Laplacian defined using the largest eigenvalue of LLL:

 We discuss the eigenvalues of the Laplacian LLL in more detail in [a later section](#spectral).

L~=2Lλmax(L)−In.
 \tilde{L} = \frac{2L}{\lambda\_{\max}(L)} - I\_n.
 L~=λmax(L)2L−In.

 What is the motivation behind these choices?

* LLL is actually positive semi-definite: all of the eigenvalues of LLL are not lesser than 000.
 If λmax(L)>1\lambda\_{\max}(L) > 1λmax(L)>1, the entries in the powers of LLL rapidly increase in size.
 L~\tilde{L}L~ is effectively a scaled-down version of LLL, with eigenvalues guaranteed to be in the range [−1,1][-1, 1][−1,1].
 This prevents the entries of powers of L~\tilde{L}L~ from blowing up.
 Indeed, in the [visualization above](#polynomial-convolutions): we restrict the higher-order coefficients
 when the unnormalized Laplacian LLL is selected, but allow larger values when the normalized Laplacian L~\tilde{L}L~ is selected,
 in order to show the result x′x’x′ on the same color scale.
* The Chebyshev polynomials have certain interesting properties that make interpolation more numerically stable.
 We won’t talk about this in more depth here,
 but will advise interested readers to take a look at  as a definitive resource.

###
 Polynomial Filters are Node-Order Equivariant

 The polynomial filters we considered here are actually independent of the ordering of the nodes.
 This is particularly easy to see when the degree of the polynomial pwp\_wpw is 111:
 where each node’s feature is aggregated with the sum of its neighbour’s features.
 Clearly, this sum does not depend on the order of the neighbours.
 A similar proof follows for higher degree polynomials:
 the entries in the powers of LLL are equivariant to the ordering of the nodes.

**Details for the Interested Reader**

 As above, let’s assume an arbitrary node-order over the nnn nodes of our graph.
 Any other node-order can be thought of as a permutation of this original node-order.
 We can represent any permutation by a
 [permutation matrix](https://en.wikipedia.org/wiki/Permutation_matrix) PPP.
 PPP will always be an orthogonal 0−10-10−1 matrix:
 PPT=PTP=In.
 PP^T = P^TP = I\_n.
 PPT=PTP=In.
 Then, we call a function fff node-order equivariant iff for all permutations PPP:
 f(Px)=Pf(x).
 f(Px) = P f(x).
 f(Px)=Pf(x).

 When switching to the new node-order using the permutation PPP,
 the quantities below transform in the following way:
 x→PxL→PLPTLi→PLiPT
 \begin{aligned}
 x &\to Px \\
 L &\to PLP^T \\
 L^i &\to PL^iP^T
 \end{aligned}
 xLLi→Px→PLPT→PLiPT
 and so, for the case of polynomial filters where f(x)=pw(L) xf(x) = p\_w(L) \ xf(x)=pw(L) x, we can see that:
 f(Px)=∑i=0dwi(PLiPT)(Px)=P∑i=0dwiLix=Pf(x).
 \begin{aligned}
 f(Px) & = \sum\_{i = 0}^d w\_i (PL^iP^T) (Px) \\
 & = P \sum\_{i = 0}^d w\_i L^i x \\
 & = P f(x).
 \end{aligned}
 f(Px)=i=0∑dwi(PLiPT)(Px)=Pi=0∑dwiLix=Pf(x).
 as claimed.

###
 Embedding Computation

 We now describe how we can build a graph neural network
 by stacking ChebNet (or any polynomial filter) layers
 one after the other with non-linearities,
 much like a standard CNN.
 In particular, if we have KKK different polynomial filter layers,
 the kthk^{\text{th}}kth of which has its own learnable weights w(k)w^{(k)}w(k),
 we would perform the following computation:

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";
 import define from "./notebooks/updated-chebnet-equations.js";
 setTimeout(() => {
 new Runtime().module(define, name => {
 if (name === "cheb\_figure") return new Inspector(document.querySelector("#observablehq-cheb\_figure-fa1f970f"));
 if (name === "style") return new Inspector(document.querySelector("#observablehq-style-fa1f970f"));
 });
 }, 200);

 Note that these networks
 reuse the same filter weights across different nodes,
 exactly mimicking weight-sharing in Convolutional Neural Networks (CNNs)
 which reuse weights for convolutional filters across a grid.

 Modern Graph Neural Networks
------------------------------

 ChebNet was a breakthrough in learning localized filters over graphs,
 and it motivated many to think of graph convolutions from a different perspective.

 We return back to the result of convolving xxx by the polynomial kernel pw(L)=Lp\_w(L) = Lpw(L)=L,
 focussing on a particular vertex vvv:

(Lx)v=Lvx=∑u∈GLvuxu=∑u∈G(Dvu−Avu)xu=Dv xv−∑u∈N(v)xu
 \begin{aligned}
 (Lx)\_v &= L\_v x \\
 &= \sum\_{u \in G} L\_{vu} x\_u \\
 &= \sum\_{u \in G} (D\_{vu} - A\_{vu}) x\_u \\
 &= D\_v \ x\_v - \sum\_{u \in \mathcal{N}(v)} x\_u
 \end{aligned}
 (Lx)v=Lvx=u∈G∑Lvuxu=u∈G∑(Dvu−Avu)xu=Dv xv−u∈N(v)∑xu

 As we noted before, this is a 111-hop localized convolution.
 But more importantly, we can think of this convolution as arising of two steps:

* Aggregating over immediate neighbour features xux\_uxu.
* Combining with the node’s own feature xvx\_vxv.

**Key Idea:**
 What if we consider different kinds of ‘aggregation’ and ‘combination’ steps,
 beyond what are possible using polynomial filters?

 By ensuring that the aggregation is node-order equivariant,
 the overall convolution becomes node-order equivariant.

 These convolutions can be thought of as ‘message-passing’ between adjacent nodes:
 after each step, every node receives some ‘information’ from its neighbours.

 By iteratively repeating the 111-hop localized convolutions KKK times (i.e., repeatedly ‘passing messages’),
 the receptive field of the convolution effectively includes all nodes upto KKK hops away.

###
 Embedding Computation

 Message-passing forms the backbone of many GNN architectures today.
 We describe the most popular ones in depth below:

* Graph Convolutional Networks (GCN)
* Graph Attention Networks (GAT)
* Graph Sample and Aggregate (GraphSAGE)
* Graph Isomorphism Network (GIN)

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";
 import define from "./notebooks/interactive-gnn-equations.js";
 setTimeout(() => {
 new Runtime().module(define, name => {
 if (name === "fig\_div") return Inspector.into(".interactive-gnn-equations-fig\_div")();
 if (name === "text\_div") return Inspector.into(".interactive-gnn-equations-text\_div")();
 if (name === "interactive\_list") return Inspector.into(".interactive-gnn-equations-interactive\_list")();
 if (name === "style") return Inspector.into(".interactive-gnn-equations-style")();
 });
 }, 200);

###
 Thoughts

 An interesting point is to assess different aggregation functions: are some better and others worse?
  demonstrates that aggregation functions indeed can be compared on how well
 they can uniquely preserve node neighbourhood features;
 we recommend the interested reader take a look at the detailed theoretical analysis there.

 Here, we’ve talk about GNNs where the computation only occurs at the nodes.
 More recent GNN models
 such as Message-Passing Neural Networks
 and Graph Networks
 perform computation over the edges as well;
 they compute edge embeddings together with node embeddings.
 This is an even more general framework -
 but the same ‘message passing’ ideas from this section apply.

 Interactive Graph Neural Networks
-----------------------------------

 Below is an interactive visualization of these GNN models on small graphs.
 For clarity, the node features are just real numbers here, shown inside the squares next to each node,
 but the same equations hold when the node features are vectors.

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";
 import define from "./notebooks/interactive-gnn-visualizations.js";
 setTimeout(() => {
 new Runtime().module(define, name => {
 if (name === "viz\_list") return Inspector.into(".interactive-gnn-visualizations-viz\_list")();
 if (name === "buttons") return Inspector.into(".interactive-gnn-visualizations-buttons")();
 if (name === "fig") return Inspector.into(".interactive-gnn-visualizations-fig")();
 if (name === "eqn") return Inspector.into(".interactive-gnn-visualizations-eqn")();
 if (name === "network\_display\_hack") return Inspector.into(".interactive-gnn-visualizations-network\_display\_hack")();
 if (name === "style") return Inspector.into(".interactive-gnn-visualizations-style")();
 return ["interactive\_list","select\_fig","handle\_click"].includes(name) || null;
 });
 }, 200);

 Choose a GNN model using the tabs at the top. Click on a node to see the update equation at that node for the next iteration.
 Use the sliders on the left to change the weights for the current iteration, and watch how the update equation changes.

 In practice, each iteration above is generally thought of as a single ‘neural network layer’.
 This ideology is followed by many popular Graph Neural Network libraries,

 For example: [PyTorch Geometric](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html)
 and [StellarGraph](https://stellargraph.readthedocs.io/en/stable/api.html#module-stellargraph.layer).

 allowing one to compose different types of graph convolutions in the same model.

 From Local to Global Convolutions
-----------------------------------

 The methods we’ve seen so far perform ‘local’ convolutions:
 every node’s feature is updated using a function of its local neighbours’ features.

 While performing enough steps of message-passing will eventually ensure that
 information from all nodes in the graph is passed,
 one may wonder if there are more direct ways to perform ‘global’ convolutions.

 The answer is yes; we will now describe an approach that was actually first put forward
 in the context of neural networks by ,
 much before any of the GNN models we looked at above.

###
 Spectral Convolutions

 As before, we will focus on the case where nodes have one-dimensional features.
 After choosing an arbitrary node-order, we can stack all of the node features to get a
 ‘feature vector’ x∈Rnx \in \mathbb{R}^nx∈Rn.

**Key Idea:**
 Given a feature vector xxx,
 the Laplacian LLL allows us to quantify how smooth xxx is, with respect to GGG.

 How?

 After normalizing xxx such that ∑i=1nxi2=1\sum\_{i = 1}^n x\_i^2 = 1∑i=1nxi2=1,
 if we look at the following quantity involving LLL:

RLR\_LRL is formally called the [Rayleigh quotient](https://en.wikipedia.org/wiki/Rayleigh_quotient).

RL(x)=xTLxxTx=∑(i,j)∈E(xi−xj)2∑ixi2=∑(i,j)∈E(xi−xj)2.
 R\_L(x) = \frac{x^T L x}{x^T x} = \frac{\sum\_{(i, j) \in E} (x\_i - x\_j)^2}{\sum\_i x\_i^2} = \sum\_{(i, j) \in E} (x\_i - x\_j)^2.
 RL(x)=xTxxTLx=∑ixi2∑(i,j)∈E(xi−xj)2=(i,j)∈E∑(xi−xj)2.
 we immediately see that feature vectors xxx that assign similar values to
 adjacent nodes in GGG (hence, are smooth) would have smaller values of RL(x)R\_L(x)RL(x).

LLL is a real, symmetric matrix, which means it has all real eigenvalues λ1≤…≤λn\lambda\_1 \leq \ldots \leq \lambda\_{n}λ1≤…≤λn.

 An eigenvalue λ\lambdaλ of a matrix AAA is a value
 satisfying the equation Au=λuAu = \lambda uAu=λu for a certain vector uuu, called an eigenvector.
 For a friendly introduction to eigenvectors,
 please see [this tutorial](http://www.sosmath.com/matrix/eigen0/eigen0.html).

 Further, the corresponding eigenvectors u1,…,unu\_1, \ldots, u\_{n}u1,…,un can be taken to be orthonormal:
 uk1Tuk2={1 if k1=k2.0 if k1≠k2.
 u\_{k\_1}^T u\_{k\_2} =
 \begin{cases}
 1 \quad \text{ if } {k\_1} = {k\_2}. \\
 0 \quad \text{ if } {k\_1} \neq {k\_2}.
 \end{cases}
 uk1Tuk2={1 if k1=k2.0 if k1≠k2.
 It turns out that these eigenvectors of LLL are successively less smooth, as RLR\_LRL indicates:
 This is the [min-max theorem for eigenvalues.](https://en.wikipedia.org/wiki/Min-max_theorem)
argminx, x⊥{u1,…,ui−1}RL(x)=ui.minx, x⊥{u1,…,ui−1}RL(x)=λi.
 \underset{x, \ x \perp \{u\_1, \ldots, u\_{i - 1}\}}{\text{argmin}} R\_L(x) = u\_i.
 \qquad
 \qquad
 \qquad
 \min\_{x, \ x \perp \{u\_1, \ldots, u\_{i - 1}\}} R\_L(x) = \lambda\_i.
 x, x⊥{u1,…,ui−1}argminRL(x)=ui.x, x⊥{u1,…,ui−1}minRL(x)=λi.
 The set of eigenvalues of LLL are called its ‘spectrum’, hence the name!
 We denote the ‘spectral’ decomposition of LLL as:
 L=UΛUT.
 L = U \Lambda U^T.
 L=UΛUT.
 where Λ\LambdaΛ is the diagonal matrix of sorted eigenvalues,
 and UUU denotes the matrix of the eigenvectors (sorted corresponding to increasing eigenvalues):
 Λ=[λ1⋱λn]U=[u1 ⋯ un].
 \Lambda = \begin{bmatrix}
 \lambda\_{1} & & \\
 & \ddots & \\
 & & \lambda\_{n}
 \end{bmatrix}
 \qquad
 \qquad
 \qquad
 \qquad
 U = \begin{bmatrix} \\ u\_1 \ \cdots \ u\_n \\ \end{bmatrix}.
 Λ=⎣⎡λ1⋱λn⎦⎤U=⎣⎡u1 ⋯ un⎦⎤.
 The orthonormality condition between eigenvectors gives us that UTU=IU^T U = IUTU=I, the identity matrix.
 As these nnn eigenvectors form a basis for Rn\mathbb{R}^nRn,
 any feature vector xxx can be represented as a linear combination of these eigenvectors:
 x=∑i=1nxi^ui=Ux^.
 x = \sum\_{i = 1}^n \hat{x\_i} u\_i = U \hat{x}.
 x=i=1∑nxi^ui=Ux^.
 where x^\hat{x}x^ is the vector of coefficients [x0,…xn][x\_0, \ldots x\_n][x0,…xn].
 We call x^\hat{x}x^ as the spectral representation of the feature vector xxx.
 The orthonormality condition allows us to state:
 x=Ux^⟺UTx=x^.
 x = U \hat{x} \quad \Longleftrightarrow \quad U^T x = \hat{x}.
 x=Ux^⟺UTx=x^.
 This pair of equations allows us to interconvert
 between the ‘natural’ representation xxx and the ‘spectral’ representation x^\hat{x}x^
 for any vector x∈Rnx \in \mathbb{R}^nx∈Rn.

###
 Spectral Representations of Natural Images

 As discussed before, we can consider any image as a grid graph, where each pixel is a node,
 connected by edges to adjacent pixels.
 Thus, a pixel can have either 3,5,3, 5,3,5, or 888 neighbours, depending on its location within the image grid.
 Each pixel gets a value as part of the image. If the image is grayscale, each value will be a single
 real number indicating how dark the pixel is. If the image is colored, each value will be a 333-dimensional
 vector, indicating the values for the red, green and blue (RGB) channels.
 We use the alpha channel as well in the visualization below, so this is actually RGBA.

 This construction allows us to compute the graph Laplacian and the eigenvector matrix UUU.
 Given an image, we can then investigate what its spectral representation looks like.

 To shed some light on what the spectral representation actually encodes,
 we perform the following experiment over each channel of the image independently:

* We first collect all pixel values across a channel into a feature vector xxx.
* Then, we obtain its spectral representation x^\hat{x}x^.
 x^=UTx
 \hat{x} = U^T x
 x^=UTx
* We truncate this to the first mmm components to get x^m\hat{x}\_mx^m.
 By truncation, we mean zeroing out all of the remaining n−mn - mn−m components of x^\hat{x}x^.
 This truncation is equivalent to using only the first mmm eigenvectors to compute the spectral representation.
 x^m=Truncatem(x^)
 \hat{x}\_m = \text{Truncate}\_m(\hat{x})
 x^m=Truncatem(x^)
* Then, we convert this truncated representation x^m\hat{x}\_mx^m back to the natural basis to get xmx\_mxm.
 xm=Ux^m
 x\_m = U \hat{x}\_m
 xm=Ux^m

 Finally, we stack the resulting channels back together to get back an image.
 We can now see how the resulting image changes with choices of mmm.
 Note that when m=nm = nm=n, the resulting image is identical to the original image,
 as we can reconstruct each channel exactly.

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";
 import define from "./notebooks/spectral-decompositions-of-natural-images.js";
 setTimeout(() => {
 new Runtime().module(define, name => {
 if (name === "spectralDecompositionsAll") return new Inspector(document.querySelector("#observablehq-spectralDecompositionsAll-59114e0b"));
 if (name === "updateArrowCaption") return new Inspector(document.querySelector("#observablehq-updateArrowCaption-59114e0b"));
 if (name === "drawArrow") return new Inspector(document.querySelector("#observablehq-drawArrow-59114e0b"));
 if (name === "drawCurrBaseImg") return new Inspector(document.querySelector("#observablehq-drawCurrBaseImg-59114e0b"));
 if (name === "drawCurrUpdImg") return new Inspector(document.querySelector("#observablehq-drawCurrUpdImg-59114e0b"));
 if (name === "style") return new Inspector(document.querySelector("#observablehq-style-59114e0b"));
 });
 }, 200);

 Use the radio buttons at the top to chose one of the four sample images.
 Each of these images has been taken from the ImageNet
 dataset and downsampled to 505050 pixels wide and 404040 pixels tall.
 As there are n=50×40=2000n = 50 \times 40 = 2000n=50×40=2000 pixels in each image, there are 200020002000 Laplacian eigenvectors.
 Use the slider at the bottom to change the number of spectral components to keep, noting how
 images get progressively blurrier as the number of components decrease.

 As mmm decreases, we see that the output image xmx\_mxm gets blurrier.
 If we decrease mmm to 111, the output image xmx\_mxm is entirely the same color throughout.
 We see that we do not need to keep all nnn components;
 we can retain a lot of the information in the image with significantly fewer components.

 We can relate this to the Fourier decomposition of images:
 the more eigenvectors we use, the higher frequencies we can represent on the grid.

 To complement the visualization above,
 we additionally visualize the first few eigenvectors on a smaller 8×88 \times 88×8 grid below.
 We change the coefficients of the first 101010 out of 646464 eigenvectors
 in the spectral representation
 and see how the resulting image changes:

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";
 import define from "./notebooks/interactive-spectral-conversions.js";
 setTimeout(() => {
 new Runtime().module(define, name => {
 if (name === "figcaptions") return new Inspector(document.querySelector("#observablehq-figcaptions-6ac8e785"));
 if (name === "spec\_conv\_main\_div") return new Inspector(document.querySelector("#observablehq-spec\_conv\_main\_div-6ac8e785"));
 if (name === "spec\_color\_scale") return new Inspector(document.querySelector("#observablehq-spec\_color\_scale-6ac8e785"));
 if (name === "subgrid\_main\_div") return new Inspector(document.querySelector("#observablehq-subgrid\_main\_div-6ac8e785"));
 if (name === "spec\_conv\_sliders") return new Inspector(document.querySelector("#observablehq-spec\_conv\_sliders-6ac8e785"));
 if (name === "spec\_conv\_buttons\_display") return new Inspector(document.querySelector("#observablehq-spec\_conv\_buttons\_display-6ac8e785"));
 if (name === "spec\_input\_slider\_watch") return new Inspector(document.querySelector("#observablehq-spec\_input\_slider\_watch-6ac8e785"));
 return ["svg","draw\_img","draw\_static\_graph","draw\_dyn\_graph","draw\_eigenvectors"].includes(name);
 });
 }, 200);

 Move the sliders to change the spectral representation x^\hat{x}x^ (right),
 and see how xxx itself changes on the image (left).
 Note how the first eigenvectors are much ‘smoother’ than the later ones,
 and the many patterns we can make with only 101010 eigenvectors.

 These visualizations should convince you that the first eigenvectors are indeed smooth,
 and the smoothness correspondingly decreases as we consider later eigenvectors.

 For any image xxx, we can think of
 the initial entries of the spectral representation x^\hat{x}x^
 as capturing ‘global’ image-wide trends, which are the low-frequency components,
 while the later entries as capturing ‘local’ details, which are the high-frequency components.

### Embedding Computation

 We now have the background to understand spectral convolutions
 and how they can be used to compute embeddings/feature representations of nodes.

 As before, the model we describe below has KKK layers:
 each layer kkk has learnable parameters w^(k)\hat{w}^{(k)}w^(k),
 called the ‘filter weights’.
 These weights will be convolved with the spectral representations of the node features.
 As a result, the number of weights needed in each layer is equal to mmm, the number of
 eigenvectors used to compute the spectral representations.
 We had shown in the previous section that we can take m≪nm \ll nm≪n
 and still not lose out on significant amounts of information.

 Thus, convolution in the spectral domain enables the use of significantly fewer parameters
 than just direct convolution in the natural domain.
 Further, by virtue of the smoothness of the Laplacian eigenvectors across the graph,
 using spectral representations automatically enforces an inductive bias for
 neighbouring nodes to get similar representations.

 Assuming one-dimensional node features for now,
 the output of each layer is a vector of node representations h(k)h^{(k)}h(k),
 where each node’s representation corresponds to a row
 of the vector.

 We fix an ordering of the nodes in GGG. This gives us the adjacency matrix AAA and the graph Laplacian LLL,
 allowing us to compute UmU\_mUm.
 Finally, we can describe the computation that the layers perform, one after the other:

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";
 import define from "./notebooks/spectral-convolutions-equation.js";
 setTimeout(() => {
 new Runtime().module(define, name => {
 if (name === "spec\_figure\_init") return Inspector.into(".spec\_figure\_init")();
 if (name === "spec\_figure") return Inspector.into(".spec\_figure")();
 if (name === "style") return Inspector.into(".spec\_figure\_style")();
 });
 }, 200);

 The method above generalizes easily to the case where each h(k)∈Rdkh^{(k)} \in \mathbb{R}^{d\_k}h(k)∈Rdk, as well:
 see  for details.

 With the insights from the previous section, we see that convolution in the spectral-domain of graphs
 can be thought of as the generalization of convolution in the frequency-domain of images.

###
 Spectral Convolutions are Node-Order Equivariant

 We can show spectral convolutions are node-order equivariant using a similar approach
 as for Laplacian polynomial filters.

**Details for the Interested Reader**

 As in [our proof before](#poly-filters-equivariance),
 let’s fix an arbitrary node-order.
 Then, any other node-order can be represented by a
 permutation of this original node-order.
 We can associate this permutation with its permutation matrix PPP.

 Under this new node-order,
 the quantities below transform in the following way:
 x→PxA→PAPTL→PLPTUm→PUm
 \begin{aligned}
 x &\to Px \\
 A &\to PAP^T \\
 L &\to PLP^T \\
 U\_m &\to PU\_m
 \end{aligned}
 xALUm→Px→PAPT→PLPT→PUm
 which implies that, in the embedding computation:
 x^→(PUm)T(Px)=UmTx=x^w^→(PUm)T(Pw)=UmTw=w^g^→g^g→(PUm)g^=P(Umg^)=Pg
 \begin{aligned}
 \hat{x} &\to \left(PU\_m\right)^T (Px) = U\_m^T x = \hat{x} \\
 \hat{w} &\to \left(PU\_m\right)^T (Pw) = U\_m^T w = \hat{w} \\
 \hat{g} &\to \hat{g} \\
 g &\to (PU\_m)\hat{g} = P(U\_m\hat{g}) = Pg
 \end{aligned}
 x^w^g^g→(PUm)T(Px)=UmTx=x^→(PUm)T(Pw)=UmTw=w^→g^→(PUm)g^=P(Umg^)=Pg
 Hence, as σ\sigmaσ is applied elementwise:
 f(Px)=σ(Pg)=Pσ(g)=Pf(x)
 f(Px) = \sigma(Pg) = P \sigma(g) = P f(x)
 f(Px)=σ(Pg)=Pσ(g)=Pf(x)
 as required.
 Further, we see that the spectral quantities x^,w^\hat{x}, \hat{w}x^,w^ and g^\hat{g}g^
 are unchanged by permutations of the nodes.

 Formally, they are what we would call node-order invariant.

 The theory of spectral convolutions is mathematically well-grounded;
 however, there are some key disadvantages that we must talk about:

* We need to compute the eigenvector matrix UmU\_mUm from LLL. For large graphs, this becomes quite infeasible.
* Even if we can compute UmU\_mUm, global convolutions themselves are inefficient to compute,
 because of the repeated
 multiplications with UmU\_mUm and UmTU\_m^TUmT.
* The learned filters are specific to the input graphs,
 as they are represented in terms
 of the spectral decomposition of input graph Laplacian LLL.
 This means they do not transfer well to new graphs
 which have significantly different structure (and hence, significantly
 different eigenvalues) .

 While spectral convolutions have largely been superseded by
 ‘local’ convolutions for the reasons discussed above,
 there is still much merit to understanding the ideas behind them.
 Indeed, a recently proposed GNN model called Directional Graph Networks

 actually uses the Laplacian eigenvectors
 and their mathematical properties
 extensively.

###
 Global Propagation via Graph Embeddings

 A simpler way to incorporate graph-level information
 is to compute embeddings of the entire graph by pooling node
 (and possibly edge) embeddings,
 and then using the graph embedding to update node embeddings,
 following an iterative scheme similar to what we have looked at here.
 This is an approach used by Graph Networks
 .
 We will briefly discuss how graph-level embeddings
 can be constructed in [Pooling](#pooling).
 However, such approaches tend to ignore the underlying
 topology of the graph that spectral convolutions can capture.

 Learning GNN Parameters
-------------------------

 All of the embedding computations we’ve described here, whether spectral or spatial, are completely differentiable.
 This allows GNNs to be trained in an end-to-end fashion, just like a standard neural network,
 once a suitable loss function L\mathcal{L}L is defined:

* **Node Classification**: By minimizing any of the standard losses for classification tasks,
 such as categorical cross-entropy when multiple classes are present:
 L(yv,yv^)=−∑cyvclogyvc^.
 \mathcal{L}(y\_v, \hat{y\_v}) = -\sum\_{c} y\_{vc} \log{\hat{y\_{vc}}}.
 L(yv,yv^)=−c∑yvclogyvc^.
 where yvc^\hat{y\_{vc}}yvc^ is the predicted probability that node vvv is in class ccc.
 GNNs adapt well to the semi-supervised setting, which is when only some nodes in the graph are labelled.
 In this setting, one way to define a loss LG\mathcal{L}\_{G}LG over an input graph GGG is:
 LG=∑v∈Lab(G)L(yv,yv^)∣Lab(G)∣
 \mathcal{L}\_{G} = \frac{\sum\limits\_{v \in \text{Lab}(G)} \mathcal{L}(y\_v, \hat{y\_v})}{| \text{Lab}(G) |}
 LG=∣Lab(G)∣v∈Lab(G)∑L(yv,yv^)
 where, we only compute losses over labelled nodes Lab(G)\text{Lab}(G)Lab(G).
* **Graph Classification**: By aggregating node representations,
 one can construct a vector representation of the entire graph.
 This graph representation can be used for any graph-level task, even beyond classification.
 See [Pooling](#pooling) for how representations of graphs can be constructed.
* **Link Prediction**: By sampling pairs of adjacent and non-adjacent nodes,
 and use these vector pairs as inputs to predict the presence/absence of an edge.
 For a concrete example, by minimizing the following ‘logistic regression’-like loss:
 L(yv,yu,evu)=−evulog(pvu)−(1−evu)log(1−pvu)pvu=σ(yvTyu)
 \begin{aligned}
 \mathcal{L}(y\_v, y\_u, e\_{vu}) &= -e\_{vu} \log(p\_{vu}) - (1 - e\_{vu}) \log(1 - p\_{vu}) \\
 p\_{vu} &= \sigma(y\_v^Ty\_u)
 \end{aligned}
 L(yv,yu,evu)pvu=−evulog(pvu)−(1−evu)log(1−pvu)=σ(yvTyu)
 where σ\sigmaσ is the [sigmoid function](https://en.wikipedia.org/wiki/Sigmoid_function),
 and evu=1e\_{vu} = 1evu=1 iff there is an edge between nodes vvv and uuu, being 000 otherwise.
* **Node Clustering**: By simply clustering the learned node representations.

 The broad success of pre-training for natural language processing models
 such as ELMo  and BERT
 has sparked interest in similar techniques for GNNs
 .
 The key idea in each of these papers is to train GNNs to predict
 local (eg. node degrees, clustering coefficient, masked node attributes)
 and/or global graph properties (eg. pairwise distances, masked global attributes).

 Another self-supervised technique is to enforce that neighbouring nodes get similar embeddings,
 mimicking random-walk approaches such as node2vec  and DeepWalk :

LG=∑v∑u∈NR(v)logexpzvTzu∑u′expzu′Tzu.
 L\_{G} = \sum\_{v} \sum\_{u \in N\_R(v)} \log\frac{\exp{z\_v^T z\_u}}{\sum\limits\_{u’} \exp{z\_{u’}^T z\_u}}.
 LG=v∑u∈NR(v)∑logu′∑expzu′TzuexpzvTzu.

 where NR(v)N\_R(v)NR(v) is a multi-set of nodes visited when random walks are started from vvv.
 For large graphs, where computing the sum over all nodes may be computationally expensive,
 techniques such as Noise Contrastive Estimation  are especially useful.

 Conclusion and Further Reading
--------------------------------

 While we have looked at many techniques and ideas in this article,
 the field of Graph Neural Networks is extremely vast.
 We have been forced to restrict our discussion to a small subset of the entire literature,
 while still communicating the key ideas and design principles behind GNNs.
 We recommend the interested reader take a look at
  for a more comprehensive survey.

 We end with pointers and references for additional concepts readers might be interested in:

###
 GNNs in Practice

 It turns out that accomodating the different structures of graphs is often hard to do efficiently,
 but we can still represent many GNN update equations using
 as sparse matrix-vector products (since generally, the adjacency matrix is sparse for most real-world graph datasets.)
 For example, the GCN variant discussed here can be represented as:
 h(k)=D−1A⋅h(k−1)W(k)T+h(k−1)B(k)T.
 h^{(k)} = D^{-1} A \cdot h^{(k - 1)} {W^{(k)}}^T + h^{(k - 1)} {B^{(k)}}^T.
 h(k)=D−1A⋅h(k−1)W(k)T+h(k−1)B(k)T.
 Restructuring the update equations in this way allows for efficient vectorized implementations of GNNs on accelerators
 such as GPUs.

 Regularization techniques for standard neural networks,
 such as Dropout ,
 can be applied in a straightforward manner to the parameters
 (for example, zero out entire rows of W(k)W^{(k)}W(k) above).
 However, there are graph-specific techniques such as DropEdge
 that removes entire edges at random from the graph,
 that also boost the performance of many GNN models.

###
 Different Kinds of Graphs

 Here, we have focused on undirected graphs, to avoid going into too many unnecessary details.
 However, there are some simple variants of spatial convolutions for:

* Directed graphs: Aggregate across in-neighbourhood and/or out-neighbourhood features.
* Temporal graphs: Aggregate across previous and/or future node features.
* Heterogeneous graphs: Learn different aggregation functions for each node/edge type.

 There do exist more sophisticated techniques that can take advantage of the different structures of these graphs:
 see  for more discussion.

###
 Pooling

 This article discusses how GNNs compute useful representations of nodes.
 But what if we wanted to compute representations of graphs for graph-level tasks (for example, predicting the toxicity of a molecule)?

 A simple solution is to just aggregate the final node embeddings and pass them through another neural network PREDICTG\text{PREDICT}\_GPREDICTG:
 hG=PREDICTG(AGGv∈G({hv}))
 h\_G = \text{PREDICT}\_G \Big( \text{AGG}\_{v \in G}\left(\{ h\_v \} \right) \Big)
 hG=PREDICTG(AGGv∈G({hv}))
 However, there do exist more powerful techniques for ‘pooling’ together node representations:

* SortPool: Sort vertices of the graph to get a fixed-size node-order invariant representation of the graph, and then apply any standard neural network architecture.
* DiffPool: Learn to cluster vertices, build a coarser graph over clusters instead of nodes, then apply a GNN over the coarser graph. Repeat until only one cluster is left.
* SAGPool: Apply a GNN to learn node scores, then keep only the nodes with the top scores, throwing away the rest. Repeat until only one node is left.

 Supplementary Material
------------------------

###
 Reproducing Experiments

 The experiments from
 [Spectral Representations of Natural Images](#spectral-decompositions-of-natural-images)
 can be reproduced using the following
 Colab ![Google Colaboratory](images/colab.svg) notebook:
 [Spectral Representations of Natural Images](https://colab.research.google.com/github/google-research/google-research/blob/master/understanding_convolutions_on_graphs/SpectralRepresentations.ipynb).

###
 Recreating Visualizations

 To aid in the creation of future interactive articles,
 we have created ObservableHQ
 ![ObservableHQ](images/observable.svg)
 notebooks for each of the interactive visualizations here:

* [Neighbourhood Definitions for CNNs and GNNs](https://observablehq.com/@ameyasd/neighbourhoods-for-cnns-and-gnns)
* [Graph Polynomial Convolutions on a Grid](https://observablehq.com/@ameyasd/cleaner-interactive-graph-polynomial-convolutions)
* [Graph Polynomial Convolutions: Equations](https://observablehq.com/@ameyasd/updated-chebnet-equations)
* [Modern Graph Neural Networks: Equations](https://observablehq.com/@ameyasd/interactive-gnn-equations)
* [Modern Graph Neural Networks: Interactive Models](https://observablehq.com/@ameyasd/interactive-gnn-visualizations)
 which pulls together the following standalone notebooks:
	+ [Graph Convolutional Networks](https://observablehq.com/@ameyasd/graph-convolutional-networks)
	+ [Graph Attention Networks](https://observablehq.com/@ameyasd/graph-attention-networks)
	+ [GraphSAGE](https://observablehq.com/@ameyasd/graph-sample-and-aggregate-graphsage)
	+ [Graph Isomorphism Networks](https://observablehq.com/@ameyasd/graph-isomorphism-networks)
* [Laplacian Eigenvectors for Grids](https://observablehq.com/@ameyasd/interactive-spectral-conversions)
* [Spectral Decomposition of Natural Images](https://observablehq.com/@ameyasd/spectral-decompositions-of-natural-images)
* [Spectral Convolutions: Equations](https://observablehq.com/@ameyasd/spectral-convolutions-equation)

Understanding Convolutions on Graphs

Ameya Daigavane
Balaraman Ravindran
Gaurav Aggarwal

Summary: Understanding the building blocks and design choices of graph neural networks.

Graph Kernels

Node2vec: Scalable Feature Learning for Networks

DeepWalk: Online Learning of Social Representations

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Neural Message Passing for Quantum Chemistry

A Tutorial on Spectral Clustering

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Wavelets on Graphs via Spectral Graph Theory

Chebyshev Polynomials

Semi-Supervised Classification with Graph Convolutional Networks

Graph Attention Networks

Inductive Representation Learning on Large Graphs

How Powerful are Graph Neural Networks?

Relational inductive biases, deep learning, and graph networks

Spectral Networks and Locally Connected Networks on Graphs

On the Transferability of Spectral Graph Filters

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Strategies for Pre-training Graph Neural Networks

Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes

When Does Self-Supervision Help Graph Convolutional Networks?

Self-supervised Learning on Graphs: Deep Insights and New Direction

Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

Learning word embeddings efficiently with noise-contrastive estimation

A Comprehensive Survey on Graph Neural Networks

Graph Neural Networks: A Review of Methods and Applications

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

An End-to-End Deep Learning Architecture for Graph Classification

Hierarchical Graph Representation Learning with Differentiable Pooling

Self-Attention Graph Pooling