Understanding Convolutions on Graphs

Ameya Daigavane
Balaraman Ravindran
Gaurav Aggarwal

Abstract: This article is one of two Distill publications about graph neural networks.         Take a look at         A Gentle Introduction to Graph Neural Networks          for a companion view on many things graph and neural network related.

### Contents

[Introduction](#introduction)

[The Challenges of Computation on Graphs](#challenges)

* [Lack of Consistent Structure](#lack-of-consistent-structure)

* [Node-Order Equivariance](#node-order)

* [Scalability](#scalability)

[Problem Setting and Notation](#problem-and-notation)

[Extending Convolutions to Graphs](#extending)

[Polynomial Filters on Graphs](#polynomial-filters)

[Modern Graph Neural Networks](#modern-gnns)

[Interactive Graph Neural Networks](#interactive)

[From Local to Global Convolutions](#from-local-to-global)

* [Spectral Convolutions](#spectral)

* [Global Propagation via Graph Embeddings](#graph-embeddings)

[Learning GNN Parameters](#learning)

[Conclusions and Further Reading](#further-reading)

* [GNNs in Practice](#practical-techniques)

* [Different Kinds of Graphs](#different-kinds-of-graphs)

* [Pooling](#pooling)

[Supplementary Material](#supplementary)

* [Reproducing Experiments](#experiments-notebooks)

* [Recreating Visualizations](#visualizations-notebooks)

*This article is one of two Distill publications about graph neural networks.

 Take a look at

 for a companion view on many things graph and neural network related.*

 Many systems and interactions - social networks, molecules,

organizations, citations, physical models, transactions - can be

represented quite naturally as graphs.

 How can we reason about and make predictions within these

systems?

 One idea is to look at tools that have worked well in other

domains: neural networks have shown immense predictive power in a

variety of learning tasks.

 However, neural networks have been traditionally used to operate

 on fixed-size and/or regular-structured inputs (such as sentences,

images and video).

 This makes them unable to elegantly process graph-structured

data.

 By extracting and utilizing features from the underlying graph,

 GNNs can make more informed predictions about entities in these interactions,

 as compared to models that consider individual entities in isolation.

 GNNs are not the only tools available to model graph-structured data:

 graph kernels

 and random-walk methods

 were some of the most popular ones.

 Today, however, GNNs have largely replaced these techniques

 because of their inherent flexibility to model the underlying systems

 better.

 In this article, we will illustrate

 the challenges of computing over graphs,

 describe the origin and design of graph neural networks,

 and explore the most popular GNN variants in recent times.

 Particularly, we will see that many of these variants

 are composed of similar building blocks.

 First, let’s discuss some of the complications that graphs come with.

 The Challenges of Computation on Graphs

-----------------------------------------

###

 Lack of Consistent Structure

 Consider the task of predicting whether a given chemical molecule is toxic  :

**Left:** A non-toxic 1,2,6-trigalloyl-glucose molecule.

**Right:** A toxic caramboxin molecule.

 Looking at a few examples, the following issues quickly become apparent:

* Molecules may have different numbers of atoms.

* The atoms in a molecule may be of different types.

* Each of these atoms may have different number of connections.

* These connections can have different strengths.

 Representing graphs in a format that can be computed over is non-trivial,

###

 Node-Order Equivariance

 Representing the graph as one vector requires us to fix an order on the nodes.

 But what do we do when the nodes have no inherent order?

 **Above:**

 As a result, we would like our algorithms to be node-order equivariant:

 they should not depend on the ordering of the nodes of the graph.

 If we permute the nodes in some way, the resulting representations of

###

 Scalability

 Operating on data this large is not easy.

 Luckily, most naturally occuring graphs are ‘sparse’:

 they tend to have their number of edges linear in their number of vertices.

 We will see that this allows the use of clever methods

 to efficiently compute representations of nodes within the graph.

 in comparison to the size of the graphs they operate on.

 Problem Setting and Notation

------------------------------

 There are many useful problems that can be formulated over graphs:

* **Node Classification:** Classifying individual nodes.

* **Graph Classification:** Classifying entire graphs.

* **Node Clustering:** Grouping together similar nodes based on connectivity.

* **Link Prediction:** Predicting missing links.

* **Influence Maximization:** Identifying influential nodes.

 Examples of problems that can be defined over graphs.

 This list is not exhaustive!

 Generally, however, GNNs compute node representations in an iterative process.

 For example, the ‘node features’ for a pixel in a color image

 would be the red, green and blue channel (RGB) values at that pixel.

 These kinds of graphs are called ‘homogeneous’.

 Many of the same ideas we will see here

 apply to other kinds of graphs:

 Sometimes we will need to denote a graph property by a matrix MMM,

 Extending Convolutions to Graphs

----------------------------------

 they depend on the absolute positions of pixels.

 where the neighbourhood structure differs from node to node.

 The curious reader may wonder if performing some sort of padding and ordering

 This has been attempted with some success ,

 but the techniques we will look at here are more general and powerful.

image/svg+xml

Convolution in CNNs

1

7

6

7

1

6

4

5

6

3

 Convolutions in CNNs are inherently localized.

image/svg+xml

4

6

1

2

5

4

1

7

3

6

1

7

6

Localized Convolution in GNNs

2

 GNNs can perform localized convolutions mimicking CNNs.

 Hover over a node to see its immediate neighbourhood highlighted on the left.

 The structure of this neighbourhood changes from node to node.

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/neighbourhoods-for-cnns-and-gnns.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 much like how CNNs compute localized filters over neighbouring pixels.

 Finally, we will discuss alternative methods

 Polynomial Filters on Graphs

------------------------------

###

 The Graph Laplacian

 Given a graph GGG, let us fix an arbitrary ordering of the nnn nodes of GGG.

Dv=∑uAvu.

 D\_v = \sum\_u A\_{vu}.

 Dv=u∑Avu.

 The degree of node vvv is the number of edges incident at vvv.

 in the matrix AAA. We will use this notation throughout this section.

 Then, the graph Laplacian LLL is the square n×nn \times nn×n matrix defined as:

 L=D−A.

 L = D - A.

 L=D−A.

![](Understanding%20Convolutions%20on%20Graphs_files/laplacian.svg)

 Zeros in LLL are not displayed above.

 The graph Laplacian gets its name from being the discrete analog of the

 [Laplacian operator](https://mathworld.wolfram.com/Laplacian.html)

 from calculus.

 Although it encodes precisely the same information as the adjacency matrix AAA

 ,

 the graph Laplacian has many interesting properties of its own.

 The graph Laplacian shows up in many mathematical problems involving graphs:

 [spectral clustering](https://arxiv.org/abs/0711.0189),

 and

 We will see some of these properties

 in [a later section](#spectral),

 but will instead point readers to

 for greater insight into the graph Laplacian.

###

 Polynomials of the Laplacian

 Now that we have understood what the graph Laplacian is,

 we can build polynomials  of the form:

 pw(L)=w0In+w1L+w2L2+…+wdLd=∑i=0dwiLi.

 pw(L)=w0In+w1L+w2L2+…+wdLd=i=0∑dwiLi.

 Each polynomial of this form can alternately be represented by

 its vector of coefficients w=[w0,…,wd]w = [w\_0, \ldots, w\_d]w=[w0,…,wd].

 These polynomials can be thought of as the equivalent of ‘filters’ in CNNs,

 and the coefficients www as the weights of the ‘filters’.

 each of the xvx\_vxv for v∈Vv \in Vv∈V is just a real number.

 Using the previously chosen ordering of the nodes,

 we can stack all of the node features xvx\_vxv

 to get a vector x∈Rnx \in \mathbb{R}^nx∈Rn.

 Once we have constructed the feature vector xxx,

 we can define its convolution with a polynomial filter pwp\_wpw as:

 x′=pw(L) x

 x’ = p\_w(L) \ x

 x′=pw(L) x

 To understand how the coefficients www affect the convolution,

 let us begin by considering the ‘simplest’ polynomial:

 when w0=1w\_0 = 1w0=1 and all of the other coefficients are 000.

 In this case, x′x’x′ is just xxx:

 x′=pw(L) x=∑i=0dwiLix=w0Inx=x.

 x’ = p\_w(L) \ x = \sum\_{i = 0}^d w\_i L^ix = w\_0 I\_n x = x.

 x′=pw(L) x=i=0∑dwiLix=w0Inx=x.

 Now, if we increase the degree, and consider the case where

 instead w1=1w\_1 = 1w1=1 and and all of the other coefficients are 000.

 Then, x′x’x′ is just LxLxLx, and so:

 xv′=(Lx)v=Lvx=∑u∈GLvuxu=∑u∈G(Dvu−Avu)xu=Dv xv−∑u∈N(v)xu

 \begin{aligned}

 x’\_v = (Lx)\_v &= L\_v x \\

 &= \sum\_{u \in G} L\_{vu} x\_u \\

 &= \sum\_{u \in G} (D\_{vu} - A\_{vu}) x\_u \\

 &= D\_v \ x\_v - \sum\_{u \in \mathcal{N}(v)} x\_u

 \end{aligned}

 xv′=(Lx)v=Lvx=u∈G∑Lvuxu=u∈G∑(Dvu−Avu)xu=Dv xv−u∈N(v)∑xu

 We see that the features at each node vvv are combined

 with the features of its immediate neighbours u∈N(v)u \in \mathcal{N}(v)u∈N(v).

 For readers familiar with

 this is the exact same idea. When xxx is an image,

 At this point, a natural question to ask is:

 Indeed, it is not too hard to show that:

 This is Lemma 5.2 from .

distG(v,u)>i⟹Lvui=0.

 \text{dist}\_G(v, u) > i \quad \Longrightarrow \quad L\_{vu}^i = 0.

 distG(v,u)>i⟹Lvui=0.

 \begin{aligned}

 x’\_v = (p\_w(L)x)\_v &= (p\_w(L))\_v x \\

 &= \sum\_{i = 0}^d w\_i L\_v^i x \\

 &= \sum\_{i = 0}^d w\_i \sum\_{u \in G} L\_{vu}^i x\_u \\

 \end{aligned}

 the resulting pixel in x′x’x′.

 Note that even after adjusting for position,

 #button-container {

 text-align: center;

 }

Reset Grid

Randomize Grid

−2−1012Color Scale

Input Grid

x∈{0,1}25x \in \{0, 1\}^{25}x∈{0,1}25

pw(L)p\_w(L)pw(L)

Convolutional Kernel at Highlighted Pixel

Output Grid

x′∈R25x' \in \mathbb{R}^{25}x′∈R25

 #poly-main-div {

 position: relative;

 height: 220px;

 }

 #poly-svg {

 position: absolute;

 left: 0px;

 top: 0px;

 }

 #figcaptions {

 position: relative;

 }

 #figcaptions figcaption {

 position: absolute;

 }

 #orig-nat-caption-1 {

 left: 170px;

 top: 140px;

 }

 #orig-nat-caption-2 {

 left: 170px;

 top: 160px;

 }

 #poly-caption {

 left: 460px;

 top: 90px;

 }

 #kernel-caption {

 left: 360px;

 top: 230px;

 }

 #upd-nat-caption-1 {

 left: 725px;

 top: 140px;

 }

 #upd-nat-caption-2 {

 left: 740px;

 top: 160px;

 }

 #poly\_conv\_sliders form {

 margin-left: 5%;

 margin-right: 5%;

 }

 #poly\_conv\_sliders input {

 width: 80%;

 }

 #poly\_conv\_sliders output {

 width: 3em;

 margin-left: 0 !important;

 }

 #poly\_conv\_sliders {

 display: flex;

 width: 60%;

 margin: 0 auto;

 text-align: center;

 align-items: center;

 justify-content: center;

 }

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

Convolve

pw(L)=∑i=02wiLi=1I + 0.1L + 0L2.

 p\_w(L)

 = \sum\_{i = 0}^2 w\_i L^i

 = 1I \ + \ 0.1L \ + \ 0L^2.

pw(L)=i=0∑2wiLi=1I + 0.1L + 0L2.

w0{w\_{0}}w0

1w1{w\_{1}}w1

0.1w2{w\_{2}}w2

0

Choice of Laplacian

Unnormalized L\text{Unnormalized} \ LUnnormalized L

Normalized L~\text{Normalized} \ \tilde{L}Normalized L~

 #button-container {

 text-align: center;

 }

Reset Coefficients

poly\_input\_slider\_watch = undefined

highlight\_selected\_cell = ƒ()

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Hover over a pixel in the input grid (left, representing xxx)

 to highlight it and see the equivalent convolutional kernel

 for that pixel under the arrow.

 The result x′x’x′ of the convolution is shown on the right:

 note that different convolutional kernels are applied at different pixels,

 depending on their location.

 Use the sliders at the bottom to change the coefficients www.

 To reset all coefficients www to 000, press ‘Reset Coefficients.’

###

 ChebNet

pw(L)=∑i=1dwiTi(L~)

 p\_w(L) = \sum\_{i = 1}^d w\_i T\_i(\tilde{L})

 pw(L)=i=1∑dwiTi(L~)

 where TiT\_iTi is the degree-iii

L~=2Lλmax(L)−In.

 \tilde{L} = \frac{2L}{\lambda\_{\max}(L)} - I\_n.

 L~=λmax(L)2L−In.

 What is the motivation behind these choices?

 This prevents the entries of powers of L~\tilde{L}L~ from blowing up.

 in order to show the result x′x’x′ on the same color scale.

 We won’t talk about this in more depth here,

 but will advise interested readers to take a look at  as a definitive resource.

###

 Polynomial Filters are Node-Order Equivariant

 Clearly, this sum does not depend on the order of the neighbours.

 A similar proof follows for higher degree polynomials:

 the entries in the powers of LLL are equivariant to the ordering of the nodes.

**Details for the Interested Reader**

 As above, let’s assume an arbitrary node-order over the nnn nodes of our graph.

 We can represent any permutation by a

 [permutation matrix](https://en.wikipedia.org/wiki/Permutation_matrix) PPP.

 PPP will always be an orthogonal 0−10-10−1 matrix:

 PPT=PTP=In.

 PP^T = P^TP = I\_n.

 PPT=PTP=In.

 f(Px)=Pf(x).

 f(Px) = P f(x).

 f(Px)=Pf(x).

 When switching to the new node-order using the permutation PPP,

 the quantities below transform in the following way:

 x→PxL→PLPTLi→PLiPT

 \begin{aligned}

 x &\to Px \\

 L &\to PLP^T \\

 L^i &\to PL^iP^T

 \end{aligned}

 xLLi→Px→PLPT→PLiPT

 and so, for the case of polynomial filters, we can see that:

 f(Px)=∑i=0dwi(PLiPT)(Px)=P∑i=0dwiLix′=Pf(x).

 \begin{aligned}

 f(Px) & = \sum\_{i = 0}^d w\_i (PL^iP^T) (Px) \\

 & = P \sum\_{i = 0}^d w\_i L^i x’ \\

 & = P f(x).

 \end{aligned}

 f(Px)=i=0∑dwi(PLiPT)(Px)=Pi=0∑dwiLix′=Pf(x).

 as needed.

###

 Embedding Computation

 We now describe how we can build a graph neural network

 by stacking ChebNet (or any polynomial filter) layers

 one after the other with non-linearities,

 much like a standard CNN.

 In particular, if we have KKK different polynomial filter layers,

 the kthk^{\text{th}}kth of which has its own learnable weights w(k)w^{(k)}w(k),

 we would perform the following computation:

 Start with the original features.

h(0)=x\quad {\color{#FE6100} h^{(0)}} = xh(0)=x

 Then iterate, for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

p(k)=pw(k)(L)g(k)=p(k)×h(k−1)h(k)=σ(g(k))

 \begin{aligned}

 \quad {p^{(k)}} &= p\_{\color{#4D9DB5} w^{(k)}}(L) \\

 \\

 \quad {g^{(k)}} &= p^{(k)} \times {\color{#FE6100} h^{(k - 1)}} \\

 \\

 \quad {\color{#FE6100} h^{(k)}} &= \sigma \left({g^{(k)}} \right)

 \end{aligned}

 p(k)g(k)h(k)=pw(k)(L)=p(k)×h(k−1)=σ(g(k))

 Color Codes:

 * Computed node embeddings.

* Learnable parameters.

 #spectral-conv div {

 margin: 0 auto 10px auto;

 display: block;

 }

 #cheb-conv {

 width: 800px;

 height: 310px;

 display: block;

 margin-top: 0;

 margin-bottom: 0;

 margin-left: auto;

 margin-right: auto;

 position: relative;

 }

 span.green {

 color: #177245;

 }

 span.blue {

 color: #0047AB;

 }

 /\* Legend \*/

 .legend {

 list-style: none;

 margin-top: 5px;

 }

 .legend li {

 margin-bottom: 5px;

 }

 .legend span.box {

 border: 1px solid #ccc;

 float: left;

 width: 12px;

 height: 12px;

 margin: 2px;

 margin-right: 5px;

 }

 .legend .main { background-color: #FE6100; }

 .legend .param { background-color: #4D9DB5; }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/updated-chebnet-equations.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Note that these networks

 reuse the same filter weights across different nodes,

 exactly mimicking weight-sharing in Convolutional Neural Networks (CNNs)

 which reuse weights for convolutional filters across a grid.

 Modern Graph Neural Networks

------------------------------

 ChebNet was a breakthrough in learning localized filters over graphs,

 focussing on a particular vertex vvv:

(Lx)v=Lvx=∑u∈GLvuxu=∑u∈G(Dvu−Avu)xu=Dv xv−∑u∈N(v)xu

 \begin{aligned}

 (Lx)\_v &= L\_v x \\

 &= \sum\_{u \in G} L\_{vu} x\_u \\

 &= \sum\_{u \in G} (D\_{vu} - A\_{vu}) x\_u \\

 &= D\_v \ x\_v - \sum\_{u \in \mathcal{N}(v)} x\_u

 \end{aligned}

 (Lx)v=Lvx=u∈G∑Lvuxu=u∈G∑(Dvu−Avu)xu=Dv xv−u∈N(v)∑xu

 As we noted before, this is a 111-hop localized convolution.

 But more importantly, we can think of this convolution as arising of two steps:

* Aggregating over immediate neighbour features xux\_uxu.

* Combining with the node’s own feature xvx\_vxv.

**Key Idea:**

 What if we consider different kinds of ‘aggregation’ and ‘combination’ steps,

 beyond what are possible using polynomial filters?

 By ensuring that the aggregation is node-order equivariant,

 the overall convolution becomes node-order equivariant.

 after each step, every node receives some ‘information’ from its neighbours.

###

 Embedding Computation

 Message-passing forms the backbone of many GNN architectures today.

 We describe the most popular ones in depth below:

* Graph Convolutional Networks (GCN)

* Graph Attention Networks (GAT)

* Graph Sample and Aggregate (GraphSAGE)

* Graph Isomorphism Network (GIN)

* GCN

* GAT

* GraphSAGE

* GIN

hv(0)\color{#FE6100} h\_v^{(0)}hv(0)

===

xvx\_vxv

 Node vvv's initial embedding.

 ... is just node vvv's original features.

 for all v∈V.v \in V.v∈V.

 and for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

hv(k)\color{#FE6100} h\_v^{(k)}hv(k)

===

f(k)(W(k)⋅∑u∈N(v)hu(k−1)∣N(v)∣+B(k)⋅hv(k−1)){\color{#4D9DB5}

 f^{(k)}}\left({\color{#4D9DB5} W^{(k)}} \cdot \frac{\sum\limits\_{u

\in \mathcal{N}(v)} {\color{#A95AA1} h\_u^{(k - 1)}}}{|\mathcal{N}(v)|} +

 {\color{#4D9DB5} B^{(k)}} \cdot {\color{#FE6100} h\_v^{(k -

1)}}\right)f(k)⎝⎜⎜⎜⎛W(k)⋅∣N(v)∣u∈N(v)∑hu(k−1)+B(k)⋅hv(k−1)⎠⎟⎟⎟⎞

 Node vvv's embedding at step kkk.

 Mean of vvv's neighbour's embeddings at step k−1k - 1k−1.

 Node vvv's embedding at step k−1k - 1k−1.

 for all v∈V.v \in V.v∈V.

 Color Codes:

 * Embedding of node vvv.

* Embedding of a neighbour of node vvv.

* (Potentially) Learnable parameters.

hv(0)\color{#FE6100} h\_v^{(0)}hv(0)

===

xvx\_vxv

 Node vvv's initial embedding.

 ... is just node vvv's original features.

 for all v∈V.v \in V.v∈V.

 and for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

hv(k)\color{#FE6100} h\_v^{(k)}hv(k)

===

f(k)(W(k)⋅[∑u∈N(v)αvu(k−1)hu(k−1)+αvv(k−1)hv(k−1)]){\color{#4D9DB5}

 f^{(k)}}\left({\color{#4D9DB5} W^{(k)}} \cdot \left[\sum\limits\_{u

\in \mathcal{N}(v)} \alpha\_{vu}^{(k - 1)} {\color{#A95AA1} h\_u^{(k -

1)}} + \alpha\_{vv}^{(k - 1)} {\color{#FE6100} h\_v^{(k -

 Node vvv's embedding at step kkk.

 Weighted mean of vvv's neighbour's embeddings at step k−1k - 1k−1.

 Node vvv's embedding at step k−1k - 1k−1.

 for all v∈V.v \in V.v∈V.

αvu(k)\alpha\_{vu}^{(k)}αvu(k)

===

A(k)(hv(k),hu(k))∑w∈N(v)A(k)(hv(k),hw(k))\frac{{\color{#4D9DB5}

 A^{(k)}}({\color{#FE6100} h\_v^{(k)}}, {\color{#A95AA1}

h\_u^{(k)}})}{\sum\limits\_{w \in \mathcal{N}(v)}{\color{#4D9DB5}

 for all (v,u)∈E.(v, u) \in E.(v,u)∈E.

 Color Codes:

 * Embedding of node vvv.

* Embedding of a neighbour of node vvv.

* (Potentially) Learnable parameters.

hv(0)\color{#FE6100} h\_v^{(0)}hv(0)

===

xvx\_vxv

 Node vvv's initial embedding.

 ... is just node vvv's original features.

 for all v∈V.v \in V.v∈V.

 and for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

hv(k)\color{#FE6100} h\_v^{(k)}hv(k)

===

f(k)(∑u∈N(v)hu(k−1)+(1+ϵ(k))⋅hv(k−1)){\color{#4D9DB5}

 f^{(k)}}\left(\sum\limits\_{u \in \mathcal{N}(v)} {\color{#A95AA1}

h\_u^{(k - 1)}} + (1 + {\color{#4D9DB5} \epsilon^{(k)}}) \cdot

 Node vvv's embedding at step kkk.

 Sum of vvv's neighbour's embeddings at step k−1k - 1k−1.

 Node vvv's embedding at step k−1k - 1k−1.

 for all v∈V.v \in V.v∈V.

 Color Codes:

 * Embedding of node vvv.

* Embedding of a neighbour of node vvv.

* (Potentially) Learnable parameters.

hv(0)\color{#FE6100} h\_v^{(0)}hv(0)

===

xvx\_vxv

 Node vvv's initial embedding.

 ... is just node vvv's original features.

 for all v∈V.v \in V.v∈V.

 and for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

hv(k)\color{#FE6100} h\_v^{(k)}hv(k)

===

f(k)(W(k)⋅[AGGu∈N(v)({hu(k−1)}), hv(k−1)]){\color{#4D9DB5}

 f^{(k)}}\left({\color{#4D9DB5} W^{(k)}} \cdot \left[\underset{{u \in

\mathcal{N}(v)}}{\color{#4D9DB5} \text{AGG}}(\{ {\color{#A95AA1}

 Node vvv's embedding at step kkk.

 Aggregation of vvv's neighbour's embeddings at step k−1k - 1k−1 ...

 ... concatenated with ...

 ... Node vvv's embedding at step k−1k - 1k−1.

 for all v∈V.v \in V.v∈V.

 Color Codes:

 * Embedding of node vvv.

* Embedding of a neighbour of node vvv.

* (Potentially) Learnable parameters.

Predictions can be made at each node by using the final computed embedding:

yv^=PREDICT(hv(K))

\hat{y\_v} = \text{PREDICT}({\color{#FE6100} h\_v^{(K)}})

yv^=PREDICT(hv(K))

The variant we discuss here is the 2-parameter model from the original paper ,

f(W⋅∑u∈N(v)hu∣N(v)∣+B⋅hv)

{\color{#4D9DB5} f}\left({\color{#4D9DB5} W} \cdot \sum\_{u \in

\mathcal{N}(v)} \frac{\color{#A95AA1} h\_u}{|\mathcal{N}(v)|} +

instead of the normalization defined in the original paper:

f(W⋅∑u∈N(v)hu∣N(u)∣∣N(v)∣+B⋅hv)

{\color{#4D9DB5} f}\left({\color{#4D9DB5} W} \cdot \sum\_{u \in

\mathcal{N}(v)} \frac{\color{#A95AA1}

h\_u}{\sqrt{|\mathcal{N}(u)||\mathcal{N}(v)|}} + {\color{#4D9DB5} B}

\cdot {\color{#FE6100} h\_v} \right)f⎝⎜⎛W⋅u∈N(v)∑∣N(u)∣∣N(v)∣hu+B⋅hv⎠⎟⎞

 for ease of exposition.

interactive\_list = undefined

 #gnn-models ul.list {

 display: grid;

 grid-template-columns: 1fr 1fr 1fr 1fr;

 grid-column-gap: 4px;

 text-align: center;

 padding: 0;

 list-style: none;

 }

 #gnn-models ul.list li {

 margin: 0;

 background: #f3f3f3;

 border-bottom: 3px solid #e3e3e3;

 cursor: pointer;

 line-height: 2em;

 border-top-left-radius: 3px;

 border-top-right-radius: 3px;

 }

 #gnn-models ul.list li:hover {

 border-bottom-color: #D9D9D9;

 }

 #gnn-models ul.list li.selected {

 background: #D9D9D9;

 border-bottom-color: #D9D9D9;

 }

 #gnn-models div {

 margin: 0 auto 10px auto;

 display: none;

 }

 #gnn-models {

 display: block;

 margin-left: auto;

 margin-right: auto;

 position: relative;

 }

 #gnn-models[data-show='gcn'] .gnn-model-gcn,

 #gnn-models[data-show='gin'] .gnn-model-gin,

 #gnn-models[data-show='graphsage'] .gnn-model-graphsage,

 #gnn-models[data-show='gat'] .gnn-model-gat {

 display: block;

 }

 /\* Legend \*/

 .legend {

 list-style: none;

 margin-top: 5px;

 }

 .legend li {

 margin-bottom: 0;

 }

 .legend span.box {

 border: 1px solid #ccc;

 float: left;

 width: 12px;

 height: 12px;

 margin: 2px;

 margin-right: 5px;

 }

 .legend .main { background-color: #FE6100; }

 .legend .neigh { background-color: #A95AA1; }

 .legend .param { background-color: #4D9DB5; }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/interactive-gnn-equations.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

###

 Thoughts

  demonstrates that aggregation functions indeed can be compared on how well

 they can uniquely preserve node neighbourhood features;

 Here, we’ve talk about GNNs where the computation only occurs at the nodes.

 More recent GNN models

 such as Message-Passing Neural Networks

 and Graph Networks

 perform computation over the edges as well;

 they compute edge embeddings together with node embeddings.

 This is an even more general framework -

 but the same ‘message passing’ ideas from this section apply.

 Interactive Graph Neural Networks

-----------------------------------

 Below is an interactive visualization of these GNN models on small graphs.

 but the same equations hold when the node features are vectors.

* GCN

* GAT

* GraphSAGE

* GIN

 #button-container {

 text-align: center;

 }

Reset

Undo Last Update

Update All Nodes

Randomize Graph

 #button-container {

 text-align: center;

 }

Reset

Undo Last Update

Update All Nodes

Randomize Graph

 #button-container {

 text-align: center;

 }

Reset

Undo Last Update

Update All Nodes

Randomize Graph

 #button-container {

 text-align: center;

 }

Reset

Undo Last Update

Update All Nodes

Randomize Graph

 #main-div {

 display: flex;

 flex-direction: row;

 align-items: center;

 justify-content: center;

 }

 #labels-div form {

 margin-top: 2%;

 margin-bottom: 2%;

 }

 #labels-div {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 width: 65%;

 }

 #iteration {

 font-weight: bold;

 margin-bottom: 20%;

 }

 #sliders {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 Initial Graph

 Parameters for Next Update

W(1)W^{(1)}W(1)

1

B(1)B^{(1)}B(1)

1

A6B2C-10D1E3

 #main-div {

 display: flex;

 flex-direction: row;

 align-items: center;

 justify-content: center;

 }

 #labels-div {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 width: 65%;

 }

 #iteration {

 font-weight: bold;

 margin-bottom: 20%;

 }

 #labels-div form {

 margin-top: 2%;

 margin-bottom: 2%;

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 Initial Graph

 Parameters for Next Update

W(1)W^{(1)}W(1)

1

AW(1)A\_W^{(1)}AW(1)

1

A6B2C-10D1E3

 #main-div {

 display: flex;

 flex-direction: row;

 align-items: center;

 justify-content: center;

 }

 #labels-div form {

 margin-top: 2%;

 margin-bottom: 2%;

 }

 #labels-div {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 width: 65%;

 }

 #iteration {

 font-weight: bold;

 margin-bottom: 20%;

 }

 #sliders {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 Initial Graph

 Parameters for Next Update

W(1)W^{(1)}W(1)

1

B(1)B^{(1)}B(1)

1

Us(1)U\_s^{(1)}Us(1)

1

Uh(1)U\_h^{(1)}Uh(1)

1

A6B2C-10D1E3

 #main-div {

 display: flex;

 flex-direction: row;

 align-items: center;

 justify-content: center;

 }

 #labels-div form {

 margin-top: 2%;

 margin-bottom: 2%;

 }

 #labels-div {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 width: 65%;

 }

 #iteration {

 font-weight: bold;

 margin-bottom: 20%;

 }

 #sliders {

 display: flex;

 flex-direction: column;

 align-items: center;

 justify-content: center;

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 Initial Graph

 Parameters for Next Update

ϵ(1){\epsilon}^{(1)}ϵ(1)

0

A6B2C-10D1E3

 **Next Update (Iteration 1):**

 Equation for Node AAA:

 \begin{aligned}

 {\color{#FE6100} h\_{A}^{(1)}}

 &= {f} \left(

 {\color{#4D9DB5} B^{(1)}} \times {\color{#FE6100} h\_{A}^{(0)}} \right) \\

 &= f \left(

 {\color{#4D9DB5} 1} \times {\color{#FE6100} 6} \right) \\

 &= f \left(

 -3.5 +

 6 \right) \\

 &= f \left(2.5 \right) \\

 &= \text{ReLU} \left(2.5\right)

 = 2.5.

 **Next Update (Iteration 1):**

 Equation for Node AAA:

 \begin{aligned}

 {\color{#FE6100} h\_{A}^{(1)}}

 &= f \left({\color{#4D9DB5} W^{(1)}} \times

\left(\alpha\_{AC} \times {\color{#A95AA1} h\_{C}^{(0)}} + \alpha\_{AE}

\times {\color{#A95AA1} h\_{E}^{(0)}} + \alpha\_{AA} \times

{\color{#FE6100} h\_{A}^{(0)}} \right) \right) \\

 &= f \left({\color{#4D9DB5} W^{(1)}} \times \left(0

\times {\color{#A95AA1} h\_{C}^{(0)}} + 0.05 \times {\color{#A95AA1}

h\_{E}^{(0)}} + 0.95 \times {\color{#FE6100} h\_{A}^{(0)}} \right) \right)

 \\

 &= f \left({\color{#4D9DB5} W^{(1)}} \times \big(0

\times {\color{#A95AA1} -10} + 0.05 \times {\color{#A95AA1} 3} + 0.95

\times {\color{#FE6100} 6}\big) \right) \\

 &= f \left({\color{#4D9DB5} W^{(1)}} \times \big(0 +

0.14 + 5.72 \big) \right) \\

 &= f \left({\color{#4D9DB5} 1} \times \left(5.86

\right) \right) \\

 &= f \left(5.86\right) \\

 &= \text{ReLU} \left(5.86\right)

 = 5.86.

 with attention weights αA\alpha\_{A}αA computed as:

 \begin{aligned}

 \alpha\_{AC} = \frac{\exp{(e\_{AC})}}{\exp{(e\_{AC})} +

\exp{(e\_{AE})} + \exp{(e\_{AA})}} &\approx 0. \\ \alpha\_{AE} =

\frac{\exp{(e\_{AE})}}{\exp{(e\_{AC})} + \exp{(e\_{AE})} + \exp{(e\_{AA})}}

&\approx 0.05. \\ \alpha\_{AA} = \frac{\exp{(e\_{AA})}}{\exp{(e\_{AC})}

 + \exp{(e\_{AE})} + \exp{(e\_{AA})}} &\approx 0.95.

 \end{aligned}

 where the unnormalized attention weights eAe\_{A}eA are given by:

 \begin{aligned}

 e\_{AC} &= \text{ReLU}\left({\color{#4D9DB5} A\_{W}^{(1)}}

\left({\color{#4D9DB5} W^{(1)}}{\color{#FE6100} h\_{A}^{(0)}} +

{\color{#4D9DB5} W^{(1)}}{\color{#A95AA1} h\_{C}^{(0)}} \right)\right) =

 \text{ReLU}\left({\color{#4D9DB5} 1}\left({\color{#4D9DB5} 1}\times

{\color{#FE6100} 6} + {\color{#4D9DB5} 1} \times {\color{#A95AA1}

-10}\right)\right) = \text{ReLU}\left(-4\right) = 0. \\ e\_{AE} &=

\text{ReLU}\left({\color{#4D9DB5} A\_{W}^{(1)}} \left({\color{#4D9DB5}

W^{(1)}}{\color{#FE6100} h\_{A}^{(0)}} + {\color{#4D9DB5}

W^{(1)}}{\color{#A95AA1} h\_{E}^{(0)}} \right)\right) =

\text{ReLU}\left({\color{#4D9DB5} 1}\left({\color{#4D9DB5} 1}\times

{\color{#FE6100} 6} + {\color{#4D9DB5} 1} \times {\color{#A95AA1}

3}\right)\right) = \text{ReLU}\left(9\right) = 9. \\ e\_{AA} &=

\text{ReLU}\left({\color{#4D9DB5} A\_{W}^{(1)}} \left({\color{#4D9DB5}

W^{(1)}}{\color{#FE6100} h\_{A}^{(0)}} + {\color{#4D9DB5}

W^{(1)}}{\color{#FE6100} h\_{A}^{(0)}} \right)\right) =

\text{ReLU}\left({\color{#4D9DB5} 1}\left({\color{#4D9DB5} 1}\times

{\color{#FE6100} 6} + {\color{#4D9DB5} 1} \times {\color{#FE6100}

6}\right)\right) = \text{ReLU}\left(12\right) = 12. \\

 \end{aligned}

 We have omitted the superscripts on the attention weights for clarity.

 **Next Update (Iteration 1):**

 Equation for Node AAA:

 hA(1)=f(W(1)×RNN[hC(0),hE(0)]+B(1)×hA(0))=f(1×3+1×6)=f(3+6)=f(9)=ReLU(9)=9.

 \begin{aligned}

 {\color{#FE6100} h\_{A}^{(1)}}

 &= {f} \left(

 {\color{#4D9DB5} W^{(1)}} \times

\text{RNN}\left[{\color{#A95AA1} h\_{C}^{(0)}} , {\color{#A95AA1}

h\_{E}^{(0)}}\right] +

 {\color{#4D9DB5} B^{(1)}} \times {\color{#FE6100}

h\_{A}^{(0)}} \right) \\

 &= f \left(

 {\color{#4D9DB5} 1} \times {\color{#A95AA1} 3} +

 {\color{#4D9DB5} 1} \times {\color{#FE6100} 6}

\right) \\

 &= f \left(

 3 +

 6 \right) \\

 &= f \left(9 \right) \\

 &= \text{ReLU} \left(9\right)

 = 9.

 \begin{aligned}

 \qquad s\_0 &= 0. \\

 s\_{1}&= \text{ReLU}\left({\color{#4D9DB5} U\_s} s\_{0} +

{\color{#4D9DB5} U\_h} {\color{#A95AA1} h\_{C}^{(0)}}\right) =

\text{ReLU}\left(

 {\color{#4D9DB5} 1}

 \times 0 +

 {\color{#4D9DB5} 1}

 \times {\color{#A95AA1} -10} \right) =

\text{ReLU}\left(-10\right) = 0. \\ s\_{2}&=

\text{ReLU}\left({\color{#4D9DB5} U\_s} s\_{1} + {\color{#4D9DB5} U\_h}

{\color{#A95AA1} h\_{E}^{(0)}}\right) = \text{ReLU}\left(

 {\color{#4D9DB5} 1}

 \times 0 +

 {\color{#4D9DB5} 1}

 \times {\color{#A95AA1} 3} \right) = \text{ReLU}\left(3\right) =

3.

 \end{aligned}

 Concisely, the initial RNN\text{RNN}RNN state is s0=0s\_0 = 0s0=0, and

 the RNN\text{RNN}RNN state update equation is:

 st=ReLU(Usst−1+Uhhut(0))=ReLU(1st−1+1hut(0)).

 \begin{aligned}

 s\_{t} &= \text{ReLU}\left({\color{#4D9DB5} U\_s} s\_{t - 1} +

{\color{#4D9DB5} U\_h} h\_{u\_t}^{(0)}\right) \\ &=

\text{ReLU}\left({\color{#4D9DB5} 1} s\_{t - 1} + {\color{#4D9DB5} 1}

 h\_{u\_t}^{(0)}\right).

 \end{aligned}

 st=ReLU(Usst−1+Uhhut(0))=ReLU(1st−1+1hut(0)).

 RNN\text{RNN}RNN to learn the same final aggregated value.

 This tries to make the RNN\text{RNN}RNN node-order invariant.

 **Next Update (Iteration 1):**

 Equation for Node AAA:

 \begin{aligned}

 {\color{#FE6100} h\_{A}^{(1)}}

 &= {f} \left(

 \left({\color{#A95AA1} h\_{C}^{(0)}} + {\color{#A95AA1} h\_{E}^{(0)}}\right) +

 &= f \left(

 \left({\color{#A95AA1} -10} + {\color{#A95AA1} 3}\right) +

 \left(1 + {\color{#4D9DB5} 0}\right)

 \times {\color{#FE6100} 6} \right) \\

 &= f \left(

 \left(-7\right) +

 6 \right) \\

 &= f \left(-1 \right) \\

 &= \text{ReLU} \left(-1\right)

 = 0.

 div.eqn {

 margin-left: 2%;

 }

 #gnn-models-viz ul {

 display: grid;

 grid-template-columns: 1fr 1fr 1fr 1fr;

 grid-column-gap: 4px;

 text-align: center;

 padding: 0;

 justify-content: center;

 align-content: center;

 list-style: none;

 max-width: 100%;

 }

 #gnn-models-viz ul li {

 margin: 0;

 background: #f3f3f3;

 border-bottom: 3px solid #e3e3e3;

 cursor: pointer;

 line-height: 2em;

 border-top-left-radius: 3px;

 border-top-right-radius: 3px;

 }

 #gnn-models-viz ul li:hover {

 border-bottom-color: #D9D9D9;

 }

 #gnn-models-viz ul li.selected {

 background: #D9D9D9;

 border-bottom-color: #D9D9D9;

 }

 #gnn-models-viz {

 }

 /\* Nicer sliders. \*/

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/interactive-gnn-visualizations.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Choose a GNN model using the tabs at the top. Click on a node

to see the update equation at that node for the next iteration.

 Use the sliders on the left to change the weights for the

current iteration, and watch how the update equation changes.

 This ideology is followed by many popular Graph Neural Network libraries,

 From Local to Global Convolutions

-----------------------------------

 The methods we’ve seen so far perform ‘local’ convolutions:

 While performing enough steps of message-passing will eventually ensure that

 information from all nodes in the graph is passed,

 one may wonder if there are more direct ways to perform ‘global’ convolutions.

 in the context of neural networks by ,

 much before any of the GNN models we looked at above.

###

 Spectral Convolutions

 As before, we will focus on the case where nodes have one-dimensional features.

 ‘feature vector’ x∈Rnx \in \mathbb{R}^nx∈Rn.

**Key Idea:**

 Given a feature vector xxx,

 the Laplacian LLL allows us to quantify how smooth xxx is, with respect to GGG.

 How?

 if we look at the following quantity involving LLL:

RL(x)=xTLxxTx=∑(i,j)∈E(xi−xj)2∑ixi2=∑(i,j)∈E(xi−xj)2.

 R\_L(x) = \frac{x^T L x}{x^T x} = \frac{\sum\_{(i, j) \in E}

(x\_i - x\_j)^2}{\sum\_i x\_i^2} = \sum\_{(i, j) \in E} (x\_i - x\_j)^2.

 RL(x)=xTxxTLx=∑ixi2∑(i,j)∈E(xi−xj)2=(i,j)∈E∑(xi−xj)2.

 we immediately see that feature vectors xxx that assign similar values to

 An eigenvalue λ\lambdaλ of a matrix AAA is a value

 For a friendly introduction to eigenvectors,

 please see [this tutorial](http://www.sosmath.com/matrix/eigen0/eigen0.html).

 uk1Tuk2={1 if k1=k2.0 if k1≠k2.

 u\_{k\_1}^T u\_{k\_2} =

 \begin{cases}

 1 \quad \text{ if } {k\_1} = {k\_2}. \\

 0 \quad \text{ if } {k\_1} \neq {k\_2}.

 \end{cases}

 uk1Tuk2={1 if k1=k2.0 if k1≠k2.

argminx, x⊥{u1,…,ui−1}RL(x)=ui.minx, x⊥{u1,…,ui−1}RL(x)=λi.

 \qquad

 \qquad

 \qquad

 \min\_{x, \ x \perp \{u\_1, \ldots, u\_{i - 1}\}} R\_L(x) = \lambda\_i.

 x, x⊥{u1,…,ui−1}argminRL(x)=ui.x, x⊥{u1,…,ui−1}minRL(x)=λi.

 The set of eigenvalues of LLL are called its ‘spectrum’, hence the name!

 We denote the ‘spectral’ decomposition of LLL as:

 L=UΛUT.

 L = U \Lambda U^T.

 L=UΛUT.

 where Λ\LambdaΛ is the diagonal matrix of sorted eigenvalues,

 Λ=[λ1⋱λn]U=[u1 ⋯ un].

 \Lambda = \begin{bmatrix}

 \lambda\_{1} & & \\

 & \ddots & \\

 & & \lambda\_{n}

 \end{bmatrix}

 \qquad

 \qquad

 \qquad

 \qquad

 U = \begin{bmatrix} \\ u\_1 \ \cdots \ u\_n \\ \end{bmatrix}.

 Λ=⎣⎡λ1⋱λn⎦⎤U=⎣⎡u1 ⋯ un⎦⎤.

 As these nnn eigenvectors form a basis for Rn\mathbb{R}^nRn,

 x=∑i=1nxi^ui=Ux^.

 x = \sum\_{i = 1}^n \hat{x\_i} u\_i = U \hat{x}.

 x=i=1∑nxi^ui=Ux^.

 We call x^\hat{x}x^ as the spectral representation of the feature vector xxx.

 The orthonormality condition allows us to state:

 x=Ux^⟺UTx=x^.

 x = U \hat{x} \quad \Longleftrightarrow \quad U^T x = \hat{x}.

 x=Ux^⟺UTx=x^.

 This pair of equations allows us to interconvert

 for any vector x∈Rnx \in \mathbb{R}^nx∈Rn.

###

 Spectral Representations of Natural Images

 connected by edges to adjacent pixels.

 vector, indicating the values for the red, green and blue (RGB) channels.

 To shed some light on what the spectral representation actually encodes,

* We first collect all pixel values across a channel into a feature vector xxx.

* Then, we obtain its spectral representation x^\hat{x}x^.

 x^=UTx

 \hat{x} = U^T x

 x^=UTx

* We truncate this to the first mmm components to get x^m\hat{x}\_mx^m.

 x^m=Truncatem(x^)

 \hat{x}\_m = \text{Truncate}\_m(\hat{x})

 x^m=Truncatem(x^)

 xm=Ux^m

 x\_m = U \hat{x}\_m

 xm=Ux^m

 Finally, we stack the resulting channels back together to get back an image.

 We can now see how the resulting image changes with choices of mmm.

 as we can reconstruct each channel exactly.

Sample Image

 Chicken

 Fish

 Frog

 Spider

 Original Image

 xxx

Keep First 200 Spectral Components

 Transformed Image

 x′x'x′

Number of Spectral Components (m)

200

updateArrowCaption = undefined

drawArrow = undefined

drawCurrBaseImg = undefined

drawCurrUpdImg = undefined

 #specDecompBody {

 display: flex;

 justify-content: center;

 }

 #svgArrow {

 margin-top: 83px;

 margin-left: 2em;

 margin-right: 2em;

 }

 #specDecompAll {

 display: flex;

 flex-direction: column;

 text-align: center;

 height: 350px;

 justify-content: space-evenly;

 }

 #specDecompInputsRadio input {

 margin-top: 0.5em;

 vertical-align: unset !important;

 }

 #origCaption {

 margin-top: 1em;

 }

 #updCaption {

 margin-top: 1em;

 }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/spectral-decompositions-of-natural-images.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Use the radio buttons at the top to chose one of the four sample images.

 Each of these images has been taken from the ImageNet

 dataset and downsampled to 505050 pixels wide and 404040 pixels tall.

 images get progressively blurrier as the number of components decrease.

 As mmm decreases, we see that the output image xmx\_mxm gets blurrier.

 We see that we do not need to keep all nnn components;

 We can relate this to the Fourier decomposition of images:

 To complement the visualization above,

 We change the coefficients of the first 101010 out of 646464 eigenvectors

 in the spectral representation

 and see how the resulting image changes:

Spectral Representation

x^=[x^1 x^2 x^3 x^4 x^5 x^6 x^7 x^8 x^9 x^10]\hat{x}

 = \left[

 \hat{x}\_{1} \ \hat{x}\_{2} \ \hat{x}\_{3} \ \hat{x}\_{4} \ \hat{x}\_{5} \

 \hat{x}\_{6} \ \hat{x}\_{7} \ \hat{x}\_{8} \ \hat{x}\_{9} \ \hat{x}\_{10}

 \right]x^=[x^1 x^2 x^3 x^4 x^5 x^6 x^7 x^8 x^9 x^10]

RL(x)=1.77R\_L(x) = 1.77RL(x)=1.77

Natural Representation

x∈R64x \in \mathbb{R}^{64}x∈R64

 #spec-main-div {

 position: relative;

 height: 400px;

 }

 #spec-svg {

 position: absolute;

 left: 0px;

 top: 0px;

 }

 #figcaptions {

 position: relative;

 }

 #figcaptions figcaption {

 position: absolute;

 }

 #spec-caption-1 {

 left: 660px;

 top: 400px;

 }

 #spec-caption-2 {

 left: 620px;

 top: 420px;

 }

 #quot-caption-div {

 position: relative;

 width: 200px;

 left: 170px;

 top: 350px;

 }

 #figcaptions #quot-caption-1 {

 position: relative;

 text-align: center;

 font: inherit;

 color: inherit;

 }

 #nat-caption-1 {

 left: 205px;

 top: 400px;

 }

 #nat-caption-2 {

 left: 255px;

 top: 420px;

 }

 #spec-sliders form {

 margin-left: 2%;

 margin-right: 2%;

 }

 #spec-sliders input {

 width: 100%;

 }

 #spec-sliders output {

 width: 3em;

 margin-left: 0 !important;

 }

 #spec-sliders {

 display: flex;

 margin: 0 auto;

 text-align: center;

 }

 #subgrid-main-div {

 display: flex;

 margin: 0 auto;

 }

 input[type="range"] {

 -webkit-appearance: none;

 grid-column: middle;

 height: 5px;

 border-radius: 1px;

 background: #d3d3d3;

 outline: none;

 opacity: 0.7;

 -webkit-transition: .2s;

 transition: opacity .2s;

 }

 input[type="range"]::-webkit-slider-thumb {

 -webkit-appearance: none;

 appearance: none;

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #575245;

 cursor: pointer;

 }

 input[type="range"]::-moz-range-thumb {

 width: 16px;

 height: 16px;

 border-radius: 50%;

 background: #f5f2eb;

 cursor: pointer;

 }

 output[name="output"] {

 width: 1em;

 display: inline-block;

 }

−0.4−0.20.00.20.4Color Scale

x^1{\hat{x}\_{1}}x^1

0x^2{\hat{x}\_{2}}x^2

-0.3x^3{\hat{x}\_{3}}x^3

-0.3x^4{\hat{x}\_{4}}x^4

0.6x^5{\hat{x}\_{5}}x^5

-0.7x^6{\hat{x}\_{6}}x^6

0.4x^7{\hat{x}\_{7}}x^7

0.4x^8{\hat{x}\_{8}}x^8

0.4x^9{\hat{x}\_{9}}x^9

0.4x^10{\hat{x}\_{10}}x^10

0.7

 #button-container {

 text-align: center;

 }

Reset Coefficients

spec\_input\_slider\_watch = undefined

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/interactive-spectral-conversions.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 });

 }, 200);

 Move the sliders to change the spectral representation x^\hat{x}x^ (right),

 and see how xxx itself changes on the image (left).

 Note how the first eigenvectors are much ‘smoother’ than the later ones,

 and the many patterns we can make with only 101010 eigenvectors.

 and the smoothness correspondingly decreases as we consider later eigenvectors.

 For any image xxx, we can think of

 the initial entries of the spectral representation x^\hat{x}x^

### Embedding Computation

 We now have the background to understand spectral convolutions

 As before, the model we describe below has KKK layers:

 each layer kkk has learnable parameters w^(k)\hat{w}^{(k)}w^(k),

 called the ‘filter weights’.

 eigenvectors used to compute the spectral representations.

 We had shown in the previous section that we can take m≪nm \ll nm≪n

 and still not lose out on significant amounts of information.

 than just direct convolution in the natural domain.

 using spectral representations automatically enforces an inductive bias for

 neighbouring nodes to get similar representations.

 Assuming one-dimensional node features for now,

 the output of each layer is a vector of node representations h(k)h^{(k)}h(k),

 where each node’s representation corresponds to a row

 of the vector.

h(k)=[h1(k)⋮hn(k)]

 h(k)=⎣⎢⎢⎢⎡h1(k)⋮hn(k)⎦⎥⎥⎥⎤

 for each k=0,1,2,…k = 0, 1, 2, \ldots k=0,1,2,… upto K KK.

 allowing us to compute UmU\_mUm.

 Start with the original features.

h(0)=x\quad {\color{#FE6100} h^{(0)}} = xh(0)=x

 Then iterate, for k=1,2,…k = 1, 2, \ldots k=1,2,… upto K KK:

h^(k−1)=UmTh(k−1)g^(k)=w^(k)⊙h^(k−1)g(k)=Umg^(k)h(k)=σ(g(k))

 \begin{aligned}

 {\hat{h}^{(k - 1)}} &= U\_m^T {\color{#FE6100} h^{(k - 1)}} \\

 \\

 {\hat{g}^{(k)}} &=

 {\color{#4D9DB5} \hat{w}^{(k)}} \odot {\color{#FE6100} \hat{h}^{(k - 1)}} \\

 \\

 {g^{(k)}} &= U\_m {\hat{g}^{(k)}} \\

 \\

 {\color{#FE6100} h^{(k)}} &= \sigma \left({g^{(k)}} \right)

 \end{aligned}

 h^(k−1)g^(k)g(k)h(k)=UmTh(k−1)=w^(k)⊙h^(k−1)=Umg^(k)=σ(g(k))

⊙\odot⊙ represents element-wise multiplication.

 Color Codes:

 * Computed node embeddings.

* Learnable parameters.

 #spectral-conv div {

 margin: 0 auto 10px auto;

 display: block;

 }

 #spectral-conv {

 width: 800px;

 height: 380px;

 display: block;

 margin-top: 0;

 margin-bottom: 0;

 margin-left: auto;

 margin-right: auto;

 position: relative;

 }

 #spectral-conv-init {

 width: 800px;

 height: 100px;

 display: block;

 margin-top: 0;

 margin-left: auto;

 margin-right: auto;

 position: relative;

 }

 span.green {

 color: #177245;

 }

 span.blue {

 color: #0047AB;

 }

 /\* Legend \*/

 .legend {

 list-style: none;

 margin-top: 5px;

 }

 .legend li {

 margin-bottom: 5px;

 }

 .legend span.box {

 border: 1px solid #ccc;

 float: left;

 width: 12px;

 height: 12px;

 margin: 2px;

 margin-right: 5px;

 }

 .legend .main { background-color: #FE6100; }

 .legend .param { background-color: #4D9DB5; }

 import {Runtime, Inspector} from "./observablehq-base/runtime.js";

 import define from "./notebooks/spectral-convolutions-equation.js";

 setTimeout(() => {

 new Runtime().module(define, name => {

 if (name === "spec\_figure") return Inspector.into(".spec\_figure")();

 if (name === "style") return Inspector.into(".spec\_figure\_style")();

 });

 }, 200);

 see  for details.

###

 Spectral Convolutions are Node-Order Equivariant

 as for Laplacian polynomial filters.

**Details for the Interested Reader**

 As in [our proof before](#poly-filters-equivariance),

 let’s fix an arbitrary node-order.

 Then, any other node-order can be represented by a

 permutation of this original node-order.

 We can associate this permutation with its permutation matrix PPP.

 Under this new node-order,

 the quantities below transform in the following way:

 x→PxA→PAPTL→PLPTUm→PUm

 \begin{aligned}

 x &\to Px \\

 A &\to PAP^T \\

 L &\to PLP^T \\

 U\_m &\to PU\_m

 \end{aligned}

 xALUm→Px→PAPT→PLPT→PUm

 which implies that, in the embedding computation:

 x^→(PUm)T(Px)=UmTx=x^w^→(PUm)T(Pw)=UmTw=w^g^→g^g→(PUm)g^=P(Umg^)=Pg

 \begin{aligned}

 \hat{x} &\to \left(PU\_m\right)^T (Px) = U\_m^T x = \hat{x} \\

 \hat{w} &\to \left(PU\_m\right)^T (Pw) = U\_m^T w = \hat{w} \\

 \hat{g} &\to \hat{g} \\

 g &\to (PU\_m)\hat{g} = P(U\_m\hat{g}) = Pg

 \end{aligned}

 x^w^g^g→(PUm)T(Px)=UmTx=x^→(PUm)T(Pw)=UmTw=w^→g^→(PUm)g^=P(Umg^)=Pg

 Hence, as σ\sigmaσ is applied elementwise:

 f(Px)=σ(Pg)=Pσ(g)=Pf(x)

 f(Px) = \sigma(Pg) = P \sigma(g) = P f(x)

 f(Px)=σ(Pg)=Pσ(g)=Pf(x)

 as required.

 are unchanged by permutations of the nodes.

 Formally, they are what we would call node-order invariant.

 The theory of spectral convolutions is mathematically well-grounded;

 however, there are some key disadvantages that we must talk about:

 because of the repeated

 multiplications with UmU\_mUm and UmTU\_m^TUmT.

* The learned filters are specific to the input graphs,

 as they are represented in terms

 of the spectral decomposition of input graph Laplacian LLL.

 This means they do not transfer well to new graphs

 which have significantly different structure (and hence, significantly

 different eigenvalues) .

 While spectral convolutions have largely been superseded by

 ‘local’ convolutions for the reasons discussed above,

 there is still much merit to understanding the ideas behind them.

 Indeed, a recently proposed GNN model called Directional Graph Networks

 actually uses the Laplacian eigenvectors

 and their mathematical properties

 extensively.

###

 Global Propagation via Graph Embeddings

 A simpler way to incorporate graph-level information

 is to compute embeddings of the entire graph by pooling node

 (and possibly edge) embeddings,

 and then using the graph embedding to update node embeddings,

 following an iterative scheme similar to what we have looked at here.

 This is an approach used by Graph Networks

 .

 We will briefly discuss how graph-level embeddings

 can be constructed in [Pooling](#pooling).

 However, such approaches tend to ignore the underlying

 topology of the graph that spectral convolutions can capture.

 Learning GNN Parameters

-------------------------

 once a suitable loss function L\mathcal{L}L is defined:

 such as categorical cross-entropy when multiple classes are present:

 L(yv,yv^)=−∑cyvclogyvc^.

 \mathcal{L}(y\_v, \hat{y\_v}) = -\sum\_{c} y\_{vc} \log{\hat{y\_{vc}}}.

 L(yv,yv^)=−c∑yvclogyvc^.

 LG=∑v∈Lab(G)L(yv,yv^)∣Lab(G)∣

 LG=∣Lab(G)∣v∈Lab(G)∑L(yv,yv^)

 where, we only compute losses over labelled nodes Lab(G)\text{Lab}(G)Lab(G).

* **Graph Classification**: By aggregating node representations,

 one can construct a vector representation of the entire graph.

 See [Pooling](#pooling) for how representations of graphs can be constructed.

* **Link Prediction**: By sampling pairs of adjacent and non-adjacent nodes,

 L(yv,yu,evu)=−evulog(pvu)−(1−evu)log(1−pvu)pvu=σ(yvTyu)

 \begin{aligned}

 p\_{vu} &= \sigma(y\_v^Ty\_u)

 \end{aligned}

 L(yv,yu,evu)pvu=−evulog(pvu)−(1−evu)log(1−pvu)=σ(yvTyu)

* **Node Clustering**: By simply clustering the learned node representations.

 The broad success of pre-training for natural language processing models

 such as ELMo  and BERT

 has sparked interest in similar techniques for GNNs

 .

 The key idea in each of these papers is to train GNNs to predict

 local (eg. node degrees, clustering coefficient, masked node attributes)

 mimicking random-walk approaches such as node2vec  and DeepWalk :

LG=∑v∑u∈NR(v)logexpzvTzu∑u′expzu′Tzu.

 LG=v∑u∈NR(v)∑logu′∑expzu′TzuexpzvTzu.

 techniques such as Noise Contrastive Estimation  are especially useful.

 Conclusion and Further Reading

--------------------------------

 While we have looked at many techniques and ideas in this article,

 the field of Graph Neural Networks is extremely vast.

 while still communicating the key ideas and design principles behind GNNs.

 We recommend the interested reader take a look at

  for a more comprehensive survey.

###

 GNNs in Practice

 but we can still represent many GNN update equations using

 For example, the GCN variant discussed here can be represented as:

 h(k)=D−1A⋅h(k−1)W(k)T+h(k−1)B(k)T.

 h^{(k)} = D^{-1} A \cdot h^{(k - 1)} {W^{(k)}}^T + h^{(k - 1)} {B^{(k)}}^T.

 h(k)=D−1A⋅h(k−1)W(k)T+h(k−1)B(k)T.

 such as GPUs.

 Regularization techniques for standard neural networks,

 such as Dropout ,

 can be applied in a straightforward manner to the parameters

 (for example, zero out entire rows of W(k)W^{(k)}W(k) above).

 However, there are graph-specific techniques such as DropEdge

 that removes entire edges at random from the graph,

 that also boost the performance of many GNN models.

###

 Different Kinds of Graphs

 However, there are some simple variants of spatial convolutions for:

* Temporal graphs: Aggregate across previous and/or future node features.

 see  for more discussion.

###

 Pooling

 This article discusses how GNNs compute useful representations

 of nodes.

 But what if we wanted to compute representations of graphs for

 graph-level tasks (for example, predicting the toxicity of a molecule)?

 hG=PREDICTG(AGGv∈G({hv}))

 hG=PREDICTG(AGGv∈G({hv}))

* SortPool: Sort vertices

of the graph to get a fixed-size node-order invariant representation of

the graph, and then apply any standard neural network architecture.

* DiffPool: Learn to

cluster vertices, build a coarser graph over clusters instead of nodes,

then apply a GNN over the coarser graph. Repeat until only one cluster

is left.

* SAGPool: Apply a GNN to

learn node scores, then keep only the nodes with the top scores,

throwing away the rest. Repeat until only one node is left.

 Supplementary Material

------------------------

###

 Reproducing Experiments

 The experiments from

 can be reproduced using the following

###

 Recreating Visualizations

 To aid in the creation of future interactive articles,

 we have created ObservableHQ

 notebooks for each of the interactive visualizations here:

 which pulls together the following standalone notebooks:

Graph Kernels

Node2vec: Scalable Feature Learning for Networks

DeepWalk: Online Learning of Social Representations

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Neural Message Passing for Quantum Chemistry

Learning Convolutional Neural Networks for Graphs

A Tutorial on Spectral Clustering

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Wavelets on Graphs via Spectral Graph Theory

Chebyshev Polynomials

Semi-Supervised Classification with Graph Convolutional Networks

Graph Attention Networks

Inductive Representation Learning on Large Graphs

How Powerful are Graph Neural Networks?

Relational inductive biases, deep learning, and graph networks

Spectral Networks and Locally Connected Networks on Graphs

ImageNet: A Large-Scale Hierarchical Image Database

On the Transferability of Spectral Graph Filters

Directional Graph Networks

Deep contextualized word representations

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Strategies for Pre-training Graph Neural Networks

Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes

When Does Self-Supervision Help Graph Convolutional Networks?

Self-supervised Learning on Graphs: Deep Insights and New Direction

Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

Learning word embeddings efficiently with noise-contrastive estimation

A Comprehensive Survey on Graph Neural Networks

Graph Neural Networks: A Review of Methods and Applications

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

An End-to-End Deep Learning Architecture for Graph Classification

Hierarchical Graph Representation Learning with Differentiable Pooling

Self-Attention Graph Pooling

Understanding Convolutions on Graphs.html


Differentiable Image Parameterizations

Alexander Mordvintsev
Nicola Pezzotti
Ludwig Schubert
Chris Olah

Abstract: Neural networks trained to classify images have a remarkable — and surprising! — capacity to generate images.     Techniques such as DeepDream , style transfer, and feature visualization  leverage this capacity as a powerful tool for exploring the inner  workings of neural networks, and to fuel a small artistic movement based  on neural art.

 Techniques such as DeepDream , style transfer, and feature visualization

 leverage this capacity as a powerful tool for exploring the inner

workings of neural networks, and to fuel a small artistic movement based

 on neural art.

 All these techniques work in roughly the same way.

 Neural networks used in computer vision have a rich internal

representation of the images they look at.

 We can use this representation to describe the properties we want an

 image to have (e.g. style), and then optimize the input image to have

those properties.

 This kind of optimization is possible because the networks are

differentiable with respect to their inputs: we can slightly tweak the

image to better fit the desired properties, and then iteratively apply

such tweaks in gradient descent.

 Typically, we parameterize the input image as the RGB values of each

 pixel, but that isn’t the only way.

 As long as the mapping from parameters to images is differentiable,

we can still optimize alternative parameterizations with gradient

descent.

[1](#figure-differentiable-parameterizations):

 As long as an

image parameterization

is differentiable, we can backpropagate

(    )

through it.

MappingParametersimage/RGB spaceLossFunction

 Differentiable image parameterizations invite us to ask “what kind

of image generation process can we backpropagate through?”

 The answer is quite a lot, and some of the more exotic possibilities

 can create a wide range of interesting effects, including 3D neural

art, images with transparency, and aligned interpolation.

 Previous work using specific unusual image parameterizations

 has shown exciting results — we think that zooming out and looking at

this area as a whole suggests there’s even more potential.

### Why Does Parameterization Matter?

 It may seem surprising that changing the parameterization of an

optimization problem can significantly change the result, despite the

objective function that is actually being optimized remaining the same.

 We see four reasons why the choice of parameterization can have a

significant effect:

**(1) - Improved Optimization** -

Transforming the input to make an optimization problem easier — a

technique called “preconditioning” — is a staple of optimization.

 Preconditioning is most often presented as a transformation of the gradient

 (usually multiplying it by a positive definite “preconditioner” matrix).

**(2) - Basins of Attraction** -

When we optimize the input to a neural network, there are often many

different solutions, corresponding to different local minima.

The probability of our optimization process falling into any particular

local minima is controlled by its basin of attraction (i.e., the region

of the optimization landscape under the influence of the minimum).

Changing the parameterization of an optimization problem is known to

change the sizes of different basins of attraction, influencing the

likely result.

**(3) - Additional Constraints**  -

Some parameterizations cover only a subset of possible inputs, rather

than the entire space.

An optimizer working in such a parameterization will still find

solutions that minimize or maximize the objective function, but they’ll

be subject to the constraints of the parameterization.

By picking the right set of constraints, one can impose a variety of

constraints, ranging from simple constraints (e.g., the boundary of the

image must be black), to rich, subtle constraints.

**(4) - Implicitly Optimizing other Objects** -

 A parameterization may internally use a different kind of object than

the one it outputs and we optimize for.

 For example, while the natural input to a vision network is an RGB

image, we can parameterize that image as a rendering of a 3D object and,

 by backpropagating through the rendering process, optimize that

instead.

 Because the 3D object has more degrees of freedom than the image, we

In the rest of the article we give concrete examples where such

approaches are beneficial and lead to surprising and interesting visual

results.

---

[1](#section-aligned-interpolation)

[Aligned Feature Visualization Interpolation](#section-aligned-interpolation)

-----------------------------------------------------------------------------

 Feature visualization is most often used to visualize individual neurons,

 When we want to really understand the interaction between two neurons,

 we can go a step further and create multiple visualizations,

 Despite this, there is a small challenge: feature visualization is stochastic.

 If we make them naively, the resulting visualizations will be *unaligned*:

 visual landmarks such as eyes appear in different locations in each image.

 because they’re swamped by the much larger differences in layout.

[2](#figure-aligned-interpolation-comparison)

#### Unaligned Interpolation

Visual landmarks, such as eyes, change position from one frame to the next.

pause

#### Aligned Interpolation

Frames are easier to compare because visual landmarks stay in place.

pause

 There are a number of possible approaches one could try

 For example, one could explicitly penalize differences between

adjacent frames. Our final result and our colab notebook use this

technique in combination with a shared parameterization.

[3](#figure-aligned-interpolation-examples)

We start with independently parameterized frames.

Each frame is then combined with a single, shared parameterization…

…to create a visually aligned neuron interpolation.

 By partially sharing a parameterization between frames, we encourage

 the resulting visualizations to naturally align.

 Intuitively, the shared parameterization provides a common reference

 for the displacement of visual landmarks, while the unique one gives to

 each frame its own visual appeal based on its interpolation weights.

 Let’s consider randomly initializing xxx and yyy, and then optimizing them.

 Normally, the optimization problems are independent, so xxx and yyy

 are equally likely to come to unaligned solutions (where they have

different signs) as aligned ones.

 But if we add a shared parameterization, the problems become

coupled and the basin of attraction where they’re aligned becomes

bigger.

![](Differentiable%20Image%20Parameterizations_files/basin-alignment.png)

 This is an initial example of how differentiable parameterizations

in general can be a useful additional tool in visualizing neural

networks.

---

[2](#section-styletransfer)

[Style Transfer with non-VGG architectures](#section-styletransfer)

-------------------------------------------------------------------

 Neural style transfer has a mystery:

 This isn’t because no one is interested in doing style transfer on

other architectures, but because attempts to do it on other

architectures consistently work poorly.

 Several hypotheses have been proposed to explain why VGG works so

much better than other models.

 One suggested explanation is that VGG’s large size causes it to

capture information that other models discard.

 This extra information, the hypothesis goes, isn’t helpful for

classification, but it does cause the model to work better for style

transfer.

 An alternate hypothesis is that other models downsample more

aggressively than VGG, losing spatial information.

 We suspect that there may be another factor: most modern vision

 We find the same approach also improves style transfer, allowing us

to use a model that did not otherwise produce visually appealing style

transfer results:

#### Content Image

#### Style Image

#### Final image Optimization

Pixel Space

Decorrelated Space

[4](#figure-style-transfer-examples):

 Move the slider under “final image optimization” to compare

optimization in pixel space with optimization in a decorrelated space.

Both images were created with the same objective and differ only in

their parameterization.

 Let’s consider this change in a bit more detail. Style transfer

involves three images: a content image, a style image, and the image we

optimize.

 All three of these feed into the CNN, and the style transfer

objective is based on the differences

in how these images activate the CNN.

 The only change we make is how we parameterize the optimized image.

Instead of parameterizing it in terms of pixels (which are highly

correlated with their neighbors), we use a scaled Fourier transform.

[5](#figure-style-transfer-diagram)

---

[3](#section-xy2rgb)

[Compositional Pattern Producing Networks](#section-xy2rgb)

-----------------------------------------------------------

 So far, we’ve explored image parameterizations that are relatively

close to how we normally think of images, using pixels or Fourier

components.

 CPPNs are neural networks that map (x,y)(x,y)(x,y) positions to image colors:

(x,y) →CPPN (r,g,b)(x,y) ~\xrightarrow{\tiny CPPN}~ (r,g,b)(x,y) CPPN (r,g,b)

 By applying the CPPN to a grid of positions, one can make arbitrary

resolution images.

 The parameters of the CPPN network — the weights and

biases — determine what image is produced.

 Depending on the architecture chosen for the CPPN, pixels in the

resulting image are constraint to share, up to a certain degree, the

color of their neighbors.

 Often this is done by evolution ;

 here we explore the possibility to backpropagate some objective

function, such a feature visualization objective.

 This is easily done since the CPPN network is differentiable as the

convolutional neural network and the objective function can be

propagated also through the CPPN to update its parameters accordingly.

 That is to say, CPPNs are a differentiable image

parameterization — a general tool for parameterizing images in any

neural art or visualization task.

WeightsChannelCNNCPPNimage/RGB

[6](#figure-xy2rgb-diagram):

 CPPNs are a differentiable image parameterization. We can use them

 for neural art or visualization tasks by backpropagating past the

image, through the CPPN to its parameters.

 Using CPPNs as image parameterization can add an interesting

 is an artistic medium where images are created by manipulating colorful

 light beams with prisms and mirrors. Notable examples of this technique

 are the [work of Stephen Knapp](http://www.lightpaintings.com/).

Note

 that light-painting metaphor here is rather fragile: for example light

composition is an additive process, while CPPNs can have

negative-weighted connections between layers.

[7](#figure-xy2rgb-examples):

 The visual quality of the generated images is heavily influenced by

the architecture of the chosen CPPN.

 Not only the shape of the network, i.e., the number of layers and

filters, plays a role, but also the chosen activation functions and

normalization. For example, deeper networks produce more fine grained

details compared to shallow ones.

 We encourage readers to experiment in generating different images by

changing the architecture of the CPPN. This can be easily done by

changing the code in the supplementary notebook.

 The evolution of the patterns generated by the CPPN are artistic

artifacts themselves.

 To maintain the metaphor of light-paintings, the optimization process

correspond to an iterative adjustments of the beam directions and

shapes.

 Because the iterative changes have a more global effect compared to,

for example, a pixel parameterization, at the beginning of the

optimization only major patterns are visible.

 By iteratively adjusting the weights, our imaginary beams are

positioned in such a way that fine details emerge.

![](Differentiable%20Image%20Parameterizations_files/pointer.svg)

[8](#figure-xy2rgb-training):

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 By playing with this metaphor, we can also create a new kind of

animation that morph one of the above images into a different one.

 Intuitively, we start from one of the light-paintings and we move the

beams to create a different one.

 This result is in fact achieved by interpolating the weights of the

CPPN representations of the two patterns. A number of intermediate

frames are then generated by generating an image given the interpolated

CPPN representation.

 As before, changes in the parameter have a global effect and create

visually appealing intermediate frames.

![](Differentiable%20Image%20Parameterizations_files/pointer.svg)

[9](#figure-xy2rgb-interpolation):

 Interpolating CPPN weights between two learned points.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 Your browser does not support the video tag.

 In this section we presented a parameterization that goes beyond a

standard image representation.

 Neural networks, a CPPN in this case, can be used to parameterize an

image that is optimized for a given objective function.

 More specifically, we combined a feature-visualization objective

function with a CPPN parameterization to create infinite-resolution

images of distinctive visual style.

---

[4](#section-rgba)

[Generation of Semi-Transparent Patterns](#section-rgba)

--------------------------------------------------------

 of images instead of a single image, and then sampling one or a few

images from that family at each optimization step.

 This is important because many of the objects we’ll explore

optimizing have more degrees of freedom than the images going into the

network.

 To be concrete, let’s consider the case of semi-transparent images.

These images have, in addition to the RGB channels, an alpha channel

that encodes each pixel’s opacity (in the range [0,1][0,1][0,1]).

 In order to feed such images into a neural network trained on RGB

images, we need to somehow collapse the alpha channel. One way to

 where IaI\_aIa is the alpha channel of the image III.

 If we used a static background BGBGBG,

 such as black, the transparency would merely indicate pixel positions

in which that background contributes directly to the optimization

objective.

 In fact, this is equivalent to optimizing an RGB image and making it

 transparent in areas where its color matches with the background!

 Intuitively, we’d like transparent areas to correspond to something

like “the content of this area could be anything.”

 Building on this intuition, we use a different random background at

every optimization step.

 We have tried both sampling from real images, and using different

types of noise.

 As long as they were sufficiently randomized, the different

distributions did not meaningfully influence the resulting optimization.

 Thus, for simplicity, we use a smooth 2D gaussian noise.

ChannelCNNrandom noise

[10](#figure-rgba-diagram):

 By default, optimizing our semi-transparent image will make the

image fully opaque, so the network can always get its optimal input.

 To avoid this, we need to change our objective with an objective

that encourages some transparency.

 We find it effective to replace the original objective with:

 with reducing the mean transparency.

 If the image becomes very transparent, it will focus on the original

 objective. If it becomes too opaque, it will temporarily stop caring

about the original objective and focus on decreasing the average

opacity.

bright

dark

checkerboard

Alpha mask

masked

opaque

[11](#figure-rgba-examples):

 It turns out that the generation of semi-transparent images is

useful in feature visualization.

 Feature visualization aims to understand what neurons in a vision

model are looking for, by creating images that maximally activate them.

 Unfortunately, there is no way for these visualizations to

distinguish which areas of an image strongly influence a neuron’s

activation and those which only marginally do so.

 This issue does not occur when

optimizing for the activation of entire channels, because in that case

every pixel has multiple neurons that are close to centered on it. As a

consequence, the entire input image gets filled with copies of what

those neurons care about strongly.

 Ideally, we would like a way for our visualizations to make this

distinction in importance — one natural way to represent that a part of

the image doesn’t matter is for it to be transparent.

 Thus, if we optimize an image with an alpha channel and encourage

the overall image to be transparent, parts of the image that are

unimportant according to the feature visualization objective should

become transparent.

[12](#figure-rgba-interpretability-examples)

figure 12

Introducing transparency helps separate those areas.

---

[5](#section-featureviz-3d)

[Efficient Texture Optimization through 3D Rendering](#section-featureviz-3d)

-----------------------------------------------------------------------------

 We use a 3D rendering process to turn them into 2D RGB images that

can be fed into the network, and backpropagate through the rendering

process to optimize the texture of the 3D object.

 Our technique is similar to the approach that Athalye et al.

 used for the creation of real-world adversarial examples, as we rely on

 the backpropagation of the objective function to randomly sampled views

 of the 3D model.

 We differ from existing approaches for artistic texture generation,

 as we do not modify the geometry of the object during back-propagation.

 By disentangling the generation of the texture from the position of

their vertices, we can create very detailed texture for complex objects.

Before we can describe our approach, we first need to understand how a

3D object is stored and rendered on screen. The object’s geometry is

 coordinate in the texture image. The model is then rendered, i.e. drawn

 on screen, by coloring every triangle with the region of the image that

 is delimited by the (u,v)(u,v)(u,v) coordinates of its vertices.

 A simple naive way to create the 3D object texture would be to

optimize an image the normal way and then use it as a texture to paint

on the object.

 However, this approach generates a texture that does not consider

the underlying UV-mapping and, therefore, will create a variety of

visual artifacts in the rendered object.

 First, **seams** are visible on the rendered texture, because the

 optimization is not aware of the underlying UV-mapping and, therefore,

does not optimize the texture consistently along the split patches of

the texture.

 Second, the generated patterns are **randomly oriented** on

different parts of the object (see, e.g., the vertical and wiggly

patterns) because they are not consistently oriented in the underlying

UV-mapping.

#### Input Controls

 seams

 or

 wrongly oriented patterns.

Unfold Texture

#### Naïve Optimization

#### Render-based Optimization

[13](#figure-featureviz-3d-explanation):

 3D model of the famous Stanford Bunny.

 You can interact with the model by rotating and zooming. Moreover, you

can unfold the object to its two-dimensional texture representation.

This unfolding reveals the UV mapping used to store the texture in the

texture image. Note how the render-based optimized texture is divided in

 several patches that allows for a complete and undistorted coverage of

the object.

 We take a different approach.

 The following diagram presents an overview of the proposed pipeline:

[14](#figure-featureviz-3d-diagram):

 We optimize a texture by backpropagating through the rendering

process. This is possible because we know how pixels in the rendered

image correspond to pixels in the texture.

We start the process by randomly initializing the texture with a Fourier

 parameterization.

At every training iteration we sample a random camera position, which is

 oriented towards the center of the bounding box of the object, and we

render the textured object as an image.

We then backpropagate the gradient of the desired objective function,

i.e., the feature of interest in the neural network, to the rendered

image.

However, an update of the rendered image does not correspond to an

update to the texture that we aim at optimizing. Hence, we need to

further propagate the changes to the object’s texture.

The propagation is easily implemented by applying a reverse UV-mapping,

as for each pixel on screen we know its coordinate in the texture.

By modifying the texture, during the following optimization iterations,

the rendered image will incorporate the changes applied in the previous

iterations.

Unfold Texture

[15](#figure-featureviz-3d-examples):

The resulting textures are consistently optimized along the cuts, hence

removing the seams and enforcing an uniform orientation for the rendered

 object.

Morever, since the function optimization is disentangled by the geometry

 of the object, the resolution of the texture can be arbitrary high.

In the next section we will se how this framework can be reused for

performing an artistic style transfer to the object’s texture.

---

[6](#section-style-transfer-3d)

[Style Transfer for Textures through 3D Rendering](#section-style-transfer-3d)

------------------------------------------------------------------------------

Now that we have established a framework for efficient backpropagation

into the UV-mapped texture, we can use it to adapt existing style

transfer techniques for 3D objects.

Similarly to the 2D case, we aim at redrawing the original object’s

texture with the style of a user-provided image.

The following diagram presents an overview of the approach:

[16](#figure-style-transfer-3d-diagram)

The algorithm works in similar way to the one presented in the previous

section, starting from a randomly initialized texture.

At each iteration, we sample a random view point oriented toward the

center of the bounding box of the object and we render two images of it:

#### Content Model

#### Style Image

#### Model with optimized texture

Unfold Texture

[17](#figure-style-transfer-3d-examples):

 Style Transfer onto various 3D models. Note that visual landmarks in

 the content texture, such as eyes, show up correctly in the generated

texture.

Because every view is optimized independently, the optimization is

forced to try to add all the style’s elements at every iteration.

For example, if we use as style image the Van Gogh’s “Starry Night”

painting, stars will be added in every single view.

We found we obtain more pleasing results, such as those presented above,

 by introducing a sort of “memory” of the style of

previous views. To this end, we maintain moving averages of

style-representing Gram matrices

over the recently sampled viewpoints. On each optimization iteration we

compute the style loss against those averaged matrices,

instead of the ones computed for that particular view.

 to the current Gram matrices.

 An alternative approach, such as the one employed by ,

 would require sampling multiple viewpoints of the scene at each step,

 increasing memory requirements. In contrast, our substitution trick can be also

 used to apply style transfer to high resolution (>10M pixels) images on a

 single consumer-grade GPU.

Take as an example the model created by imposing Van Gogh’s starry night

 as style image.

The resulting texture contains the rythmic and vigorous brush strokes

that characterize Van Gogh’s work.

However, despite the style image’s primarily cold tones, the resulting

fur has a warm orange undertone as it is preserved from the original

texture.

Even more interesting is how the eyes of the bunny are preserved when

different styles are transfered.

For example, when the style is obtained from the Van Gogh’s painting,

![](Differentiable%20Image%20Parameterizations_files/printed_bunny_extended.jpg)

[18](#figure-style-transfer-3d-picture):

 Textured models produced with the presented method can be easily used

with popular 3D modeling software or game engines. To show this, we 3D

---

[Conclusions](#conclusions)

---------------------------

 For the creative artist or researcher, there’s a large space of ways

 to parameterize images for optimization.

 This opens up not only dramatically different image results, but

also animations and 3D objects!

 We think the possibilities explored in this article only scratch the

 surface.

 For example, one could explore extending the optimization of 3D

object textures to optimizing the material or reflectance — or even go

the direction of Kato et al. and optimize the mesh vertex positions.

 This article focused on *differentiable* image

parameterizations, because they are easy to optimize and cover a wide

range of possible applications.

 But it’s certainly possible to optimize image parameterizations that

 aren’t differentiable, or are only partly differentiable, using

reinforcement learning or evolutionary strategies .

Inceptionism: Going deeper into neural networks

A Neural Algorithm of Artistic Style

Feature Visualization

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Synthesizing robust adversarial examples

The loss surfaces of multilayer networks

Very deep convolutional networks for large-scale image recognition

Deconvolution and checkerboard artifacts

Deep Image Prior

Compositional pattern producing networks: A novel abstraction of development

Neural Network Generative Art in Javascript

Image Regression

Generating Large Images from Latent Vectors

Artificial Evolution for Computer Graphics

Neural 3D Mesh Renderer

The Stanford Bunny

Natural evolution strategies.

Evolution strategies as a scalable alternative to reinforcement learning

Tensorflow: Large-scale machine learning on heterogeneous distributed systems


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Learning from Incorrectly Labeled Data

Abstract: This article is part of a discussion of the Ilyas et al. paper         “Adversarial examples are not bugs, they are features”.         You can learn more in the                      main discussion article         .

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *“Adversarial examples are not bugs, they are features”.*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

 about the trained model is being “leaked” into the dataset.

 Nevertheless, we show that this intuition fails — a model *can* generalize.

 examples on the left of the Figure below:

[1](https://distill.pub/2019/advex-bugs-discussion/response-6/#figure-1)

 incorrectly labeled, unperturbed images but can still non-trivially generalize.

This is Model Distillation Using Incorrect Predictions

------------------------------------------------------

 of model distillation — training on this dataset allows a new

 model to somewhat recover the features of the original model.

 another task.

### Two-dimensional Illustration of Model Distillation

 (panel (a) in the Figure below).

[2](https://distill.pub/2019/advex-bugs-discussion/response-6/#figure-2)

 panel (c).

 predictions.

###

 Other Peculiar Forms of Distillation

 we learn about the original model? Can we use only *out-of-domain* data?

 labeled as an “8″.

[3](https://distill.pub/2019/advex-bugs-discussion/response-6/#figure-3)

 training on erroneous predictions.

###

 Summary

 Ilyas et al. (2019) are not necessary to enable learning.

 To cite Ilyas et al.’s response, please cite their

**Response

 Summary**: Note that since our experiments work across different architectures,

 “distillation” in weight space does not occur. The only distillation that can

 have been able to “distill” a useful model from them. (In fact, one might think

 of normal model training as just “feature distillation” of the humans that

 labeled the dataset.) Furthermore, the hypothesis that all we need is enough

 model-consistent points in order to recover a model, seems to be disproven by

 and other (e.g. ) settings.

**Response**: Since our experiments work across different architectures,

 “distillation” in weight space cannot arise. Thus, from what we understand, the

 “distillation” hypothesis suggested here is referring to “feature distillation”

 (i.e. getting models which use the same features as the original), which is

 actually precisely our hypothesis too. Notably, this feature distillation would

 are good for classification (see [World

 1](https://distill.pub/2019/advex-bugs-discussion/original-authors/#world1) and

 model would only use features that generalize poorly, and would thus generalize

 poorly itself.

 Moreover, we would argue that in the experiments presented (learning from

 mislabeled data), the same kind of distillation is happening. For instance, a

 moderately accurate model might associate “green background” with “frog” thus

 labeling “green” images as “frogs” (e.g., the horse in the comment’s figure).

 Training a new model on this dataset will thus associate “green” with “frog”

 from Fashion-MNIST” experiment in the comment). This corresponds exactly to

 learning features from labels, akin to how deep networks “distill” a good

 decision boundary from human annotators. In fact, we find these experiments

 a very interesting illustration of feature distillation that complements

 our findings.

 We also note that an analogy to logistic regression here is only possible

 due to the low VC-dimension of linear classifiers (namely, these classifiers

 have dimension ddd). In particular, given any classifier with VC-dimension

 networks have been shown to have extremely large VC-dimension (in particular,

 bigger than the size of the training set ). So even though

 labelling d+1d+1d+1 random

 points model-consistently is sufficient to recover a linear model, it is not

 necessarily sufficient to recover a deep neural network. For instance, Milli et

 al.  are not able to reconstruct a ResNet-18

 using only its predictions on random Gaussian inputs. (Note that we are using a

 ResNet-50 in our experiments.)

 Finally, it seems that the only potentially problematic explanation for

 our experiments (namely, that enough model-consistent points can recover a

 In particular, Preetum is able to design a

 dataset where training on mislabeled inputs *that are model-consistent*

 does not at all recover the decision boundary of the original model. More

 generally, the “model distillation” perspective raised here is unable to

 distinguish between the dataset created by Preetum below, and those created

 with standard PGD (as in our D^det\widehat{\mathcal{D}}\_{det}D

det and

 D^rand\widehat{\mathcal{D}}\_{rand}D

rand datasets).

Distilling the Knowledge in a Neural Network

Model reconstruction from model explanations

Understanding deep learning requires rethinking generalization

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'_ Learning from Incorrectly Labeled Data.html