Understanding RL Vision

Jacob Hilton
Nick Cammarata
Shan Carter
Gabriel Goh
Chris Olah

### Contents

[Introduction](#introduction)

[Our CoinRun model](#coinrun)

[Model analysis](#analysis)

* [Dissecting failure](#dissecting-failure)

* [Hallucinations](#hallucinations)

* [Model editing](#model-editing)

[The diversity hypothesis](#diversity-hypothesis)

[Feature visualization](#feature-visualization)

[Attribution](#attribution)

[Questions for further research](#questions)

 In this article, we apply interpretability techniques to a

reinforcement learning (RL) model trained to play the video game CoinRun

 . Using attribution  combined with dimensionality reduction as in ,

 we build an interface for exploring the objects detected by the model,

and how they influence its value function and policy. We leverage this

interface in several ways.

* **[Dissecting failure](#dissecting-failure).**

 We perform a step-by-step analysis of the agent’s behavior in cases

where it failed to achieve the maximum reward, allowing us to understand

 what went wrong, and why. For example, one case of failure was caused

by an obstacle being temporarily obscured from view.

* **[Hallucinations](#hallucinations).** We find

 situations when the model “hallucinated” a feature not present in the

observation, thereby explaining inaccuracies in the model’s value

function. These were brief enough that they did not affect the agent’s

behavior.

* **[Model editing](#model-editing).** We

hand-edit the weights of the model to blind the agent to certain

hazards, without otherwise changing the agent’s behavior. We verify the

effects of these edits by checking which hazards cause the new agents to

 fail. Such editing is only made possible by our previous analysis, and

thus provides a quantitative validation of this analysis.

 for interpretability. If it is correct, then we can expect RL models to

 become more interpretable as the environments they are trained on

become more diverse. We provide evidence for our hypothesis by measuring

 the relationship between interpretability and generalization.

Our CoinRun model

-----------------

 CoinRun is a side-scrolling platformer in which the agent must

dodge enemies and other traps and collect the coin at the end of the

level.

 CoinRun is procedurally-generated, meaning that each new level

encountered by the agent is randomly generated from scratch. This

incentivizes the model to learn how to spot the different kinds of

objects in the game, since it cannot get away with simply memorizing a

|  |  |  |  |  |  |  |  |

| --- | --- | --- | --- | --- | --- | --- | --- |

| Full resolution |

 |  |  |

 |

 |  |  |

| Model resolution |

 |  |  |

 |

 |  |  |

 in the velocity info allows the model to infer the agent’s motion from a

 single frame. The shade of the left square indicates the agent’s

horizontal velocity (black for left at full speed, white for right at

full speed), and the shade of the right square indicates the agent’s

vertical velocity (black for down at full speed, white for up at full

speed). In this example, the agent is moving forward and about to land

(and is thus moving right and down). |

 There are 9 actions available to the agent in CoinRun:

|  |  |  |  |

| --- | --- | --- | --- |

| ← | → |  | Left and right change the agent’s horizontal

velocity. They still work while the agent is in mid-air, but have less

of an effect. |

| ↑ | ↖ | ↗ | Up causes the agent to jump after the next

non-up action. Diagonal directions have the same effect as both

component directions combined. |

| A | B | C | A, B and C do nothing.The

 original version of CoinRun only has 1 “do nothing” action, but our

version ended up with 3 when “A” and “B” actions were added to be used

in other games. For consistency, we have relabeled the original “do

nothing” action as “C”. |

 except that we used twice as many copies of the environment per worker

and twice and many workers. The effect of these changes was to increase

the effective batch size, which seemed to be necessary to reach the same

 We used a non-recurrent network, to avoid any need to visualize

multiple frames at once. Thus our model observes a single downsampled

64x64 image, and outputs a value function (an estimate of the total

future time-discounted reward) and a policy (a probability distribution

over the actions, from which the next action is sampled).

observation

CNN

value function

logits

softmax

policy

Model analysis

--------------

 we developed an interface for examining trajectories of the agent

playing the game. This incorporates attribution from a hidden layer that

 recognizes objects, which serves to highlight objects that positively

or negatively influence a particular network output. By applying

dimensionality reduction, we obtain attribution vectors whose components

 correspond to different types of object, which we indicate using

different colors.

 Here is our interface for a typical trajectory, with the value

function as the network output. It reveals the model using obstacles,

coins, enemies and more to compute the value function.

| Value function attribution |

| --- |

| Observation

  Video game pixels seen by the model | Positive attribution

 (bad news)  Color overlay shows objects predictive of failure |

| --- | --- | --- |

|     |      |      |

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

  Timeline ← ►

 Speed:  FPS →

        Value function

 Estimated discounted win probability 93.9%

### Dissecting failure

 Our fully-trained model fails to complete around 1 in every 200

levels. We explored a few of these failures using our interface, and

found that we were usually able to understand why they occurred.

 The failure often boils down to the fact that the model has no

memory, and must therefore choose its action based only on the current

observation. It is also common for some unlucky sampling of actions from

 the agent’s policy to be partly responsible.

|  |  |

| --- | --- |

|

 **Buzzsaw obstacle obscured by enemy**

 **Stepping down to avoid jumping**

 **Landing platform moving off-screen**

 |

 The agent moves too far to the right while in mid-air as a

 result of a buzzsaw obstacle being temporarily hidden from view by a

moving enemy. The buzzsaw comes back into view, but too late to avoid a

collision.

 The agent presses down in a bid to delay a jump. This

causes the agent to inadvertently step down from a box and onto an

enemy.

 The agent fails to move far enough to the right while in

mid-air, as a result of the platform where it was intending to land

moving below the field of view.

 |

Prev

Start

►

Next

End

Timestep **1** of **18**

 The agent jumps by releasing up, analyzing something (◦) on the wall.

(The "residual" feature can be triggered by a number of different

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO).**Timesteps 4–6**:

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO).**Timesteps 4–6**:

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

 Having adjusted its horizontal motion, the agent returns to a policy

 Having adjusted its horizontal motion, the agent returns to a policy

 Having adjusted its horizontal motion, the agent returns to a policy

 Having adjusted its horizontal motion, the agent returns to a policy

 Having adjusted its horizontal motion, the agent returns to a policy

 becomes clear to the agent again. The agent tries to move left to avoid

 it, and also prepares to immediately jump upon landing. Pulling off the

 jump actually matters more to its chances of survival, since it is

already moving right.**Timesteps 16–17**: The danger of the buzzsaw obstacle (•)

 becomes clear to the agent again. The agent tries to move left to avoid

 it, and also prepares to immediately jump upon landing. Pulling off the

 jump actually matters more to its chances of survival, since it is

 immediately in front of it. The agent cannot jump when it is in

mid-air, but jumping is only triggered when up is released, and it is

allowed to start pressing up before landing.**Timesteps 2–3**:

 The agent's policy is dominated by the 3 upward directions (which delay

 the jump) and down (which cancels it). Any other action would trigger

the jump. So the agent is trying to delay the jump, partly because of

 The agent's policy is dominated by the 3 upward directions (which delay

 the jump) and down (which cancels it). Any other action would trigger

the jump. So the agent is trying to delay the jump, partly because of

 The agent's policy is still dominated by the 3 updward directions and

down, and at this point the agent happens to move down. Unfortunately,

the agent does not seem to have taken into account that this will cause

 across the other side of the chasm ahead of it. Even though it is not

yet at the edge of the platform it is currently on, it is timing its

jump so as to land on the nearest platform up ahead, perhaps because it

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO). Unfortunately, it happens

to move left at every timestep during this period.**Timesteps 4–6**:

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO). Unfortunately, it happens

to move left at every timestep during this period.**Timesteps 4–6**:

 The agent is in mid-air, trying to control its horizontal motion. Its

policy has higher entropy as its actions matter less at the start of a

jump (due to the entropy bonus used in PPO). Unfortunately, it happens

 has been reduced, and tries to move right to compensate. (Several

features read from the velocity info as a secondary purpose.) With some

 has been reduced, and tries to move right to compensate. (Several

features read from the velocity info as a secondary purpose.) With some

bad luck, it moves up but not right on its first move.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

bias is not reflected in the actions sampled.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

bias is not reflected in the actions sampled.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

bias is not reflected in the actions sampled.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

bias is not reflected in the actions sampled.**Timesteps 9–13**:

 The platforms have now moved below the agent's field of view. With

little to go by, its policy has high entropy, with some bias towards

moving right rather than left. With further bad luck, this rightward

|

 Attribution for: | → | show choices |  |

 Policy: (probabilities and sampled action) | ←→↓↑↗↖ABC |

| --- | --- | --- | --- | --- | --- |

| Choose attribution: |

 Value function |  ←  →  ↓  ↑  ↗  ↖  A  B  C  |

| --- | --- | --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

|

 Attribution for: | Value function | show choices |  |

 Policy: (probabilities and sampled action) | ←→↓↑↗↖ABC |

| --- | --- | --- | --- | --- | --- |

| Choose attribution: |

 Value function |  ←  →  ↓  ↑  ↗  ↖  A  B  C  |

| --- | --- | --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

|

 Attribution for: | Value function | show choices |  |

 Policy: (probabilities and sampled action) | ←→↓↑↗↖ABC |

| --- | --- | --- | --- | --- | --- |

| Choose attribution: |

 Value function |  ←  →  ↓  ↑  ↗  ↖  A  B  C  |

| --- | --- | --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

### Hallucinations

 which measures how successful each action turned out relative to the

agent’s expectations. An unusually high or low GAE indicates that either

 something unexpected occurred, or the agent’s expectations were

miscalibrated. Filtering for such timesteps can therefore find problems

with the value function or policy.

 Using our interface, we found a couple of cases in which the model

 “hallucinated” a feature not present in the observation, causing the

value function to spike.

|  |  |

| --- | --- |

|

 **Coin hallucination**

 **Buzzsaw hallucination**

 |

 At one point the value function spiked upwards from 95% to

 98% for a single timestep. This was due to a curved yellow-brown shape

in the background, which happened to appear next to a wall, being

mistaken for a coin.

 At another point the value function spiked downwards from

94% to 85% for a single timestep. This was due to the agent, colored in

gray-blue and crouching against a mottled background, being mistaken for

 a buzzsaw obstacle. An actual buzzsaw was also present in the

observation, but the main effect was from the misjudged agent, as shown

by the larger red circle around the agent (hover over the first legend

item to isolate).

 |

        Value function  98.0%

| Value function attribution |

| --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

        Value function  85.0%

| Value function attribution |

| --- |

| Observation

  | Positive attribution

  | Negative attribution

  |

| --- | --- | --- |

|     |      |      |

 Attribution totals

 (colors summed over spatial positions) 0

 Attribution legend

 (hover to isolate)

   Buzzsaw

obstacle  Coin  Enemy

moving

left  Agent

or enemy

moving right  Buzzsaw

obstacle

or platform  Platform  Velocity

info or left-

facing wall  Step hover

to show residual

 Everything

else

### Model editing

 Our analysis so far has been mostly qualitative. To quantitatively

 validate our analysis, we hand-edited the model to make the agent blind

 to certain features identified by our interface: buzzsaw obstacles in

one case, and left-moving enemies in another. Our method for this can be

 We evaluated each edit by measuring the percentage of levels that

the new agent failed to complete, broken down by the object that the

agent collided with to cause the failure. Our results show that our

edits were successful and targeted, with no statistically measurable

effects on the agent’s other abilities.The data for this plot are as follows.

- Original model: 0.37% / 0.16% / 0.12% / 0.08%

- Buzzsaw obstacle blindness: 12.76% / 0.16% / 0.08% / 0.05%

- Enemy moving left blindness: 0.36% / 4.69% / 0.97% / 0.07%

Each model was tested on 10,000 levels.

 We did not manage to achieve complete blindness, however: the

buzzsaw-edited model still performed significantly better than the

Original model, invisible buzzsaws: 32.20% / 0.05% / 0.05% / 0.05%

We tested the model on 10,000 levels.

We

 experimented briefly with iterating the editing procedure, but were not

 able to achieve more than around 50% buzzsaw blindness by this metric

 Here are the original and edited models playing some cherry-picked levels.

|  |  |  |  |

| --- | --- | --- | --- |

|

 **Level 1**

 **Level 2**

 **Level 3**

►

►

►

 |

 |

 |

 |

The diversity hypothesis

------------------------

 All of the above analysis uses the same hidden layer of our

network, the third of five convolutional layers, since it was much

harder to find interpretable features at other layers. Interestingly,

the level of abstraction at which this layer operates – finding the

locations of various in-game objects – is exactly the level at which

CoinRun levels are randomized using procedural generation. Furthermore,

we found that training on many randomized levels was essential for us to

 be able to find any interpretable features at all.

 This led us to suspect that the diversity introduced by CoinRun’s

randomization is linked to the formation of interpretable features. We

call this the diversity hypothesis:

>

>  Interpretable features tend to arise (at a given level of

> abstraction) if and only if the training distribution is diverse enough

> (at that level of abstraction).

>

 Our explanation for this hypothesis is as follows. For the forward

 implication (“only if”), we only expect features to be interpretable if

 they are general enough, and when the training distribution is not

diverse enough, models have no incentive to develop features that

generalize instead of overfitting. For the reverse implication (“if”),

we do not expect it to hold in a strict sense: diversity on its own is

not enough to guarantee the development of interpretable features, since

 they must also be relevant to the task. Rather, our intention with the

reverse implication is to hypothesize that it holds very often in

practice, as a result of generalization being bottlenecked by diversity.

 of each level is randomized, and correspondingly, we were only able to

find interpretable features at the level of abstraction of objects. At a

 lower level, there are only a handful of visual patterns in the game,

and the low-level features of our model seem to consist mostly of

memorized color configurations used for picking these out. Similarly,

the game’s high-level dynamics follow a few simple rules, and

accordingly the high-level features of our model seem to involve

mixtures of combinations of objects that are hard to decipher. To

### Interpretability and generalization

 To test our hypothesis, we made the training distribution less

diverse, by training the agent on a fixed set of 100 levels. This

dramatically reduced our ability to interpret the model’s features. Here

 we display an interface for the new model, generated in the same way as

 the one [above](#interface). The smoothly increasing value

function suggests that the model has memorized the number of timesteps

until the end of the level, and the features it uses for this focus on

irrelevant background objects. Similar overfitting occurs for other

video games with a limited number of levels .

| Value function attribution |

| --- |

| Observation

  Video game pixels seen by the model | Positive attribution

 (bad news)  Color overlay shows objects predictive of failure |

| --- | --- | --- |

|     |      |      |

 Attribution legend

 (hover to isolate)

    ???   ???   ???   ???   ???   ???   ???  Agent, walls

and velocity

info? hover

to show residual

 Everything

else

  Timeline ← ►

 Speed:  FPS →

        Value function

 Estimated discounted win probability 94.4%

 We attempted to quantify this effect by varying the number of

levels used to train the agent, and evaluating the 8 features identified

 Features were scored based on how consistently they focused on the same

 objects, and whether the value function attribution made sense – for

example, background objects should not be relevant. This process was

subjective and noisy, but that may be unavoidable. We also measured the

generalization ability of each model, by testing the agent on unseen

levels .The data for this plot are as follows.

Comparison of models trained on different numbers of

levels. Two models were trained for each number of levels, and two

researchers independently evaluated how interpretable the features of

each model were, without being shown the number of levels.Our

 methodology had some flaws. Firstly, the researchers were not

completely blind to the number of levels: for example, it is possible to

 infer something about the number of levels from the smoothness of

graphs of the value function, since with fewer levels the model is

better able to memorize the number of timesteps until the end of the

level. Secondly, since evaluations are somewhat tedious, we stopped them

 once we thought the trend had become clear, introducing some selection

bias. Therefore these results should be considered primarily

illustrative. Each model was tested on 10,000 train and

10,000 test levels sampled with replacement. Shaded areas in the left

plot show the range of values over both models, though these are mostly

too narrow to be visible. Error bars in the right plot show ±1

population standard deviation over all four model–researcher pairs.

 Our results illustrate how diversity may lead to interpretable

features via generalization, lending support to the diversity

hypothesis. Nevertheless, we still consider the hypothesis to be highly

unproven.

Feature visualization

---------------------

[Feature visualization](https://distill.pub/2017/feature-visualization/)

 answers questions about what certain parts of a network are looking for

 by generating examples. This can be done by applying gradient descent

to the input image, starting from random noise, with the objective of

activating a particular neuron or group of neurons. While this method

works well for an image classifier trained on ImageNet ,

 for our CoinRun model it yields only featureless clouds of color. Only

for the first layer, which computes simple convolutions of the input,

does the method produce comparable visualizations for the two models.

|  | ImageNet | CoinRun |

| --- | --- | --- |

|

 First layer

  |

 |

 |

|

 Intermediate layer

  |

 |

 |

 Each image was chosen to activate a neuron in the center, with the 3

images corresponding to the first 3 channels. Jittering was applied

between optimization steps of up to 2 pixels for the first layer, and up

 To try to get it to work for CoinRun, we varied the method in a number

of ways. Nothing we tried had any noticeable effect on the quality of

the visualizations.

* **Transformation robustness.** This is the method of

stochastically jittering, rotating and scaling the image between

optimization steps, to search for examples that are robust to these

transformations . We tried both

increasing and decreasing the size of the jittering. Rotating and

scaling are less appropriate for CoinRun, since the observations

themselves are not invariant to these transformations.

* **Penalizing extremal colors.**By

 an “extremal” color we mean one of the 8 colors with maximal or minimal

 RGB values (black, white, red, green, blue, yellow, cyan and magenta).

 Noticing that our visualizations tend to use extremal colors towards

the middle, we tried including in the visualization objective an L2

penalty of various strengths on the activations of the first layer,

which successfully reduced the size of the extremally-colored region but

 did not otherwise help.

 caricature objective is to maximize the dot product between the

activations of the input image and the activations of a reference image.

 Caricatures are often an especially easy type of feature visualization

to make work, and helpful for getting a first glance into what features a

* **Low-level visual diversity.** In an attempt to

broaden the distribution of images seen by the model, we retrained it on

 a version of the game with procedurally-generated sprites. We

additionally tried adding noise to the images, both independent

per-pixel noise and spatially-correlated noise. Finally, we experimented

 As shown [below](#dataset-examples), we were able to

use dataset examples to identify a number of channels that pick out

human-interpretable features. It is therefore striking how resistant

gradient-based methods were to our efforts. We believe that this is

because solving CoinRun does not ultimately require much visual ability.

 Even with our modifications, it is possible to solve the game using

simple visual shortcuts, such as picking out certain small

configurations of pixels. These shortcuts work well on the narrow

distribution of images on which the model is trained, but behave

unpredictably in the full space of images in which gradient-based

optimization takes place.

 In support of the hypothesis, we have examples of features that are

hard to interpret in the absence of diversity. But there is also

evidence that the hypothesis may need to be refined. Firstly, it seems

to be a lack of diversity at a low level of abstraction that harms our

ability to interpret features at all levels of abstraction, which could

be due to the fact that gradient-based feature visualization needs to

back-propagate through earlier layers. Secondly, the failure of our

efforts to increase low-level visual diversity suggests that diversity

may need to be assessed in the context of the requirements of the task.

### Dataset example-based feature visualization

 As an alternative to gradient-based feature visualization, we use

dataset examples. This idea has a long history, and can be thought of as

 a heavily-regularized form of feature visualization .

 In more detail, we sample a few thousand observations infrequently from

 the agent playing the game, and pass them through the model. We then

apply a dimensionality reduction method known as non-negative matrix

factorization (NMF) to the activation channels .More

 precisely, we find a non-negative approximate low-rank factorization of

 the matrix obtained by flattening the spatial dimensions of the

activations into the batch dimension. This matrix has one row per

 For each of the resulting channels (which correspond to weighted

combinations of the original channels), we choose the observations and

spatial positions with the strongest activation (with a limited number

of examples per position, for diversity), and display a patch from the

observation at that position.

![](Understanding%20RL%20Vision_files/layer_2b_feature_6.png)Coin

![](Understanding%20RL%20Vision_files/layer_2b_feature_9.png)Step

![](Understanding%20RL%20Vision_files/layer_2b_feature_11.png)Left edge of box

![](Understanding%20RL%20Vision_files/layer_2b_feature_13.png)Buzzsaw obstacle

 Unlike gradient-based feature visualization, this method finds

some meaning to the different directions in activation space. However,

it may still fail to provide a complete picture for each direction,

since it only shows a limited number of dataset examples, and with

limited context.

### Spatially-aware feature visualization

 CoinRun observations differ from natural images in that they are

much less spatially invariant. For example, the agent always appears in

the center, and the agent’s velocity is always encoded in the top left.

As a result, some features detect unrelated things at different spatial

positions, such as reading the agent’s velocity in the top left while

detecting an unrelated object elsewhere. To account for this, we

developed a spatially-aware version of dataset example-based feature

visualization, in which we fix each spatial position in turn, and choose

 the observation with the strongest activation at that position (with a

limited number of reuses of the same observation, for diversity). This

creates a spatial correspondence between visualizations and

observations.

 Here is such a visualization for a feature that responds strongly

to coins. The white squares in the top left show that the feature also

responds strongly to the horizontal velocity info when it is white,

corresponding to the agent moving right at full speed.

![](Understanding%20RL%20Vision_files/feature_vis_spatial.png)

 Transparency (revealing the diagonally-striped background) indicates a

weak response, so the left half of the visualization is mostly

transparent because coins never appear in the left half of observations.

Attribution

-----------

 Attribution

 answers questions about the relationships between neurons. It is most

commonly used to see how the input to a network affects a particular

 how we applied this method a hidden layer, and how positive value

function attribution can be thought of as “good news” and negative value

 function attribution can as “bad news”.

### Dimensionality reduction for attribution

 We showed [above](#dataset-examples) that a

dimensionality reduction method known as non-negative matrix

factorization (NMF) could be applied to the channels of activations to

 before, we obtain the NMF directions by sampling a few thousand

observations infrequently from the agent playing the game, computing the

 attributions, flattening the spatial dimensions into the batch

 workaround is to separate out the positive and negative parts of the

attributions and concatenate them along the batch dimension. We could

also have concatenated them along the channel dimension.).

Both methods tend to produce NMF directions that are close to one-hot,

and so can be thought of as picking out the most relevant channels.

However, when reducing to a small number of dimensions, using

attributions usually picks out more salient features, because

 Following , after applying

NMF to attributions, we visualize them by assigning a different color to

 each of the resulting channels. We overlay these visualizations over

 This gives a basic version of our interface, which allows us to see the

 effect of the main features at different spatial positions.

| --- | --- | --- |

|

 |

 |

 |

Legend (hover to isolate)

Buzzsaw

obstacle

Coin

Enemy

moving

left

Agent

or enemy

moving right

 of our CoinRun model, reduced to 4 channels using attribution-based

NMF. The dataset example-based feature visualizations of these

directions reveal more salient features than the visualizations of the

first 4 activation-based NMF directions from the preceding section.

 For the [full version](#interface) of our interface, we

 simply repeat this for an entire trajectory of the agent playing the

game. We also incorporate video controls, a timeline view of compressed

observations , and additional information,

such as model outputs and sampled actions. Together these allow the

trajectory to be easily explored and understood.

### Attribution discussion

* **Sparsity.** Attribution tends to be concentrated in a

very small number of spatial positions and (post-NMF) channels. For

example, in the figure above, the top 10 position–channel pairs account

for more than 80% of the total absolute attribution. This might be

explained by our [earlier](#feature-visualization-discussion)

 hypothesis that the model identifies objects by picking out certain

small configurations of pixels. Because of this sparsity, we smooth out

attribution over nearby spatial positions for the full version of our

interface, so that the amount of visual space taken up can be used to

judge attribution strength. This trades off some spatial precision for

more precision with magnitudes.

* **Unexpected sign.** Value function attribution usually has

 the sign one would expect: positive for coins, negative for enemies,

and so on. However, this is sometimes not the case. For example, in the

figure above, the red channel that detects buzzsaw obstacles has both

positive and negative attribution in two neighboring spatial positions

towards the left. Our best guess is that this phenomenon is a result of

statistical [collinearity](https://en.wikipedia.org/wiki/Multicollinearity),

 caused by certain correlations in the procedural level generation

together with the agent’s behavior. These could be visual, such as

correlations between nearby pixels, or more abstract, such as both coins

 and long walls appearing at the end of every level. As a toy example,

supposing the value function ought to increase by 2% when the end of the

 level becomes visible, the model could either increase the value

function by 1% for coins and 1% for long walls, or by 3% for coins and

−1% for long walls, and the effect would be similar.

* **Outlier frames.** When an unusual event causes the

network to output extreme values, attribution can behave especially

strangely. For example, in the [buzzsaw hallucination](#hallucinations)

 frame, most features have a significant amount of both positive and

negative attribution. We do not have a good explanation for this, but

perhaps features are interacting in more complicated ways than usual.

Moreover, in these cases there is often a significant component of the

attribution lying outside the space spanned by the NMF directions, which

 we display as an additional “residual” feature. This could be because

each frame is weighted equally when computing NMF, so outlier frames

have little influence over the NMF directions.

Questions for further research

------------------------------

### The [diversity hypothesis](#diversity-hypothesis)

2. **Relationship to generalization.** What is the three-way

relationship between diversity, interpretable features and

generalization? Do non-interpretable features indicate that a model will

 fail to generalize in certain ways? Generalization refers implicitly to

 an underlying distribution – how should this distribution be chosen?For

 example, to measure generalization for CoinRun models trained on a

limited number of levels, we used the distribution over all possible

procedurally-generated levels. However, to formalize the sense in which

CoinRun is not diverse in its visual patterns or dynamics rules, one

would need a distribution over levels from a wider class of games.

3. **Caveats.** How are interpretable features affected by

other factors, such as the choice of task or algorithm, and how do these

 interact with diversity? Speculatively, do big enough models obtain

4. **Quantification.** Can we quantitatively predict how much

diversity is needed for interpretable features, perhaps using

generalization metrics? Can we be precise about what is meant by an

“interpretable feature” and a “level of abstraction”?

### Interpretability in the absence of diversity

1. **Pervasiveness of non-diverse features.** Do “non-diverse

features”, by which we mean the hard-to-interpret features that tend to

arise in the absence of diversity, remain when diversity is present? Is

there a connection between these non-diverse features and the

“non-robust features” that have been posited to explain adversarial

examples ?

2. **Coping with non-diverse levels of abstraction.** Are

there levels of abstraction at which even broad distributions like

ImageNet remain non-diverse, and how can we best interpret models at

these levels of abstraction?

 in the absence of diversity, and can it be made to work using

transformation robustness, regularization, data augmentation,

adversarial training, or other techniques? What property of the

optimization leads to the clouds of [extremal colors](#extremal-colors)?

 feature visualization may be untrustworthy by failing to separate the

things causing certain behavior from the things that merely correlate

### Interpretability in the RL framework

1. **Non-visual and abstract features.** What are the best

methods for interpreting models with non-visual inputs? Even vision

models may also have interpretable abstract features, such as

relationships between objects or anticipated events: will any method of

generating examples be enough to understand these, or do we need an

entirely new approach? For models with memory, how can we interpret

their hidden states ?

3. **Modifying training.** In what ways can we train RL models

 to make them more interpretable without a significant performance cost,

 such as by altering architectures or adding auxiliary predictive

losses?

4. **Leveraging the environment.** How can we enrich

interfaces using RL-specific data, such as trajectories of

agent–environment interaction, state distributions, and advantage

estimates? What are the benefits of incorporating user–environment

interaction, such as for exploring counterfactuals?

### What we would like to see from further research and why

 We are motivated to study interpretability for RL for two reasons.

* **To be able to interpret RL models.** RL can be applied to

 an enormous variety of tasks, and seems likely to be a part of

increasingly influential AI systems. It is therefore important to be

able to scrutinize RL models and to understand how they might fail. This

 may also benefit RL research through an improved understanding of the

pitfalls of different algorithms and environments.

* **As a testbed for interpretability techniques.** RL models

 pose a number of distinctive challenges for interpretability

techniques. In particular, environments like CoinRun straddle the

boundary between memorization and generalization, making them useful for

 studying the [diversity hypothesis](#diversity-hypothesis) and related ideas.

 We think that large neural networks are currently the most likely

type of model to be used in highly capable and influential AI systems in

 the future. Contrary to the traditional perception of neural networks

as black boxes, we think that there is a fighting chance that we will be

 able to clearly and thoroughly understand the behavior even of very

large networks. We are therefore most excited by neural network

interpretability research that scores highly according to the following

criteria.

* **Scalability.** The takeaways of the research should have

some chance of scaling to harder problems and larger networks. If the

techniques themselves do not scale, they should at least reveal some

relevant insight that might.

* **Trustworthiness.** Explanations should be faithful to the

 model. Even if they do not tell the full story, they should at least

not be biased in some fatal way (such as by using an approval-based

objective that leads to bad explanations that sound good, or by

depending on another model that badly distorts information).

* **Exhaustiveness.** This may turn out to be impossible at

scale, but we should strive for techniques that explain every essential

feature of our models. If there are theoretical limits to

exhaustiveness, we should try to understand these.

* **Low cost.** Our techniques should not be significantly

more computationally expensive than training the model. We hope that we

will not need to train models differently for them to be interpretable,

but if we do, we should try to minimize both the computational expense

and any performance cost, so that interpretable models are not

disincentivized from being used in practice.

 Our proposed questions reflect this perspective. One of the

reasons we emphasize diversity relates to exhaustiveness. If

“non-diverse features” remain when diversity is present, then our

current techniques are not exhaustive and could end up missing important

 features of more capable models. Developing tools to understand

non-diverse features may shed light on whether this is likely to be a

problem.

 We think there may be significant mileage in simply applying

existing interpretability techniques, with attention to detail, to more

models. Indeed, this was the mindset with which we initially approached

this project. If the diversity hypothesis is correct, then this may

become easier as we train our models to perform more complex tasks. Like

 early biologists encountering a new species, there may be a lot we can

glean from taking a magnifying glass to the creatures in front of us.

Supplementary material

----------------------

* **Model weights.** The weights of our model are available

for download, along with those of a number of other models, including

the models trained on different numbers of levels, the edited models,

Quantifying generalization in reinforcement learning

Deep inside convolutional networks: Visualising image classification models and saliency maps

Striving for simplicity: The all convolutional net

Grad-CAM: Visual explanations from deep networks via gradient-based localization

Interpretable explanations of black boxes by meaningful perturbation

PatternNet and PatternLRP--Improving the interpretability of neural networks

The (un)reliability of saliency methods

Axiomatic attribution for deep networks

The Building Blocks of Interpretability

Leveraging Procedural Generation to Benchmark Reinforcement Learning

Proximal policy optimization algorithms

High-dimensional continuous control using generalized advantage estimation

Thread: Circuits

General Video Game AI: A multi-track framework for evaluating agents, games and content generation algorithms

Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning

Observational Overfitting in Reinforcement Learning

Feature Visualization

Visualizing higher-layer features of a deep network

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Imagenet: A large-scale hierarchical image database

Going deeper with convolutions

An Atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents

Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents

Caricatures

Towards deep learning models resistant to adversarial attacks

Intriguing properties of neural networks

Visualizing and understanding Atari agents

Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution

Video Interface: Assuming Multiple Perspectives on a Video Exposes Hidden Structure

Reconciling modern machine-learning practice and the classical bias--variance trade-off

Adversarial examples are not bugs, they are features

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'

Human-level performance in 3D multiplayer games with population-based reinforcement learning

Solving Rubik's Cube with a Robot Hand

Dota 2 with Large Scale Deep Reinforcement Learning

Does Attribution Make Sense?

IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures


Benjamin Sanchez-Lengeling
Emily Reif
Adam Pearce
Alexander B. Wiltschko

Abstract: This article is one of two Distill publications about graph neural networks. Take a look at Understanding Convolutions on Graphs to understand how convolutions over images generalize naturally to convolutions over graphs.

Graphs are all around us; real world objects are often defined in

terms of their connections to other things. A set of objects, and the

connections between them, are naturally expressed as a *graph*.

Researchers have developed neural networks that operate on graph data

(called graph neural networks, or GNNs) for over a decade.

 Recent developments have increased their capabilities and expressive

power. We are starting to see practical applications in areas such as

This article explores and explains modern graph neural networks. We

divide this work into four parts. First, we look at what kind of data is

 most naturally phrased as a graph, and some common examples. Second, we

 explore what makes graphs different from other types of data, and some

of the specialized choices we have to make when using graphs. Third, we

build a modern GNN, walking through each of the parts of the model,

starting with historic modeling innovations in the field. We move

gradually from a bare-bones implementation to a state-of-the-art GNN

model. Fourth and finally, we provide a GNN playground where you can

play around with a real-word task and dataset to build a stronger

intuition of how each component of a GNN model contributes to the

predictions it makes.

Three types of attributes we might find in a graph, hover over to

highlight each attribute. Other types of graphs and attributes are

 They can also be undirected, where there is no notion of source or

destination nodes, and information flows both directions. Note that

having a single undirected edge is equivalent to having one directed

Graphs are very flexible data structures, and if this seems abstract

now, we will make it concrete with examples in the next section.

Graphs and where to find them

-----------------------------

You’re probably already familiar with some types of graph data, such

as social networks. However, graphs are an extremely powerful and

general representation of data, we will show two types of data that you

might not think could be modeled as graphs: images and text. Although

counterintuitive, one can learn more about the symmetries and structure

of images and text by viewing them as graphs,, and build an intuition

that will help understand other less grid-like graph data, which we will

 discuss later.

### Images as graphs

We typically think of images as rectangular grids with image

channels, representing them as arrays (e.g., 244x244x3 floats). Another

way to think of images is as graphs with regular structure, where each

pixel represents a node and is connected via an edge to adjacent pixels.

 Each non-border pixel has exactly 8 neighbors, and the information

stored at each node is a 3-dimensional vector representing the RGB value

 of the pixel.

 with an entry if two nodes share an edge. Note that each of these three

 representations below are different views of the same piece of data.

### Text as graphs

We can digitize text by associating indices to each character, word,

or token, and representing text as a sequence of these indices. This

creates a simple directed graph, where each character or index is a node

 and is connected via an edge to the node that follows it.

→→→→→→→→→→→→→→→→→→→→GraphsareallaroundusGraphsareallaroundus

Edit the text above to see how the graph representation changes.

Of course, in practice, this is not usually how text and images are

encoded: these graph representations are redundant since all images and

all text will have very regular structures. For instance, images have a

banded structure in their adjacency matrix because all nodes (pixels)

are connected in a grid. The adjacency matrix for text is just a

diagonal line, because each word only connects to the prior word, and to

 the next one.

This representation (a sequence of character tokens) refers to the way

text is often represented in RNNs; other models, such as Transformers,

can be considered to view text as a fully connected graph where we learn

### Graph-valued data in the wild

Graphs are a useful tool to describe data you might already be

familiar with. Let’s move on to data which is more heterogeneously

structured. In these examples, the number of neighbors to each node is

variable (as opposed to the fixed neighborhood size of images and text).

 This data is hard to phrase in any other way besides a graph.

**Molecules as graphs.** Molecules are the building

blocks of matter, and are built of atoms and electrons in 3D space. All

particles are interacting, but when a pair of atoms are stuck in a

stable distance from each other, we say they share a covalent bond.

Different pairs of atoms and bonds have different distances (e.g.

single-bonds, double-bonds). It’s a very convenient and common

abstraction to describe this 3D object as a graph, where nodes are atoms

(Left) 3d representation of the Citronellal molecule (Center) Adjacency

matrix of the bonds in the molecule (Right) Graph representation of the

molecule.

(Left) 3d representation of the Caffeine molecule (Center) Adjacency

matrix of the bonds in the molecule (Right) Graph representation of the

molecule.

**Social networks as graphs.** Social networks are tools

 to study patterns in collective behaviour of people, institutions and

organizations. We can build a graph representing groups of people by

modelling individuals as nodes, and their relationships as edges.

(Left) Image of a scene from the play “Othello”. (Center) Adjacency

matrix of the interaction between characters in the play. (Right) Graph

representation of these interactions.

(Left) Image of karate tournament. (Center) Adjacency matrix of the

interaction between people in a karate club. (Right) Graph

representation of these interactions.

**Citation networks as graphs.** Scientists routinely

cite other scientists’ work when publishing papers. We can visualize

these networks of citations as a graph, where each paper is a node, and

each *directed* edge is a citation between one paper and another.

 Additionally, we can add information about each paper into each node,

such as a word embedding of the abstract. (see ,  , ).

**Other examples.** In computer vision, we sometimes

want to tag objects in visual scenes. We can then build graphs by

 can also be phrased as graphs, where the variables are nodes, and edges

 are operations that have these variables as input and output. You might

 see the term “dataflow graph” used in some of these contexts.

The structure of real-world graphs can vary greatly between different

 types of data — some graphs have many nodes with few connections

between them, or vice versa. Graph datasets can vary widely (both within

 a given dataset, and between datasets) in terms of the number of nodes,

 edges, and the connectivity of nodes.

 Edges per node (degree)

 Dataset

 Domain

 graphs

 nodes

 edges

 min

 mean

 max

 karate club

 Social network

 1

 34

 78

 4.5

 17

 qm9

 Small molecules

 134k

 ≤ 9

 ≤26

 1

 2

 5

 Cora

 Citation network

 1

 23,166

 91,500

 1

 7.8

 379

 Wikipedia links, English

 Knowledge graph

 1

 12M

 378M

 62.24

 1M

Summary statistics on graphs found in the real world. Numbers are

dependent on featurization decisions. More useful statistics and graphs

can be found in KONECT

What types of problems have graph structured data?

--------------------------------------------------

We have described some examples of graphs in the wild, but what tasks

 do we want to perform on this data? There are three general types of

prediction tasks on graphs: graph-level, node-level, and edge-level.

In a graph-level task, we predict a single property for a whole

graph. For a node-level task, we predict some property for each node in a

 graph. For an edge-level task, we want to predict the property or

presence of edges in a graph.

For the three levels of prediction problems described above

(graph-level, node-level, and edge-level), we will show that all of the

following problems can be solved with a single model class, the GNN. But

 first, let’s take a tour through the three classes of graph prediction

problems in more detail, and provide concrete examples of each.

### Graph-level task

In a graph-level task, our goal is to predict the property of an

entire graph. For example, for a molecule represented as a graph, we

might want to predict what the molecule smells like, or whether it will

bind to a receptor implicated in a disease.

This is analogous to image classification problems with MNIST and

CIFAR, where we want to associate a label to an entire image. With text,

 a similar problem is sentiment analysis where we want to identify the

mood or emotion of an entire sentence at once.

### Node-level task

A classic example of a node-level prediction problem is Zach’s karate club.

 The dataset is a single social network graph made up of individuals

that have sworn allegiance to one of two karate clubs after a political

rift. As the story goes, a feud between Mr. Hi (Instructor) and John H

(Administrator) creates a schism in the karate club. The nodes represent

 individual karate practitioners, and the edges represent interactions

between these members outside of karate. The prediction problem is to

classify whether a given member becomes loyal to either Mr. Hi or John

H, after the feud. In this case, distance between a node to either the

Instructor or Administrator is highly correlated to this label.

On the left we have the initial conditions of the problem, on the right

we have a possible solution, where each node has been classified based

on the alliance. The dataset can be used in other graph problems like

unsupervised learning.

 where we are trying to label the role of each pixel in an image. With

text, a similar task would be predicting the parts-of-speech of each

word in a sentence (e.g. noun, verb, adverb, etc).

### Edge-level task

The remaining prediction problem in graphs is *edge prediction*.

One example of edge-level inference is in image scene understanding.

Beyond identifying objects in an image, deep learning models can be used

 to predict the relationship between them. We can phrase this as an

edge-level classification: given nodes that represent the objects in the

 image, we wish to predict which of these nodes share an edge or what

the value of that edge is. If we wish to discover connections between

entities, we could consider the graph fully connected and based on their

 predicted value prune edges to arrive at a sparse graph.

![](A%20Gentle%20Introduction%20to%20Graph%20Neural%20Networks_files/merged.png)

In (b), above, the original image (a) has been segmented into five

entities: each of the fighters, the referee, the audience and the mat.

(C) shows the relationships between these entities.

On the left we have an initial graph built from the previous visual

scene. On the right is a possible edge-labeling of this graph when some

connections were pruned based on the model’s output.

The challenges of using graphs in machine learning

--------------------------------------------------

So, how do we go about solving these different graph tasks with

neural networks? The first step is to think about how we will represent

graphs to be compatible with neural networks.

Machine learning models typically take rectangular or grid-like

arrays as input. So, it’s not immediately intuitive how to represent

them in a format that is compatible with deep learning. Graphs have up

to four types of information that we will potentially want to use to

make predictions: nodes, edges, global-context and connectivity. The

first three are relatively straightforward: for example, with nodes we

However, representing a graph’s connectivity is more complicated.

Perhaps the most obvious choice would be to use an adjacency matrix,

since this is easily tensorisable. However, this representation has a

few drawbacks. From the [example dataset table](#table), we

see the number of nodes in a graph can be on the order of millions, and

the number of edges per node can be highly variable. Often, this leads

to very sparse adjacency matrices, which are space-inefficient.

Another problem is that there are many adjacency matrices that can

encode the same connectivity, and there is no guarantee that these

different matrices would produce the same result in a deep neural

network (that is to say, they are not permutation invariant).

Learning permutation invariant operations is an area of recent research.

 from before can be described equivalently with these two adjacency

matrices. It can also be described with every other possible

permutation of the nodes.

Two adjacency matrices representing the same graph.

The example below shows every adjacency matrix that can describe this

 small graph of 4 nodes. This is already a significant number of

adjacency matrices–for larger examples like Othello, the number is

untenable.

All of these adjacency matrices represent the same graph. Click on an

edge to remove it on a “virtual edge” to add it and the matrices will

update accordingly.

One elegant and memory-efficient way of representing sparse matrices

 as a tuple (i,j) in the k-th entry of an adjacency list. Since we

expect the number of edges to be much lower than the number of entries

Nodes

[01100111

Edges

[2100000001000000010000002110

Adjacency List

Global

0

Hover and click on the edges, nodes, and global graph marker to view and

 change attribute representations. On one side we have a small graph and

 on the other the information of the graph in a tensor representation.

It should be noted that the figure uses scalar values per

node/edge/global, but most practical tensor representations have vectors

Graph Neural Networks

---------------------

Now that the graph’s description is in a matrix format that is

permutation invariant, we will describe using graph neural networks

(GNNs) to solve graph prediction tasks. **A GNN is an optimizable

transformation on all attributes of the graph (nodes, edges,

global-context) that preserves graph symmetries (permutation

 GNNs adopt a “graph-in, graph-out” architecture meaning that these

model types accept a graph as input, with information loaded into its

nodes, edges and global-context, and progressively transform these

embeddings, without changing the connectivity of the input graph.

### The simplest GNN

 (with vectors instead of scalars), we are now ready to build a GNN. We

will start with the simplest GNN architecture, one where we learn new

embeddings for all graph attributes (nodes, edges, global), but where we

 do not yet use the connectivity of the graph.

For simplicity, the previous diagrams used scalars to represent graph

attributes; in practice feature vectors, or embeddings, are much more

useful.

This GNN uses a separate multilayer perceptron (MLP) (or your

favorite differentiable model) on each component of a graph; we call

this a GNN layer. For each node vector, we apply the MLP and get back a

learned node-vector. We do the same for each edge, learning a per-edge

embedding, and also for the global-context vector, learning a single

embedding for the entire graph.

A single layer of a simple GNN. A graph is the input, and each component

 (V,E,U) gets updated by a MLP to produce a new graph. Each function

subscript indicates a separate function for a different graph attribute

at the n-th layer of a GNN model.

Because a GNN does not update the connectivity of the input graph, we

 can describe the output graph of a GNN with the same adjacency list and

 the same number of feature vectors as the input graph. But, the output

graph has updated embeddings, since the GNN has updated each of the

node, edge and global-context representations.

### GNN Predictions by Pooling Information

We will consider the case of binary classification, but this

framework can easily be extended to the multi-class or regression case.

If the task is to make binary predictions on nodes, and the graph

already contains node information, the approach is straightforward — for

 each node embedding, apply a linear classifier.

We could imagine a social network, where we wish to anonymize user data

(nodes) by not using them, and only using relational data (edges). One

 subsection. In the Karate club example, this would be just using the

number of meetings between people to determine the alliance to Mr. Hi or

 John H.

However, it is not always so simple. For instance, you might have

information in the graph stored in edges, but no information in nodes,

but still need to make predictions on nodes. We need a way to collect

information from edges and give them to nodes for prediction. We can do

this by *pooling*. Pooling proceeds in two steps:

2. The gathered embeddings are then *aggregated*, usually via a sum operation.

+Aggregate information from adjacent edges

Hover over a node (black node) to visualize which edges are gathered and

 aggregated to produce an embedding for that target node.

So If we only have edge-level features, and are trying to predict

binary node information, we can use pooling to route (or pass)

information to where it needs to go. The model looks like this.

 sub section. Nodes can be recognized as image entities, and we are

trying to predict if the entities share a relationship (binary edges).

If we only have node-level features, and need to predict a binary

global property, we need to gather all available node information

This is a common scenario for predicting molecular properties. For

example, we have atomic information, connectivity and we would like to

know the toxicity of a molecule (toxic/not toxic), or if it has a

particular odor (rose/not rose).

In our examples, the classification model *𝑐cc*

 can easily be replaced with any differentiable model, or adapted to

multi-class classification using a generalized linear model.

An end-to-end prediction task with a GNN model.

Now we’ve demonstrated that we can build a simple GNN model, and make

 binary predictions by routing information between different parts of

the graph. This pooling technique will serve as a building block for

constructing more sophisticated GNN models. If we have new graph

attributes, we just have to define how to pass information from one

attribute to another.

Note that in this simplest GNN formulation, we’re not using the

connectivity of the graph at all inside the GNN layer. Each node is

processed independently, as is each edge, as well as the global context.

 We only use connectivity when pooling information for prediction.

### Passing messages between parts of the graph

We could make more sophisticated predictions by using pooling within

the GNN layer, in order to make our learned embeddings aware of graph

Message passing works in three steps:

2. Aggregate all messages via an aggregate function (like sum).

These steps are key for leveraging the connectivity of graphs. We

will build more elaborate variants of message passing in GNN layers that

 yield GNN models of increasing expressiveness and power.

This is reminiscent of standard convolution: in essence, message

passing and convolution are operations to aggregate and process the

information of an element’s neighbors in order to update the element’s

value. In graphs, the element is a node, and in images, the element is a

 pixel. However, the number of neighboring nodes in a graph can be

variable, unlike in an image where each pixel has a set number of

neighboring elements.

By stacking message passing GNN layers together, a node can

eventually incorporate information from across the entire graph: after

three layers, a node has information about the nodes three steps away

from it.

Schematic for a GCN architecture, which updates node representations of a

 graph by pooling neighboring nodes at a distance of one degree.

### Learning edge representations

Our dataset does not always contain all types of information (node,

edge, and global context).

When we want to make a prediction on nodes, but our dataset only has

edge information, we showed above how to use pooling to route

information from edges to nodes, but only at the final prediction step

of the model. We can share information between nodes and edges within

the GNN layer using message passing.

We can incorporate the information from neighboring edges in the same

 way we used neighboring node information earlier, by first pooling the

edge information, transforming it with an update function, and storing

it.

However, the node and edge information stored in a graph are not

necessarily the same size or shape, so it is not immediately clear how

to combine them. One way is to learn a linear mapping from the space of

edges to the space of nodes, and vice versa. Alternatively, one may

concatenate them together before the update function.

Architecture schematic for Message Passing layer. The first step

“prepares” a message composed of information from an edge and it’s

connected nodes and then “passes” the message to the node.

Which graph attributes we update and in which order we update them is

 one design decision when constructing GNNs. We could choose whether to

update node embeddings before edge embeddings, or the other way around.

This is an open area of research with a variety of solutions– for

example we could update in a ‘weave’ fashion

 where we have four updated representations that get combined into new

node and edge representations: node to node (linear), edge to edge

(linear), node to edge (edge layer), edge to node (node layer).

### Adding global representations

There is one flaw with the networks we have described so far: nodes

that are far away from each other in the graph may never be able to

efficiently transfer information to one another, even if we apply

message passing several times. For one node, If we have k-layers,

information will propagate at most k-steps away. This can be a problem

for situations where the prediction task depends on nodes, or groups of

nodes, that are far apart. One solution would be to have all nodes be

able to pass information to each other.

Unfortunately for large graphs, this quickly becomes computationally

expensive (although this approach, called ‘virtual edges’, has been used

 for small graphs such as molecules).

 or context vector. This global context vector is connected to all other

 nodes and edges in the network, and can act as a bridge between them to

 pass information, building up a representation for the graph as a

whole. This creates a richer and more complex representation of the

graph than could have otherwise been learned.

Schematic of a Graph Nets architecture leveraging global representations.

In this view all graph attributes have learned representations, so we

 can leverage them during pooling by conditioning the information of our

 attribute of interest with respect to the rest. For example, for one

node we can consider information from neighboring nodes, connected edges

 and the global information. To condition the new node embedding on all

these possible sources of information, we can simply concatenate them.

Additionally we may also map them to the same space via a linear map and

Schematic for conditioning the information of one node based

 on three other embeddings (adjacent nodes, adjacent edges, global).

This step corresponds to the node operations in the Graph Nets Layer.

GNN playground

--------------

We’ve described a wide range of GNN components here, but how do they

actually differ in practice? This GNN playground allows you to see how

these different components and architectures contribute to a GNN’s

ability to learn a real task.

 which is composed of molecules with associated odor percepts (labels).

Predicting the relation of a molecular structure (graph) to its smell is

 a 100 year-old problem straddling chemistry, physics, neuroscience, and

 machine learning.

To simplify the problem, we consider only a single binary label per

molecule, classifying if a molecular graph smells “pungent” or not, as

labeled by a professional perfumer. We say a molecule has a “pungent”

scent if it has a strong, striking smell. For example, garlic and

We represent each molecule as a graph, where atoms are nodes

containing a one-hot encoding for its atomic identity (Carbon, Nitrogen,

 Oxygen, Fluorine) and bonds are edges containing a one-hot encoding its

 bond type (single, double, triple or aromatic).

Our general modeling template for this problem will be built up using

 sequential GNN layers, followed by a linear model with a sigmoid

activation for classification. The design space for our GNN has many

levers that can customize the model:

1. The number of GNN layers, also called the *depth*.

2. The dimensionality of each attribute when updated. The update

function is a 1-layer MLP with a relu activation function and a layer

norm for normalization of activations.

3. The aggregation function used in pooling: max, mean or sum.

4. The graph attributes that get updated, or styles of message

passing: nodes, edges and global representation. We control these via

boolean toggles (on or off). A baseline model would be a

graph-independent GNN (all message-passing off) which aggregates all

data at the end into a single global attribute. Toggling on all

message-passing functions yields a GraphNets architecture.

To better understand how a GNN is learning a task-optimized

representation of a graph, we also look at the penultimate layer

activations of the GNN. These ‘graph embeddings’ are the outputs of the

GNN model right before prediction. Since we are using a generalized

linear model for prediction, a linear mapping is enough to allow us to

see how we are learning representations around the decision boundary.

Since these are high dimensional vectors, we reduce them to 2D via

principal component analysis (PCA).

A perfect model would visibility separate labeled data, but since we are

 reducing dimensionality and also have imperfect models, this boundary

might be harder to see.

Play around with different model architectures to build your

intuition. For example, see if you can edit the molecule on the left to

make the model prediction increase. Do the same edits have the same

effects for different model architectures?

 Task

For this **graph level prediction task**,

 each graph is a molecule. The task is to predict whether or not a

molecule will smell pungent. In the scatter plot, each point represents a

 graph.

 Model Options

Depth

1 layer

2 layers

3 layers

4 layers

 Aggregation function

Mean

Sum

Max

Node embedding size

25

50

100

Edge embedding size

5

10

20

Global embedding size

25

50

100

 Model AUC

0.75

Reset

Model Prediction

14% pungent

Ground Truth

not pungent

Carbon

Nitrogen

Oxygen

Sulphur

 –

    Single Bond

 =

    Double Bond

 ≡

   Triple Bond

 ≡ ≡

 Aromatic Bond

 Model Prediction

 Ground Truth

Pungent

  0%  100%

epoch: 50

Edit the molecule to see how the prediction changes, or

change the model params to load a different model. Select a different

molecule in the scatter plot.

### Some empirical GNN design lessons

When exploring the architecture choices above, you might have found

some models have better performance than others. Are there some clear

GNN design choices that will give us better performance? For example, do

 deeper GNN models perform better than shallower ones? or is there a

clear choice between aggregation functions? The answers are going to

With the following interactive figure, we explore the space of GNN

architectures and the performance of this task across a few major design

 choices: Style of message passing, the dimensionality of embeddings,

number of layers, and aggregation operation type.

Each point in the scatter plot represents a model: the x axis is the

number of trainable variables, and the y axis is the performance. Hover

over a point to see the GNN architecture parameters.

var spec = "BasicArchitectures.json";

}).catch(console.error);

Scatterplot of each model’s performance vs its number of

trainable variables. Hover over a point to see the GNN architecture

parameters.

The first thing to notice is that, surprisingly, a higher number of

parameters does correlate with higher performance. GNNs are a very

parameter-efficient model type: for even a small number of parameters

(3k) we can already find models with high performance.

Next, we can look at the distributions of performance aggregated

based on the dimensionality of the learned representations for different

 graph attributes.

var spec = "ArchitectureNDim.json";

}).catch(console.error);

We can notice that models with higher dimensionality tend to have

better mean and lower bound performance but the same trend is not found

for the maximum. Some of the top-performing models can be found for

smaller dimensions. Since higher dimensionality is going to also involve

 a higher number of parameters, these observations go in hand with the

previous figure.

Next we can see the breakdown of performance based on the number of GNN layers.

var spec = "ArchitectureNLayers.json";

}).catch(console.error);

 Chart of number of layers vs model performance, and

scatterplot of model performance vs number of parameters. Each point is

colored by the number of layers. Hover over a point to see the GNN

architecture parameters.

The box plot shows a similar trend, while the mean performance tends

to increase with the number of layers, the best performing models do not

 have three or four layers, but two. Furthermore, the lower bound for

performance decreases with four layers. This effect has been observed

before, GNN with a higher number of layers will broadcast information at

 a higher distance and can risk having their node representations

‘diluted’ from many successive iterations .

Does our dataset have a preferred aggregation operation? Our

following figure breaks down performance in terms of aggregation type.

var spec = "ArchitectureAggregation.json";

}).catch(console.error);

Chart of aggregation type vs model performance, and

scatterplot of model performance vs number of parameters. Each point is

colored by aggregation type. Hover over a point to see the GNN

architecture parameters.

Overall it appears that sum has a very slight improvement on the mean

 performance, but max or mean can give equally good models. This is

The previous explorations have given mixed messages. We can find mean

 trends where more complexity gives better performance but we can find

clear counterexamples where models with fewer parameters, number of

layers, or dimensionality perform better. One trend that is much clearer

 is about the number of attributes that are passing information to each

other.

Here we break down performance based on the style of message passing.

 On both extremes, we consider models that do not communicate between

graph entities (“none”) and models that have messaging passed between

nodes, edges, and globals.

var spec = "ArchitectureMessagePassing.json";

}).catch(console.error);

Chart of message passing vs model performance, and

scatterplot of model performance vs number of parameters. Each point is

colored by message passing. Hover over a point to see the GNN

architecture parameters

Overall we see that the more graph attributes are communicating, the

better the performance of the average model. Our task is centered on

global representations, so explicitly learning this attribute also tends

 to improve performance. Our node representations also seem to be more

useful than edge representations, which makes sense since more

information is loaded in these attributes.

There are many directions you could go from here to get better

performance. We wish two highlight two general directions, one related

to more sophisticated graph algorithms and another towards the graph

itself.

Up until now, our GNN is based on a neighborhood-based pooling

operation. There are some graph concepts that are harder to express in

this way, for example a linear graph path (a connected chain of nodes).

Designing new mechanisms in which graph information can be extracted,

executed and propagated in a GNN is one current research area , , , .

One of the frontiers of GNN research is not making new models and

architectures, but “how to construct graphs”, to be more precise,

imbuing graphs with additional structure or relations that can be

leveraged. As we loosely saw, the more graph attributes are

communicating the more we tend to have better models. In this particular

 case, we could consider making molecular graphs more feature rich, by

adding additional spatial relationships between nodes, adding edges that

 are not bonds, or explicit learnable relationships between subgraphs.

See more in [Other types of graphs](#Other-types-of-graphs ).

Into the Weeds

--------------

While we only described graphs with vectorized information for each

attribute, graph structures are more flexible and can accommodate other

types of information. Fortunately, the message passing framework is

flexible enough that often adapting GNNs to more complex graph

structures is about defining how information is passed and updated by

new graph attributes.

For example, we can consider multi-edge graphs or *multigraphs*,

 where a pair of nodes can share multiple types of edges, this happens

when we want to model the interactions between nodes differently based

on their type. For example with a social network, we can specify edge

types based on the type of relationships (acquaintance, friend, family).

 A GNN can be adapted by having different types of message passing steps

 for each edge type.

We can also consider nested graphs, where for example a node represents a

 graph, also called a hypernode graph.

 Nested graphs are useful for representing hierarchical information. For

 example, we can consider a network of molecules, where a node

represents a molecule and an edge is shared between two molecules if we

have a way (reaction) of transforming one to the other  .

In this case, we can learn on a nested graph by having a GNN that learns

 representations at the molecule level and another at the reaction

network level, and alternate between them during training.

Another type of graph is a hypergraph,

 where an edge can be connected to multiple nodes instead of just two.

For a given graph, we can build a hypergraph by identifying communities

of nodes and assigning a hyper-edge that is connected to all nodes in a

community.

Schematic of more complex graphs. On the left we have an

example of a multigraph with three edge types, including a directed

edge. On the right we have a three-level hierarchical graph, the

intermediate level nodes are hypernodes.

### Sampling Graphs and Batching in GNNs

A common practice for training neural networks is to update network

parameters with gradients calculated on randomized constant size (batch

size) subsets of the training data (mini-batches). This practice

presents a challenge for graphs due to the variability in the number of

nodes and edges adjacent to each other, meaning that we cannot have a

constant batch size. The main idea for batching with graphs is to create

 subgraphs that preserve essential properties of the larger graph. This

graph sampling operation is highly dependent on context and involves

sub-selecting nodes and edges from a graph. These operations might make

sense in some contexts (citation networks) and in others, these might be

 too strong of an operation (molecules, where a subgraph simply

represents a new, smaller molecule). How to sample a graph is an open

research question.

If we care about preserving structure at a neighborhood level, one way

 Each neighborhood can be considered an individual graph and a GNN can

be trained on batches of these subgraphs. The loss can be masked to only

 consider the node-set since all neighboring nodes would have incomplete

 neighborhoods.

A more efficient strategy might be to first randomly sample a single

node, expand its neighborhood to distance k, and then pick the other

node within the expanded set. These operations can be terminated once a

certain amount of nodes, edges, or subgraphs are constructed.

If the context allows, we can build constant size neighborhoods by

picking an initial node-set and then sub-sampling a constant number of

nodes (e.g randomly, or via a random walk or Metropolis algorithm).

Four different ways of sampling the same graph. Choice of

sampling strategy depends highly on context since they will generate

different distributions of graph statistics (# nodes, #edges, etc.). For

 highly connected graphs, edges can be also subsampled.

Sampling a graph is particularly relevant when a graph is large

enough that it cannot be fit in memory. Inspiring new architectures and

### Inductive biases

When building a model to solve a problem on a specific kind of data,

we want to specialize our models to leverage the characteristics of that

 data. When this is done successfully, we often see better predictive

performance, lower training time, fewer parameters and better

generalization.

When labeling on images, for example, we want to take advantage of

the fact that a dog is still a dog whether it is in the top-left or

bottom-right corner of an image. Thus, most image models use

convolutions, which are translation invariant. For text, the order of

the tokens is highly important, so recurrent neural networks process

data sequentially. Further, the presence of one token (e.g. the word

‘not’) can affect the meaning of the rest of a sentence, and so we need

components that can ‘attend’ to other parts of the text, which

transformer models like BERT and GPT-3 can do. These are some examples

of inductive biases, where we are identifying symmetries or regularities

 in the data and adding modelling components that take advantage of

these properties.

In the case of graphs, we care about how each graph component (edge,

node, global) is related to each other so we seek models that have a

relational inductive bias. A

model should preserve explicit relationships between entities (adjacency

 matrix) and preserve graph symmetries (permutation invariance). We

expect problems where the interaction between entities is important will

 benefit from a graph structure. Concretely, this means designing

transformation on sets: the order of operation on nodes or edges should

not matter and the operation should work on a variable number of

inputs.

### Comparing aggregation operations

Pooling information from neighboring nodes and edges is a critical

step in any reasonably powerful GNN architecture. Because each node has a

 variable number of neighbors, and because we want a differentiable

method of aggregating this information, we want to use a smooth

aggregation operation that is invariant to node ordering and the number

of nodes provided.

 A desirable property of an aggregation operation is that similar inputs

 provide similar aggregated outputs, and vice-versa. Some very simple

candidate permutation-invariant operations are sum, mean, and max.

Summary statistics like variance also work. All of these take a variable

 number of inputs, and provide an output that is the same, no matter the

 input ordering. Let’s explore the difference between these operations.

24?Max4Mean3Sum644?44833?Max3Mean3Sum63-1?333

There is no operation that is uniformly the best choice. The mean

operation can be useful when nodes have a highly-variable number of

neighbors or you need a normalized view of the features of a local

neighborhood. The max operation can be useful when you want to highlight

 single salient features in local neighborhoods. Sum provides a balance

between these two, by providing a snapshot of the local distribution of

features, but because it is not normalized, can also highlight outliers.

 In practice, sum is commonly used.

 take into account several aggregation operations by concatenating them

and adding a scaling function that depends on the degree of connectivity

 of the entity to aggregate. Meanwhile, domain specific aggregation

operations can also be designed. One example lies with the “Tetrahedral

Chirality” aggregation operators .

### GCN as subgraph function approximators

Another way to see GCN (and MPNN) of k-layers with a 1-degree

neighbor lookup is as a neural network that operates on learned

embeddings of subgraphs of size k.

When focusing on one node, after k-layers, the updated node

representation has a limited viewpoint of all neighbors up to

k-distance, essentially a subgraph representation. Same is true for edge

 representations.

So a GCN is collecting all possible subgraphs of size k and learning

vector representations from the vantage point of one node or edge. The

number of possible subgraphs can grow combinatorially, so enumerating

these subgraphs from the beginning vs building them dynamically as in a

GCN, might be prohibitive.

### Edges and the Graph Dual

One thing to note is that edge predictions and node predictions,

while seemingly different, often reduce to the same problem: an edge

To obtain 𝐺GG’s

 dual, we can convert nodes to edges (and edges to nodes). A graph and

its dual contain the same information, just expressed in a different

way. Sometimes this property makes solving problems easier in one

representation than another, like frequencies in Fourier space. In

We’ve talked a lot about graph convolutions and message passing, and

of course, this raises the question of how do we implement these

operations in practice? For this section, we explore some of the

properties of matrix multiplication, message passing, and its connection

 to traversing a graph.

 It should be noted that this message passing is not updating the

representation of the node features, just pooling neighboring node

 is zero. As long as we have an operation to gather values based on an

index, we should be able to just retrieve positive entries.

Additionally, this matrix multiply-free approach frees us from using

summation as an aggregation operation.

We can imagine that applying this operation multiple times allows us

to propagate information at greater distances. In this sense, matrix

multiplication is a form of traversing over a graph. This relationship

There are deeper connections on how we can view matrices as graphs to explore .

### Graph Attention Networks

 For example, when we consider the sum-aggregation of a node and its

1-degree neighboring nodes we could also consider using a weighted

sum.The challenge then is to associate weights in a permutation

invariant fashion. One approach is to consider a scalar scoring function

 In this case, the scoring function can be interpreted as a function

that measures how relevant a neighboring node is in relation to the

center node. Weights can be normalized, for example with a softmax

function to focus most of the weight on a neighbor most relevant for a

node in relation to a task. This concept is the basis of Graph Attention

 Networks (GAT)  and Set Transformers.

 Permutation invariance is preserved, because scoring works on pairs of

nodes. A common scoring function is the inner product and nodes are

often transformed before scoring into query and key vectors via a linear

 map to increase the expressivity of the scoring mechanism. Additionally

 for interpretability, the scoring weights can be used as a measure of

the importance of an edge in relation to a task.

Schematic of attention over one node with respect to it’s

adjacent nodes. For each edge an interaction score is computed,

normalized and used to weight node embeddings.

Additionally, transformers can be viewed as GNNs with an attention mechanism .

 Under this view, the transformer models several elements (i.g.

character tokens) as nodes in a fully connected graph and the attention

mechanism is assigning edge embeddings to each node-pair which are used

to compute attention weights. The difference lies in the assumed pattern

 of connectivity between entities, a GNN is assuming a sparse pattern

and the Transformer is modelling all connections.

### Graph explanations and attributions

When deploying GNN in the wild we might care about model

interpretability for building credibility, debugging or scientific

discovery. The graph concepts that we care to explain vary from context

to context. For example, with molecules we might care about the presence

 or absence of particular subgraphs,

 while in a citation network we might care about the degree of

connectedness of an article. Due to the variety of graph concepts, there

 assign ranked importance values to parts of a graph that are relevant

for a task. Because realistic and challenging graph problems can be

generated synthetically, GNNs can serve as a rigorous and repeatable

testbed for evaluating attribution techniques .

Schematic of some explanability techniques on graphs.

Attributions assign ranked values to graph attributes. Rankings can be

used as a basis to extract connected subgraphs that might be relevant to

 a task.

### Generative modelling

Besides learning predictive models on graphs, we might also care

about learning a generative model for graphs. With a generative model we

 can generate new graphs by sampling from a learned distribution or by

completing a graph given a starting point. A relevant application is in

the design of new drugs, where novel molecular graphs with specific

properties are desired as candidates to treat a disease.

A key challenge with graph generative models lies in modelling the

 term can be avoided by only predicting known edges and a subset of the

edges that are not present. The graphVAE learns to model positive

patterns of connectivity and some patterns of non-connectivity in the

adjacency matrix.

Another approach is to build a graph sequentially, by starting with a

 graph and applying discrete actions such as addition or subtraction of

nodes and edges iteratively. To avoid estimating a gradient for discrete

 actions we can use a policy gradient. This has been done via an

Final thoughts

--------------

Graphs are a powerful and rich structured data type that have

strengths and challenges that are very different from those of images

and text. In this article, we have outlined some of the milestones that

researchers have come up with in building neural network based models

that process graphs. We have walked through some of the important design

 choices that must be made when using these architectures, and hopefully

 the GNN playground can give an intuition on what the empirical results

of these design choices are. The success of GNNs in recent years creates

 a great opportunity for a wide range of new problems, and we are

excited to see what the field will bring.

Understanding Convolutions on Graphs

The Graph Neural Network Model

A Deep Learning Approach to Antibiotic Discovery

Learning to simulate complex physics with graph networks

Fake News Detection on Social Media using Geometric Deep Learning

Traffic prediction with advanced Graph Neural Networks

Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in {Real-Time}

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Distributed Representations of Words and Phrases and their Compositionality

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Glove: Global Vectors for Word Representation

Learning to Represent Programs with Graphs

Deep Learning for Symbolic Mathematics

KONECT

An Information Flow Model for Conflict and Fission in Small Groups

Learning Latent Permutations with Gumbel-Sinkhorn Networks

Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs

Neural Message Passing for Quantum Chemistry

Relational inductive biases, deep learning, and graph networks

Deep Sets

Molecular graph convolutions: moving beyond fingerprints

Feature-wise transformations

Leffingwell Odor Dataset

Machine Learning for Scent: Learning Generalizable Perceptual Representations of Small Molecules

Benchmarking Graph Neural Networks

Design Space for Graph Neural Networks

Principal Neighbourhood Aggregation for Graph Nets

Graph Traversal with Tensor Functionals: A Meta-Algorithm for Scalable Learning

Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels

Representation Learning on Graphs with Jumping Knowledge Networks

Neural Execution of Graph Algorithms

Graph Theory

A nested-graph model for the representation and manipulation of complex objects

Modeling polypharmacy side effects with graph convolutional networks

Machine learning in chemical reaction space

Graphs and Hypergraphs

HyperGCN: A New Method of Training Graph Convolutional Networks on Hypergraphs

Hierarchical Message-Passing Graph Neural Networks

Little Ball of Fur

Sampling from large graphs

Metropolis Algorithms for Representative Subgraph Sampling

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

GraphSAINT: Graph Sampling Based Inductive Learning Method

How Powerful are Graph Neural Networks?

Rep the Set: Neural Networks for Learning Set Representations

Message Passing Networks for Molecules with Tetrahedral Chirality

N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules

Dual-Primal Graph Convolutional Networks

Viewing matrices & probability as graphs

Graphs and Matrices

Modern Graph Theory

Attention Is All You Need

Graph Attention Networks

Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

Transformers are Graph Neural Networks

Using Attribution to Decode Dataset Bias in Neural Network Models for Chemistry

GNNExplainer: Generating Explanations for Graph Neural Networks

Explainability Methods for Graph Convolutional Neural Networks

Evaluating Attribution for Graph Neural Networks

Variational Graph Auto-Encoders

GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models

Optimization of Molecules via Deep Reinforcement Learning

Self-Referencing Embedded Strings (SELFIES): A 100% robust molecular string representation

GraphGen: A Scalable Approach to Domain-agnostic Labeled Graph Generation

A Gentle Introduction to Graph Neural Networks.html