### Contents

[Adversarial MNIST CAs](#adversarial-mnist-cas) |
[Adversarial Injections for Growing CAs](#adversarial-injections-for-growing-cas) |

[Perturbing the states of Growing CAs](#perturbing-the-states-of-growing-cas) |
[Related Work](#related-work)
[Discussion](#discussion)

![](images/multiple-pages.svg)

 This article is part of the
 [Differentiable Self-organizing Systems Thread](/2020/selforg/),
 an experimental format collecting invited short articles delving into
 differentiable self-organizing systems, interspersed with critical
 commentary from several experts in adjacent fields.

[Self-Organising Textures](/selforg/2021/textures/)

 This article makes strong use of colors in figures and demos. Click [here](#colorwheel) to adjust the color palette.

In a complex system, whether biological, technological, or social, how can we discover signaling events that will alter system-level behavior in desired ways? Even when the rules governing the individual components of these complex systems are known, the inverse problem - going from desired behaviour to system design - is at the heart of many barriers for the advance of biomedicine, robotics, and other fields of importance to society.

Biology, specifically, is transitioning from a focus on mechanism (what is required for the system to work) to a focus on information (what algorithm is sufficient to implement adaptive behavior). Advances in machine learning represent an exciting and largely untapped source of inspiration and tooling to assist the biological sciences. Growing Neural Cellular Automata  and Self-classifying MNIST Digits  introduced the Neural Cellular Automata (Neural CA) model and demonstrated how tasks requiring self-organisation, such as pattern growth and self-classification of digits, can be trained in an end-to-end, differentiable fashion. The resulting models were robust to various kinds of perturbations: the growing CA expressed regenerative capabilities when damaged; the MNIST CA were responsive to changes in the underlying digits, triggering reclassification whenever necessary. These computational frameworks represent quantitative models with which to understand important biological phenomena, such as scaling of single cell behavior rules into reliable organ-level anatomies. The latter is a kind of anatomical homeostasis, achieved by feedback loops that must recognize deviations from a correct target morphology and progressively reduce anatomical error.

In this work, we *train adversaries* whose goal is to reprogram CA into doing something other than what they were trained to do. In order to understand what kinds of lower-level signals alter system-level behavior of our CA, it is important to understand how these CA are constructed and where local versus global information resides.

The system-level behavior of Neural CA is affected by:

* **Individual cell states.** States store information which is used for both diversification among cell behaviours and for communication with neighbouring cells.
* **The model parameters.** These describe the input/output behavior of a cell and are shared by every cell of the same family. The model parameters can be seen as *the way the system works*.
* **The perceptive field.** This is how cells perceive their environment. In Neural CA, we always restrict the perceptive field to be the eight nearest neighbors and the cell itself. The way cells are perceived by each other is different between the Growing CA and MNIST CA. The Growing CA perceptive field is a set of weights fixed both during training and inference, while the MNIST CA perceptive field is learned as part of the model parameters.

Perturbing any of these components will result in system-level behavioural changes.

We will explore two kinds of adversarial attacks: 1) injecting a few adversarial cells into an existing grid running a pretrained model; and 2) perturbing the global state of all cells on a grid.

For the first type of adversarial attacks we train a new CA model that, when placed in an environment running one of the original models described in the previous articles, is able to hijack the behavior of the collective mix of adversarial and non-adversarial CA. This is an example of injecting CA with differing *model parameters* into the system. In biology, numerous forms of hijacking are known, including viruses that take over genetic and biochemical information flow , bacteria that take over physiological control mechanisms  and even regenerative morphology of whole bodies , and fungi and toxoplasma that modulate host behavior . Especially fascinating are the many cases of non-cell-autonomous signaling developmental biology and cancer, showing that some cell behaviors can significantly alter host properties both locally and at long range. For example, bioelectrically-abnormal cells can trigger metastatic conversion in an otherwise normal body (with no genetic defects) , while management of bioelectrical state in one area of the body can suppress tumorigenesis on the other side of the organism . Similarly, amputation damage in one leg initiates changes to ionic properties of cells in the contralateral leg , while the size of the developing brain is in part dictated by the activity of ventral gut cells . All of these phenomena underlie the importance of understanding how cell groups make collective decisions, and how those tissue-level decisions can be subverted by the activity of a small number of cells. It is essential to develop quantitative models of such dynamics, in order to drive meaningful progress in regenerative medicine that controls system-level outcomes top-down, where cell- or molecular-level micromanagement is infeasible .

The second type of adversarial attacks interact with previously trained growing CA models by *perturbing the states within cells*. We apply a global state perturbation to all living cells. This can be seen as inhibiting or enhancing combinations of state values, in turn hijacking proper communications among cells and within the cell’s own states. Models like this represent not only ways of thinking about adversarial relationships in nature (such as parasitism and evolutionary arms races of genetic and physiological mechanisms), but also a roadmap for the development of regenerative medicine strategies. Next-generation biomedicine will need computational tools for inferring minimal, least-effort interventions that can be applied to biological systems to predictively change their large-scale anatomical and behavioral properties.

Adversarial MNIST CA [Try in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/adversarial_reprogramming_ca/adversarial_mnist_ca.ipynb)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Recall how the Self-classifying MNIST digits task consisted of placing CA cells on a plane forming the shape of an MNIST digit. The cells then had to communicate among themselves in order to come to a complete consensus as to which digit they formed.

![](images/local_global_figure.svg)
Diagram showing the local vs. global information available in the cell collective.
 (a) Local information neighbourhood - each cell can only observe itself and its neighbors’ states, or the absence of neighbours.
 (b) Globally, the cell collective aggregates information from all parts of itself.
 (c) It is able to distinguish certain shapes that compose a specific digit (3 in the example).

Below we show examples of classifications made by the model trained in Self-classifying MNIST Digits.

Your browser does not support the video tag.

The original model behavior on unseen data. Classification mistakes have a red background.

In this experiment, **the goal is to create adversarial CA that can hijack the cell collective’s classification consensus to always classify an eight**. We use the CA model from  and freeze its parameters. We then train a new CA whose model architecture is identical to the frozen model but is randomly initialized. The training regime also closely approximates that of self-classifying MNIST digits CA. There are three important differences:

* Regardless of what the actual digit is, we consider *the correct classification to always be an eight*.
* For each batch and each pixel, the CA is randomly chosen to be either the pretrained model or the new adversarial one. The adversarial CA is used 10% of the time, and the pre-trained, frozen, model the rest of the time.
* Only the adversarial CA parameters are trained, the parameters of the pretrained model are kept frozen.

The adversarial attack as defined here only modifies a small percentage of the overall system, but the goal is to propagate signals that affect all the living cells. Therefore, these adversaries have to somehow learn to communicate deceiving information that causes wrong classifications in their neighbours and further cascades in the propagation of deceiving information by ‘unaware’ cells. The unaware cells’ parameters cannot be changed so the only means of attack by the adversaries is to cause a change in the cells’ states. Cells’ states are responsible for communication and diversification.

The task is remarkably simple to optimize, reaching convergence in as little as 2000 training steps (as opposed to the two orders of magnitude more steps needed to construct the original MNIST CA). By visualising what happens when we remove the adversaries, we observe that the adversaries must be constantly communicating with their non-adversarial neighbours to keep them convinced of the malicious classification. While some digits don’t recover after the removal of adversaries, most of them self-correct to the right classification. Below we show examples where we introduce the adversaries at 200 steps and remove them after a further 200 steps.

Your browser does not support the video tag.

We introduce the adversaries (red pixels) after 200 steps and remove them after 200 more steps. Most digits recover, but not all. We highlight mistakes in classification with a red background.

While we trained the adversaries with a 10-to-90% split of adversarial vs. non-adversarial cells, we observe that often significantly fewer adversaries are needed to succeed in the deception. Below we evaluate the experiment with just one percent of cells being adversaries.

Your browser does not support the video tag.

Adversaries constituting up 1% of the cell collective (red pixels). We highlight mistakes in classification with a red background.

We created a demo playground where the reader can draw digits and place adversaries with surgical precision. We encourage the reader to play with the demo to get a sense of how easily non-adversarial cells are swayed towards the wrong classification.

Adversarial Injections for Growing CA [Try in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/adversarial_reprogramming_ca/adversarial_growing_ca.ipynb#scrollTo=ByHbsY0EuyqB)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The natural follow up question is whether these adversarial attacks work on Growing CA, too. The Growing CA goal is to be able to grow a complex image from a single cell, and having its result be persistent over time and robust to perturbations. In this article, we focus on the lizard pattern model from Growing CA.

Your browser does not support the video tag.

The target CA to hijack.

The goal is to have some adversarial cells change the global configuration of all the cells. We choose two new targets we would like the adversarial cells to try and morph the lizard into: a tailless lizard and a red lizard.

![](images/lizard_new_targets_exp2.png)
The desired mutations we want to apply.

These targets have different properties:

* **Red lizard:** converting a lizard from green to red would show a global change in the behaviour of the cell collective. This behavior is not present in the dynamics observed by the original model. The adversaries are thus tasked with fooling other cells into doing things they have never done before (create the lizard shape as before, but now colored in red).
* **Tailless lizard:** having a severed tail is a more localized change that only requires some cells to be fooled into behaving in the wrong way: the cells at the base of the tail need to be convinced they constitute the edge or silhouette of the lizard, instead of proceeding to grow a tail as before.

Just like in the previous experiment, our adversaries can only indirectly affect the states of the original cells.

We first train adversaries for the tailless target with a 10% chance for any given cell to be an adversary. We prohibit cells to be adversaries if they are outside the target pattern; i.e. the tail contains no adversaries.

Your browser does not support the video tag.

10% of the cells are adversarial.

The video above shows six different instances of the same model with differing stochastic placement of the adversaries. The results vary considerably: sometimes the adversaries succeed in removing the tail, sometimes the tail is only shrunk but not completely removed, and other times the pattern becomes unstable. Training these adversaries required many more gradient steps to achieve convergence, and the pattern converged to is qualitatively worse than what was achieved for the adversarial MNIST CA experiment.

The red lizard pattern fares even worse. Using only 10% adversarial cells results in a complete failure: the original cells are unaffected by the adversaries. Some readers may wonder whether the original pretrained CA has the requisite skill, or ‘subroutine’ of producing a red output at all, since there are no red regions in the original target, and may suspect this was an impossible task to begin with. Therefore, we increased the proportion of adversarial cells until we managed to find a successful adversarial CA, if any were possible.

Your browser does not support the video tag.

Adversaries are 60% of the cells. At step 500, we stop the image and show only cells that are from the original model.

In the video above we can see how, at least in the first stages of morphogenesis, 60% of adversaries are capable of coloring the lizard red. Take particular notice of the “step 500” The still-image of the video is on step 500, and the video stops for a bit more than a second on step 500., where we hide the adversarial cells and show only the original cells. There, we see how a handful of original cells are colored in red. This is proof that the adversaries successfully managed to steer neighboring cells to color themselves red, where needed.

However, the model is very unstable when iterated for periods of time longer than seen during training. Moreover, the learned adversarial attack is dependent on a majority of cells being adversaries. For instance, when using fewer adversaries on the order of 20-30%, the configuration is unstable.

In comparison to the results of the previous experiment, the Growing CA model shows a greater resistance to adversarial perturbation than those of the MNIST CA. A notable difference between the two models is that the MNIST CA cells have to always be ready and able to change an opinion (a classification) based on information propagated through several neighbors. This is a necessary requirement for that model because at any time the underlying digit may change, but most of the cells would not observe any change in their neighbors’ placements. For instance, imagine the case of a one turning into a seven where the lower stroke of each overlap perfectly. From the point of view of the cells in the lower stroke of the digit, there is no change, yet the digit formed is now a seven. We therefore hypothesise MNIST CA are more reliant and ‘trusting’ of continuous long-distance communication than Growing CA, where cells never have to reconfigure themselves to generate something different to before.

We suspect that more general-purpose Growing CA that have learned a variety of target patterns during training are more likely to be susceptible to adversarial attacks.

Perturbing the states of Growing CA [Try in a Notebook](https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/adversarial_reprogramming_ca/adversarial_growing_ca.ipynb#scrollTo=JaITnQv0k1iY)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

We observed that it is hard to fool Growing CA into changing their morphology by placing adversarial cells inside the cell collective. These adversaries had to devise complex local behaviors that would cause the non-adversarial cells nearby, and ultimately globally throughout the image, to change their overall morphology.

In this section, we explore an alternative approach: perturbing the global state of all cells without changing the model parameters of any cell.

As before, we base our experiments on the Growing CA model trained to produce a lizard. Every cell of a Growing CA has an internal state vector with 16 elements. Some of them are phenotypical elements (the RGBA states) and the remaining 12 serve arbitrary purposes, used for storing and communicating information. We can perturb the states of these cells to hijack the overall system in certain ways (the discovery of such perturbation strategies is a key goal of biomedicine and synthetic morphology). There are a variety of ways we can perform state perturbations. We will focus on *global state perturbations*, defined as perturbations that are applied on every living cell at every time step (analogous to “systemic” biomedical interventions, that are given to the whole organism (e.g., a chemical taken internally), as opposed to highly localized delivery systems). The new goal is to discover a certain type of global state perturbation that results in a stable new pattern.

![](images/figure_2.svg)
Diagram showing some possible stages for perturbing a lizard pattern. (a) We start from a seed that grows into a lizard (b) Fully converged lizard. (c) We apply a global state perturbation at every step. As a result, the lizard loses its tail. (d) We stop perturbing the state. We observe the lizard immediately grows back its tail.

We show 6 target patterns: the tailless and red lizard from the previous experiment, plus a blue lizard and lizards with various severed limbs and severed head.

![](images/mutations_mosaic.jpeg)
Mosaic of the desired mutations we want to apply.

We decided to experiment with a simple type of global state perturbation: applying a symmetric 16×1616\times1616×16 matrix multiplication AAA to every living cell at every step In practice, we also clip the state of cells such that they are bounded in [−3,+3][-3, +3][−3,+3]. This is a minor detail and it helps stabilise the model.. To give insight on why we chose this, an even simpler “state addition” mutation (a mutation consisting only of the addition of a vector to every state) would be insufficient because the value of the states of our models are unbounded, and often we would want to suppress something by setting it to zero. The latter is generally impossible with constant state additions, as a constant addition or subtraction of a value would generally lead to infinity, except for some fortunate cases where the natural residual updates of the cells would cancel out with the constant addition at precisely state value zero. However, matrix multiplications have the possibility of amplifying/suppressing combinations of elements in the states: multiplying a state value repeatedly for a constant value less than one can easily suppress a state value to zero. We constrain the matrix to be symmetric for reasons that will become clear in the following section.

We initialize AAA with the identity matrix III and train AAA just as we would train the original Growing CA, albeit with the following differences:

* We perform a global state perturbation as described above, using AAA, at every step.
* The underlying CA parameters are frozen and we only train AAA.
* We consider the set of initial image configurations to be both the seed state and the state with a fully grown lizard (as opposed to the Growing CA article, where initial configurations consisted of the seed state only).

Your browser does not support the video tag.

Effect of applying the trained perturbations.

The video above shows the model successfully discovering global state perturbations able to change a target pattern to a desired variation. We show what happens when we stop perturbing the states (an out-of-training situation) at step 500 through step 1000, then reapplying the mutation. This demonstrates the ability of our perturbations to achieve the desired result both when starting from a seed, and when starting from a fully grown pattern. Furthermore it demonstrates that the original CA easily recover from these state perturbations once it goes away. This last result is perhaps not surprising given how robust growing CA models are in general.

Not all perturbations are equally effective. In particular, the headless perturbation is the least successful as it results in a loss of other details across the whole lizard pattern such as the white coloring on its back. We hypothesize that the best perturbation our training regime managed to find, due to the simplicity of the perturbation, was suppressing a “structure” that contained both the morphology of the head and the white colouring. This may be related to the concept of differentiation and distinction of biological organs. Predicting what kinds of perturbations would be harder or impossible to be done, before trying them out empirically, is still an open research question in biology. On the other hand, a variant of this kind of synthetic analysis might help with defining higher order structures within biological and synthetic systems.

### Directions and compositionality of perturbations

Our choice of using a symmetric matrix for representing global state perturbations is justified by a desire to have compositionality. Every complex symmetric matrix AAA can be diagonalized as follows:

A=QΛQ⊺A = Q \Lambda Q^\intercalA=QΛQ⊺

where Λ\LambdaΛ is the diagonal eigenvalues matrix and QQQ is the unitary matrix of its eigenvectors. Another way of seeing this is applying a change of basis transformation, scaling each component proportional to the eigenvalues, and then changing back to the original basis. This should also give a clearer intuition on the ease of suppressing or amplifying combinations of states. Moreover, we can now infer what would happen if all the eigenvalues were to be one. In that case, we would naturally have QIQ⊺=IQ I Q^\intercal = IQIQ⊺=I resulting in a no-op (no change): the lizard would grow as if no perturbation was performed. We can now decompose QΛQ⊺=Q(D+I)Q⊺Q \Lambda Q^\intercal = Q (D + I) Q^\intercalQΛQ⊺=Q(D+I)Q⊺ where D is the *perturbation direction* (Λ−I\Lambda - IΛ−I) in the “eigenvalue space”. Suppose we use a coefficient kkk to scale D: Ak=Q(kD+I)Q⊺A\_k = Q (kD + I) Q^\intercalAk=Q(kD+I)Q⊺. If k=1k=1k=1, we are left with the original perturbation AAA and when k=0k=0k=0, we have the no-op III. Naturally, one question would be whether we can explore other values for kkk and discover meaningful perturbations. Since

Ak=Q(kD+I)Q⊺=kA+(1−k)IA\_k = Q (kD + I) Q^\intercal = k A + (1-k) IAk=Q(kD+I)Q⊺=kA+(1−k)I

we do not even have to compute eigenvalues and eigenvectors and we can simply scale AAA and III accordingly.

Let us then take the tailless perturbation and see what happens as we vary kkk:

Your browser does not support the video tag.

Effect of the interpolation between an identity matrix and the ’perturbation direction of the tail perturbation.

As we change k=1k=1k=1 to k=0k=0k=0 we can observe the tail becoming more complete. Surprisingly, if we make kkk negative, the lizard grows a longer tail. Unfortunately, the further away we go, the more unstable the system becomes and eventually the lizard pattern grows in an unbounded fashion. This behaviour likely stems from that perturbations applied on the states also affect the homeostatic regulation of the system, making some cells die out or grow in different ways than before, resulting in a behavior akin to “cancer” in biological systems.

**Can we perform multiple, individually trained, perturbations at the same time?**

Suppose we have two perturbations AAA and BBB and their eigenvectors are the same (or, more realistically, sufficiently similar). Then, Ak=Q(kADA+I)Q⊺A\_k = Q (k\_A D\_A + I) Q^\intercalAk=Q(kADA+I)Q⊺ and Bk=Q(kBDB+I)Q⊺B\_k = Q (k\_B D\_B + I) Q^\intercalBk=Q(kBDB+I)Q⊺.

In that case,

comb(Ak,Bk)=Q(kADA+kBDB+I)Q⊺=kAA+kBB+(1−kA−kB)Icomb(A\_k, B\_k) = Q(k\_A D\_A + k\_B D\_B + I)Q^\intercal = k\_A A + k\_B B + (1 - k\_A - k\_B)Icomb(Ak,Bk)=Q(kADA+kBDB+I)Q⊺=kAA+kBB+(1−kA−kB)I

would result in something meaningful. At the very least, if A=BA = BA=B, setting kA=kB=0.5k\_A = k\_B = 0.5kA=kB=0.5 would result in exactly the same perturbation.

We note that DAD\_ADA and DBD\_BDB are effectively a displacement from the identity III and we have empirically observed how given any trained displacement DAD\_ADA, for 0≤kA≤10 \leq k\_A \leq 10≤kA≤1 adding kADAk\_A D\_AkADA results in a stable perturbation. We then hypothesize that as long as we have two perturbations whose positive directions kkk are kA+kB≤1k\_A + k\_B \leq 1kA+kB≤1, this could result in a stable perturbation. An intuitive understanding of this is interpolating stable perturbations using the direction coefficients.

In practice, however, the eigenvectors are also different, so the results of the combination will likely be worse the more different the respective eigenvector bases are.

Below, we interpolate the direction coefficients, while keeping their sum to be one, of two types of perturbations: tailless and no-leg lizards.

Your browser does not support the video tag.

Effect of composing two trained perturbations while keeping the sum of kkks as 1.

While it largely achieves what we expect, we observe some unintended effects such as the whole pattern starting to traverse vertically in the grid. Similar results happen with other combinations of perturbations. What happens if we remove the restriction of the sum of kkks being equal to one, and instead add both perturbations in their entirety? We know that if the two perturbations were the same, we would end twice as far away from the identity perturbation, and in general we expect the variance of these perturbations to increase. Effectively, this means going further and further away from the stable perturbations discovered during training. We would expect more unintended effects that may disrupt the CA as the sum of kkks increases.

Below, we demonstrate what happens when we combine the tailless and the no-leg lizard perturbations at their fullest. Note that when we set both kkks to one, the resulting perturbation is equal to the sum of the two perturbations minus an identity matrix.

Your browser does not support the video tag.

Effect of composing two perturbations.

Surprisingly, the resulting pattern is almost as desired. However, it also suffers from the vertical movement of the pattern observed while interpolating kkks.

This framework can be generalized to any arbitrary number of perturbations. Below, we have created a small playground that allows the reader to input their desired combinations. Empirically, we were surprised by how many of these combinations result in the intended perturbations and qualitatively it appears that bounding kkk to one results in generally more stable patterns. We also observed how exploring negative kkk values is usually more unstable.

Related work
------------

This work is inspired by Generative Adversarial Networks (GANs) . While with GANs it is typical to cotrain pairs of models, in this work we froze the original CA and trained the adversaries only. This setup is to the greatest degree inspired by the seminal work *Adversarial Reprogramming of Neural Networks* .

The kinds of state perturbations performed in this article can be seen as targeted latent state manipulations. Word2vec  shows how latent vector representations can have compositional properties and Fader Networks  show similar behaviors for image processing. Both of these works and their related work were of inspiration to us.

### Influence maximization

Adversarial cellular automata have parallels to the field of influence maximization. Influence maximization involves determining the optimal nodes to influence in order to maximize influence over an entire graph, commonly a social graph, with the property that nodes can in turn influence their neighbours. Such models are used to model a wide variety of real-world applications involving information spread in a graph.    A common setting is that each vertex in a graph has a binary state, which will change if and only if a sufficient fraction of its neighbours’ states switch. Examples of such models are social influence maximization (maximally spreading an idea in a network of people), contagion outbreak modelling  (usually to minimize the spread of a disease in a network of people) and cascade modeling  (when small perturbations to a system bring about a larger ‘phase change’). At the time of writing this article, for instance, contagion minimization is a model of particular interest. NCA are a graph - each cell is a vertex and has edges to its eight neighbours, through which it can pass information. This graph and message structure is significantly more complex than the typical graph underlying much of the research in influence maximization, because NCA cells pass vector-valued messages and have a complex update rules for their internal states, whereas graphs in influence maximization research typically consist of more simple binary cells states and threshold functions on edges determining whether a node has switched states. Many concepts from the field could be applied and are of interest, however.

For example, in this work, we have made an assumption that our adversaries can be positioned anywhere in a structure to achieve a desired behaviour. A common focus of investigation in influence maximization problems is deciding which nodes in a graph will result in maximal influence on the graph, referred to as target set selection . This problem isn’t always tractable, often NP-hard, and solutions frequently involve simulations. Future work on adversarial NCA may involve applying techniques from influence maximization in order to find the optimal placement of adversarial cells.

Discussion
----------

This article showed two different kinds of adversarial attacks on Neural CA.

Injections of adversarial CA in a pretrained Self-classifying MNIST CA showed how an existing system of cells that are heavily reliant on the passing of information among each other is easily swayed by deceitful signaling. This problem is routinely faced by biological systems, which face hijacking of behavioral, physiological, and morphological regulatory mechanisms by parasites and other agents in the biosphere with which they compete. Future work in this field of computer technology can benefit from research on biological communication mechanisms to understand how cells maximize reliability and fidelity of inter- and intra-cellular messages required to implement adaptive outcomes.

The adversarial injection attack was much less effective against Growing CA and resulted in overall unstable CA. This dynamic is also of importance to the scaling of control mechanisms (swarm robotics and nested architectures): a key step in “multicellularity” (joining together to form larger systems from sub-agents ) is informational fusion, which makes it difficult to identify the source of signals and memory engrams. An optimal architecture would need to balance the need for validating control messages with a possibility of flexible merging of subunits, which wipes out metadata about the specific source of informational signals. Likewise, the ability to respond successfully to novel environmental challenges is an important goal for autonomous artificial systems, which may import from biology strategies that optimize tradeoff between maintaining a specific set of signals and being flexible enough to establish novel signaling regimes when needed.

The global state perturbation experiment on Growing CA shows how it is still possible to hijack these CA towards stable out-of-training configurations and how these kinds of attacks are somewhat composable in a similar way to how embedding spaces are manipulable in the natural language processing and computer vision fields . However, this experiment failed to discover stable out-of-training configurations that persist *after the perturbation was lifted*. We hypothesize that this is partially due to the regenerative capabilities of the pretrained CA, and that other models may be less capable of recovery from arbitrary perturbations.

Adversarial Reprogramming of Neural Cellular Automata

Ettore Randazzo
Alexander Mordvintsev
Eyvind Niklasson
Michael Levin

Summary: Reprogramming Neural CA to exhibit novel behaviour, using adversarial attacks.

Growing Neural Cellular Automata

Self-classifying MNIST Digits

Herpes Simplex Virus: The Hostile Guest That Takes Over Your Home

The role of gut microbiota (commensal bacteria) and the mucosal barrier in the pathogenesis of inflammatory and autoimmune diseases and cancer: contribution of germ-free and gnotobiotic animal models of human diseases

Regulation of axial and head patterning during planarian regeneration by a commensal bacterium

Toxoplasma gondii infection and behavioral outcomes in humans: a systematic review

Resting potential, oncogene-induced tumorigenesis, and metastasis: the bioelectric basis of cancer in vivo

Transmembrane voltage potential of somatic cells controls oncogene-mediated tumorigenesis at long-range

Cross-limb communication during Xenopus hindlimb regenerative response: non-local bioelectric injury signals

Local and long-range endogenous resting potential gradients antagonistically regulate apoptosis and proliferation in the embryonic CNS

Top-down models in biology: explanation and control of complex living systems above the molecular level

Maximizing the spread of influence through a social network

The Independent Cascade and Linear Threshold Models

A Survey on Influence Maximization in a Social Network

Simplicial models of social contagion

Cascading Behavior in Networks: Algorithmic and Economic Issues

On the Approximability of Influence in Social Networks

The Computational Boundary of a “Self”: Developmental Bioelectricity Drives Multicellularity and Scale-Free Cognition


![](images/multiple-pages.svg)

 This article is part of the [Circuits thread](/2020/circuits/), an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.

[Branch Specialization](/2020/circuits/branch-specialization/)

Introduction
------------

 Open up any ImageNet conv net and look at the weights in the last layer. You’ll find a uniform spatial pattern to them, dramatically unlike anything we see elsewhere in the network. No individual weight is unusual, but the uniformity is so striking that when we first discovered it we thought it must be a bug. Just as different biological tissue types jump out as distinct under a microscope, the weights in this final layer jump out as distinct when visualized with NMF. We call this phenomenon *weight banding*.

Microscope slides of different tissues
Muscle tissue

Epithelial tissue

Typical layer
Layer with weight banding
NMF of weights at different layers

[1](#figure-1). When [visualized with NMF](https://drafts.distill.pub/distillpub/post--circuits-visualizing-weights#one-simple-trick), the weight banding in layer `mixed_5b` is as visually striking compared to any other layer in InceptionV1 (here shown: `mixed_3a`) as the smooth, regular striation of muscle tissue is when compared to any other tissue (here shown: cardiac muscle tissue and epithelial tissue).

 So far, the [Circuits thread](https://distill.pub/2020/circuits/) has mostly focused on studying very small pieces of neural network – [individual neurons](https://distill.pub/2020/circuits/early-vision/) and small circuits. In contrast, weight banding is an example of what we call a “structural phenomenon,” a larger-scale pattern in the circuits and features of a neural network. Other examples of structural phenomena are the recurring symmetries we see in [equivariance](https://distill.pub/2020/circuits/equivariance/) motifs and the specialized slices of neural networks we see in [branch specialization](https://distill.pub/2020/circuits/branch-specialization/).

 In the case of weight banding, we think of it as a structural phenomenon because the pattern appears at the scale of an entire layer.

 Weight banding also seems similar in flavor to the [checkerboard artifacts](https://distill.pub/2016/deconv-checkerboard/) that form during deconvolution.

 In addition to describing weight banding, we’ll explore when and why it occurs. We find that there appears to be a causal link between whether a model uses global average pooling or fully connected layers at the end, suggesting that weight banding is part of an algorithm for preserving information about larger scale structure in images. Establishing causal links like this is a step towards closing the loop between practical decisions in training neural networks and the phenomena we observe inside them.

Where weight banding occurs
---------------------------

 Weight banding consistently forms in the final convolutional layer of vision models with global average pooling.

 In order to see the bands, we need to visualize the spatial structure of the weights, as shown below. We typically do this using NMF, [as described in](https://drafts.distill.pub/distillpub/post--circuits-visualizing-weights/#one-simple-trick) Visualizing Weights. For each neuron, we take the weights connecting it to the previous layer. We then use NMF to reduce the number of dimensions corresponding to channels in the previous layer down to 3 factors, which we can map to RGB channels. Since which factor is which is arbitrary, we use a heuristic to make the mapping consistent across neurons. This reveals a very prominent pattern of horizontalThe stripes aren’t always perfectly horizontal - sometimes they exhibit a slight preference for extra weight in the center of the central band, as seen in some examples below. stripes.

[2](#figure-2).
 These common networks have pooling operations before their fully
 connected layers and consistently show banding at their last
 convolutional layers.

InceptionV1
mixed 5b

ResNet50
block 4 unit 3

VGG19
conv5

 Interestingly, AlexNet does not exhibit this phenomenon.

[3](#figure-3).
 AlexNet does not have a pooling operation before its fully connected
 layers and does not show banding at its last convolutional
 layer.

 To make it easier to look for groups of similar weights, we
 sorted the neurons at each layer by similarity of their reduced
 forms.

AlexNet
conv5

 Unlike most modern vision models, AlexNet does not use global average pooling. Instead, it has a fully connected layer directly connected to its final convolutional layer, allowing it to treat different positions differently. If one looks at the weights of this fully connected layer, the weights strongly vary as a function of the global y position.

 The horizontal stripes in weight banding mean that the filters don’t care about horizontal position, but are strongly encoding relative vertical position. Our hypothesis is that weight banding is a learned way to preserve spatial information as it gets lost through various pooling operations.

 In the next section, we will construct our own simplified vision network and investigate variations on its architecture in order to understand exactly which conditions are necessary to produce weight banding.

What affects banding
--------------------

 We’d like to understand which architectural decisions affect weight banding. This will involve trying out different architectures and seeing whether weight banding persists.

 Since we will only want to change a single architectural parameter at a time, we will need a consistent baseline to apply our modifications to. Ideally, this baseline would be as simple as possible.

 We created a simplified network architecture with 6 groups of convolutions, separated by L2 pooling layers. At the end, it has a global average pooling operation that reduces the input to 512 values that are then fed to a fully connected layer with 1001 outputs.

[4](#figure-4). Our simplified vision network architecture.

 This simplified network reliably produces weight banding in its last layer
 (and usually in the two preceding layers as well).

[5](#figure-5). NMF of the weights in the last layer of the simplified model shows clear weight banding.

simplified model (`5b`), baseline

 In the rest of this section, we’ll experiment with modifying this architecture and its training settings and seeing if weight banding is preserved.

### Rotating images 90 degrees

 To rule out bugs in training or some strange numerical problem, we decided
 to do a training run with the input rotated by 90 degrees. This sanity check
 yielded a very clear result showing *vertical* banding in the resulting
 weights, instead of horizontal banding. This is a clear indication that banding is a result of properties
 within the ImageNet dataset which make spatial vertical position(or, in the case of the rotated dataset, spatial horizontal position) relevant.

[6](#figure-6). simplified model (`5b`), 90º rotation

### Fully connected layer without global average pooling

 We remove the global average pooling step in our simplified model, allowing the fully connected layer to see all spatial positions at once. This model did **not** exhibit weight banding, but used 49x more parameters in the fully connected layer and overfit to the training set. This is pretty strong evidence that the use of aggressive pooling after the last convolutions in common models causes weight banding. This result is also consistent with AlexNet not showing this banding phenomenon (since it also does not have global average pooling).

[7](#figure-7). simplified model (`5b`), no pooling before fully connected layer

### Average pooling along x-axis only

 We average out each row of the final convolutional layer, so that vertical absolute position is preserved but horizontal absolute position is not.Since this model has 7x7 spatial positions in the final convolutional layer, this modification increases the number of parameters in the fully connected layer by 7x, but not the 49x of a complete fully connected layer with no pooling at all. The banding at the last layer seems to go away, but on closer investigation, clear banding is still visible in layer `5a`, similar to the baseline model’s `5b`. We found this result surprising.

[8](#figure-8).
 NMF of weights in `5a` and `5b` in a version of the simplified model modified to have pooling only along the x-axis. Banding is gone from `5b` but reappears in `5a`!

 simplified model (`5a`), x-axis pooling

simplified model (`5b`), x-axis pooling

### Approaches where weight banding persisted

 We tried each of the modifications below, and found that weight banding was still present in each of these variants.

* Global average pooling with learned spatial masks. By applying several different spatial masks and global average pooling, we can allow the model to preserve some spatial information. Intuitively, each mask can select for a different subset of spatial positions.

 We tried experimental runs using each of 3, 5, or 16 different masks.

 The masks that were learned corresponded to large-scale global structure, but banding was still strongly present.
* Using an attention layer instead of pooling/fully connected combination after layer
 `5b`.
* Adding a 7x7x512 mask with learned weights after `5b`. The hope was that a
 mask would help each `5b` neuron focus on the right parts of the 7x7 image
 without a convolution.
* Adding CoordConv channels to the inputs
 of `5a` and `5b`.
* Splitting the output of `5b` into 16 7x7x32 channel groups and feeding
 each group its own fully connected layer. The output of the 16 fully connected layers is then
 concatenated into the input of the final 1001-class fully connected layer.
* Using a global max pool, 4096-unit fully connected layer, then 1001-unit fully connected layer (inspired
 by VGG).

 An interactive diagram allowing you to explore the weights for these experiments and more can be found in the [appendix](#figure-12).

Confirming banding interventions in common architectures
--------------------------------------------------------

 In the previous section, we observed two interventions that clearly affected weight banding: rotating the dataset by 90º and removing the global average pooling before the fully connected layer.
 To confirm that these effects hold beyond our simplified model, we decided to make the same interventions to three
 common architectures (InceptionV1, ResNet50, VGG19) and train them from
 scratch.

With one exception, the effect holds in all three models.

#### InceptionV1

[9](#figure-9). Inception V1, layer `mixed_5c`, 5x5 convolution

baseline

90º rotation

global average pooling layer removed

#### ResNet50

[10](#figure-10). ResNet50, last 3x3 convolutional layer

baseline

90º rotation

global average pooling layer removed

#### VGG19

[11](#figure-11). VGG19, last 3x3 convolutional layer.

baseline

90º rotation

global average pooling layer removed

 The one exception is VGG19, where the removal of the pooling operation before its set of fully connected layers did not eliminate weight banding as expected; these weights look fairly similar to the baseline. However, it clearly responds to rotation.

Conclusion
----------

 Once we really understand neural networks, one would expect us to be able to leverage that understanding to design more effective neural networks architectures. Early papers, like Zeiler et al, emphasized this quite strongly, but it’s unclear whether there have yet been any significant successes in doing this. This hints at significant limitations in our work. It may also be a missed opportunity: it seems likely that if interpretability was useful in advancing neural network capabilities, it would become more integrated into other research and get attention from a wider range of researchers.

 It’s unclear whether weight banding is “good” or “bad.”On one hand, the 90º rotation experiment shows that weight banding is a product of the dataset and is encoding useful information into the weights. However, if spatial information could flow through the network in a different, more efficient way, then perhaps the channels would be able to focus on encoding relationships between features without needing to track spatial positions. We don’t have any recommendation or action to take away from it. However, it is an example of a consistent link between architecture decisions and the resulting trained weights. It has the right sort of flavor for something that could inform architectural design, even if it isn’t particularly actionable itself.

 More generally, weight banding is an example of a large-scale structure. One of the major limitations of circuits has been how small-scale it is. We’re hopeful that larger scale structures like weight banding may help circuits form a higher-level story of neural networks.

![](images/multiple-pages.svg)

 This article is part of the [Circuits thread](/2020/circuits/), an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.

[Branch Specialization](/2020/circuits/branch-specialization/)

Weight Banding

Michael Petrov
Chelsea Voss
Ludwig Schubert
Nick Cammarata
Gabriel Goh
Chris Olah

Summary: Weights in the final layer of common visual models appear as horizontal bands. We investigate how and why.

Muscle Tissue: Cardiac Muscle

Epithelial Tissues: Stratified Squamous Epithelium

Deconvolution and Checkerboard Artifacts

ImageNet Classification with Deep Convolutional Neural Networks

An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution


![](images/multiple-pages.svg)

 This article is part of the [Circuits thread](/2020/circuits/), an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.

[Visualizing Weights](/2020/circuits/visualizing-weights/)
[Weight Banding](/2020/circuits/weight-banding/)

Introduction
------------

 If we think of interpretability as a kind of “anatomy of neural networks,” most of the circuits thread has involved studying tiny little veins – looking at the small-scale, at individual neurons and how they connect. However, there are many natural questions that the small-scale approach doesn’t address.

 In contrast, the most prominent abstractions in biological anatomy involve larger-scale structures: individual organs like the heart, or entire organ systems like the respiratory system. And so we wonder: is there a “respiratory system” or “heart” or “brain region” of an artificial neural network? Do neural networks have any emergent structures that we could study that are larger-scale than circuits?

 This article describes *branch specialization*, one of three larger “structural phenomena” we’ve been able observe in neural networks. (The other two, [equivariance](https://distill.pub/2020/circuits/equivariance/) and [weight banding](https://distill.pub/2020/circuits/weight-banding/), have separate dedicated articles.) Branch specialization occurs when neural network layers are split up into branches. The neurons and circuits tend to self-organize, clumping related functions into each branch and forming larger functional units – a kind of “neural network brain region.” We find evidence that these structures implicitly exist in neural networks without branches, and that branches are simply reifying structures that otherwise exist.

 The earliest example of branch specialization that we’re aware of comes from AlexNet. AlexNet is famous as a jump in computer vision, arguably starting the deep learning revolution, but buried in the paper is a fascinating, rarely-discussed detail.

 The first two layers of AlexNet are split into two branches which can’t communicate until they rejoin after the second layer. This structure was used to maximize the efficiency of training the model on two GPUs, but the authors noticed something very curious happened as a result. The neurons in the first layer organized themselves into two groups: black-and-white Gabor filters formed on one branch and low-frequency color detectors formed on the other branch.

![](images/Figure_1.png)

[1](#figure-1). Branch specialization in the first two layers of AlexNet. Krizhevsky et al. observed the phenomenon we call branch specialization in the first layer of AlexNet by visualizing their weights to RGB channels; here, we use [feature visualization](https://distill.pub/2017/feature-visualization/) to show how this phenomenon extends to the second layer of each branch.

 Although the first layer of AlexNet is the only example of branch specialization we’re aware of being discussed in the literature, it seems to be a common phenomenon. We find that branch specialization happens in later hidden layers, not just the first layer. It occurs in both low-level and high-level features. It occurs in a wide range of models, including places you might not expect it – for example, residual blocks in resnets can functionally be branches and specialize. Finally, branch specialization appears to surface as a structural phenomenon in plain convolutional nets, even without any particular structure causing it.

 Is there a large-scale structure to how neural networks operate? How are features and circuits organized within the model? Does network architecture influence the features and circuits that form? Branch specialization hints at an exciting story related to all of these questions.

What is a branch?
-----------------

 Many neural network architectures have *branches*, sequences of layers which temporarily don’t have access to “parallel” information which is still passed to later layers.

InceptionV1
 has nine sets of four-way branches called “Inception blocks.”

 has several two-way branches.
AlexNet

Residual Networks
 aren’t typically thought of as having branches, but residual blocks can be seen as a type of branch.

[2](#figure-2). Examples of branches in various types of neural network architectures.

 In the past, models with explicitly-labeled branches were popular (such as AlexNet and the Inception family of networks). In more recent years, these have become less common, but residual networks – which can be seen as implicitly having branches in their residual blocks – have become very common. We also sometimes see branched architectures develop automatically in neural architecture search, an approach where the network architecture is learned.

 The implicit branching of residual networks has some important nuances. At first glance, every layer is a two-way branch. But because the branches are combined together by addition, we can actually rewrite the model to reveal that the residual blocks can be understood as branches in parallel:

We typically think of residual blocks as sequential layers, building on top of each other.

+

+

+

+

… but we can also conceptualize them as, to some extent, being parallel branches due to the skip connections. This means that residual blocks can potentially specialize.

+

[3](#figure-3). Residual blocks as branches in parallel.

 We typically see residual blocks specialize in very deep residual networks (e.g. ResNet-152). One hypothesis for why is that, in these models, the exact depth of a layer doesn’t matter and the branching aspect becomes more important than the sequential aspect.

 One of the conceptual weaknesses of normal branching models is that although branches can save parameters, it still requires a lot of parameters to mix values between branches. However, if you buy the branch interpretation of residual networks, you can see them as a strategy to sidestep this: residual networks intermix branches (e.g. block sparse weights) with low-rank connections (projecting all the blocks into the same sum and then back up). This seems like a really elegant way to handle branching. More practically, it suggests that analysis of residual networks might be well-served by paying close attention to the units in the blocks, and that we might expect the residual stream to be unusually polysemantic.

Why does branch specialization occur?
-------------------------------------

 Branch specialization is defined by features organizing between branches. In a normal layer, features are organized randomly: a given feature is just as likely to be any neuron in a layer. But in a branched layer, we often see features of a given type cluster to one branch. The branch has specialized on that type of feature.

 How does this happen? Our intuition is that there’s a positive feedback loop during training.

A1
B1
C1
D1
A2
B2
D2
D2
The first part of the branch is incentivized to form features relevant to the second half.
The second half of the branch prefers features which the first half provides primitives for.

[4](#figure-4). Hypothetical positive feedback loop of branch specialization during training.

 Another way to think about this is that if you need to cut a neural network into pieces that have limited ability to communicate with each other, it makes sense to organize similar features close together, because they probably need to share more information.

Branch specialization beyond the first layer
--------------------------------------------

 So far, the only concrete example we’ve shown of branch specialization is the first and second layer of AlexNet. What about later layers? AlexNet also splits its later layers into branches, after all. This seems to be unexplored, since studying features after the first layer is much harder.For the first layer, one can visualize the RGB weights; for later layers, one needs to use feature visualization.

 Unfortunately, branch specialization in the later layers of AlexNet is also very subtle. Instead of one overall split, it’s more like there’s dozens of small clusters of neurons, each cluster being assigned to a branch. It’s hard to be confident that one isn’t just seeing patterns in noise.

 But other models have very clear branch specialization in later layers. This tends to happen when a branch constitutes only a very small fraction of a layer, either because there are many branches or because one is much smaller than others. In these cases, the branch can specialize on a very small subset of the features that exist in a layer and reveal a clear pattern.

 For example, most of InceptionV1′s layers have a branched structure. The branches have varying numbers of units, and varying convolution sizes. The 5x5 branch is the smallest branch, and also has the largest convolution size. It’s often very specialized:

The 5x5 branch of mixed3a, a relatively early layer, is specialized on color detection, and especially black-and-white vs. color detection.
mixed3a\_5x5:

 It also contains a disproportionate number of boundary, eye, and fur detectors, many of which share sub-components with curves.
This branch contains all 30 of the curve-related features for this layer (all curves, double curves, circles, spirals, S-shape and more features, etc).
mixed3b\_5x5:

This branch appears to be specialized in complex shapes and 3D geometry detectors. We don’t have a full taxonomy of this layer to allow for a quantitative assessment.
mixed4a\_5x5:

3D Geometry / Complex Shapes
Curve Related
BW vs Color
Fur/Eye/Face Related
Other
Boundary Detectors
Other
Other
Brightness
Other Color Contrast

[5](#figure-5). Examples of branch specialization in `[mixed3a\_5x5](https://distill.pub/2017/feature-visualization/appendix/googlenet/3a.html#3a-192)`, `[mixed3b\_5x5](https://distill.pub/2017/feature-visualization/appendix/googlenet/3b.html#3b-320)`, and `[mixed4a\_5x5](https://distill.pub/2017/feature-visualization/appendix/googlenet/4a.html#4a-396)`.

 This is exceptionally unlikely to have occurred by chance.

 For example, all 9 of the black and white vs. color detectors in `mixed3a` are in `mixed3a_5x5`, despite it only being 32 out of the 256 neurons in the layer. The probability of that happening by chance is less than 1/108. For a more extreme example, all 30 of the curve-related features in `mixed3b` are in `mixed3b_5x5`, despite it being only 96 out of the 480 neurons in the layer. The probability of that happening by chance is less than 1/1020.

 It’s worth noting one confounding factor which might be influencing the specialization. The 5x5 branches are the smallest branches, but also have larger convolutions (5x5 instead of 3x3 or 1x1) than their neighbors.There is something which suggests that the branching plays an essential role: mixed3a and mixed3b are adjacent layers which contain relatively similar features and are at the same scale. If it was only about convolution size, why don’t we see any curves in the `mixed3a_5x5` branch or color in the `mixed3b_5x5` branch?

Why is branch specialization consistent?
----------------------------------------

 Perhaps the most surprising thing about branch specialization is that the same branch specializations seem to occur again and again, across different architectures and tasks.

 For example, the branch specialization we observed in AlexNet – the first layer specializing into a black-and-white Gabor branch vs. a low-frequency color branch – is a surprisingly robust phenomenon. It occurs consistently if you retrain AlexNet. It also occurs if you train other architectures with the first few layers split into two branches. It even occurs if you train those models on other natural image datasets, like Places instead of ImageNet. Anecdotally, we also seem to see other types of branch specialization recur. For example, finding branches that seem to specialize in curve detection seems to be quite common (although InceptionV1′s `mixed3b_5x5` is the only one we’ve carefully characterized).

 So, why do the same branch specializations occur again and again?

 One hypothesis seems very tempting. Notice that many of the same features that form in normal, non-branched models also seem to form in branched models. For example, the first layer of both branched and non-branched models contain Gabor filters and color features. If the same features exist, presumably the same weights exist between them.

 Could it be that branching is just surfacing a structure that already exists? Perhaps there’s two different subgraphs between the weights of the first and second conv layer in a normal model, with relatively small weights between them, and when you train a branched model these two subgraphs latch onto the branches.

 (This would be directionally similar to work finding modular substructures within neural networks.)

 To test this, let’s look at models which have non-branched first and second convolutional layers. Let’s take the weights between them and perform a singular value decomposition (SVD) on the absolute values of the weights. This will show us the main factors of variation in which neurons connect to different neurons in the next layer (irrespective of whether those connections are excitatory or inhibitory).

 Sure enough, the singular vector (the largest factor of variation) of the weights between the first two convolutional layers of InceptionV1 is color.

first convolutional layer
Neurons in the
organized by the left singular vectors of |W|.

InceptionV1 (tf-slim version) trained on ImageNet.

The first singular vector separates color and black and white, meaning that’s the largest dimension of variation in which neurons connect to which in the next layer.
Gabor filters and color features are far apart, meaning they tend to connect to different features in the next layer.

Singular Vector 1 (frequency?)

Singular Vector 0 (color?)

InceptionV1 trained on Places365

One more, the first singular vector separates color and black and white, meaning that’s the largest dimension of variation in which neurons connect to which in the next layer.

Singular Vector 1 (frequency?)

Singular Vector 0 (color?)

Singular Vector 1 (frequency?)

Singular Vector 0 (color?)

Singular Vector 1 (frequency?)

Singular Vector 0 (color?)
second convolutional layer
Neurons in the  organized by the right singular vectors of |W|.

[6](#figure-6). Singular vectors for the first and second convolutional layers of InceptionV1, trained on ImageNet (above) or Places365 (below). One can think of neurons being plotted closer together in this diagram as meaning they likely tend to connect to similar neurons.

 We also see that the second factor appears to be [frequency](/2020/circuits/frequency-edges/). This suggests an interesting prediction: perhaps if we were to split the layer into more than two branches, we’d also observe specialization in frequency in addition to color.

 This seems like it may be true. For example, here we see a high-frequency black-and-white branch, a mid-frequency mostly black-and-white branch, a mid-frequency color branch, and a low-frequency color branch.

![](images/Figure_7.png)

[7](#figure-7). We constructed a small ImageNet model with the first layer split into four branches. The rest of the model is roughly an InceptionV1 architecture.

Parallels to neuroscience
-------------------------

 We’ve shown that branch specialization is one example of a structural phenomenon — a larger-scale structure in a neural network. It happens in a variety of situations and neural network architectures, and it happens with *consistency* – certain motifs of specialization, such as color, frequency, and curves, happen consistently across different architectures and tasks.

 Returning to our comparison with anatomy, although we hesitate to claim explicit parallels to neuroscience, it’s tempting to draw analogies between branch specialization and the existence of regions of the brain focused on particular tasks.
 The visual cortex, the auditory cortex, Broca’s area and Wernicke’s area

 The subspecialization within the V2 area of the primate visual cortex is another strong example from neuroscience. One type of stripe within V2 is sensitive to orientation or luminance, whereas the other type of stripe contains color-selective neurons.

 We are grateful to Patrick Mineault for noting this analogy, and for further noting that the high-frequency features are consistent with some of the known representations of high-level features in the primate V2 area.

 – these are all examples of brain areas with such consistent specialization across wide populations of people that neuroscientists and psychologists have been able to characterize as having remarkably consistent functions.

 As researchers without expertise in neuroscience, we’re uncertain how useful this connection is, but it may be worth considering whether branch specialization can be a useful model of how specialization might emerge in biological neural networks.

![](images/multiple-pages.svg)

 This article is part of the [Circuits thread](/2020/circuits/), an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks.

[Visualizing Weights](/2020/circuits/visualizing-weights/)
[Weight Banding](/2020/circuits/weight-banding/)

 .comment {
 background-color: hsl(54, 78%, 96%);
 border-left: solid hsl(54, 33%, 67%) 1px;
 padding: 1em;
 color: hsla(0, 0%, 0%, 0.67);
 margin-top: 1em;
 }

 .comment h3 {
 font-size: 100%;
 font-weight: bold;
 text-transform: uppercase;
 margin-top: 0px;
 }

 .comment .commenter-description {
 font-style: italic;
 margin-bottom: 1em;
 margin-top: 0px;
 }
 .comment .commenter-description, .comment .commenter-description a {
 color: #777;
 font-size: 80%;
 line-height: 160%;
 }
 .comment p {
 margin-bottom: 0px;
 }
 .comment div {
 margin-top: 1em;
 }

### Comment

**[Matthew Nolan](https://www.ed.ac.uk/discovery-brain-sciences/our-staff/research-groups/matthew-nolan)** is Professor of Neural Circuits and Computation at the Centre for Discovery Brain Sciences and Simons Initiative for the Developing Brain, University of Edinburgh.
 **Ian Hawes** is a PhD student in the Wellcome Trust programme in Translational Neuroscience at the University of Edinburgh.

As neuroscientists we’re excited by this work as it offers fresh theoretical perspectives on long-standing questions about how brains are organised and how they develop. Branching and specialisation are found throughout the brain. A well studied example is the dorsal and ventral visual streams, which are associated with spatial and non-spatial visual processing. At the microcircuit level neurons in each pathway are similar. However, recordings of neural activity demonstrate remarkable specialisation; classic experiments from the 1970s and 80s established the idea that the ventral stream enables identification of objects whereas the dorsal stream represents their location. Since then, much has been learned about signal processing in these pathways but fundamental questions such as why there are multiple streams and how they are established remain unanswered.

From the perspective of a neuroscientist, a striking result from the investigation of branch specialization by Voss and her colleagues is that robust branch specialisation emerges in the absence of any complex branch specific design rules. Their analyses show that specialisation is similar within and across architectures, and across different training tasks. The implication here is that no specific instructions are required for branch specialisation to emerge. Indeed, their analyses suggest that it even emerges in the absence of predetermined branches. By contrast, the intuition of many neuroscientists would be that specialisation of different areas of the neocortex requires developmental mechanisms that are specific to each area. For neuroscientists aiming to understand how perceptual and cognitive functions of the brain arise, an important idea here is that developmental mechanisms that drive the separation of cortical pathways, such as the dorsal and ventral visual streams, may be absolutely critical.

While the parallels between branch specialization in artificial neural networks and neural circuits in the brain are striking, there are clearly major differences and many outstanding questions. From the perspective of building artificial neural networks, we wonder if branch specific tuning of individual units and their connectivity rules would enhance performance? In the brain, there is good evidence that the activation functions of individual neurons are fine-tuned between and even within distinct neural circuits. If this fine tuning confers benefits to the brain then we might expect similar benefits in artificial networks. From the perspective of understanding the brain, we wonder whether branch specialisation could help make experimentally testable predictions? If artificial networks can be engineered with branches that have organisation similar to branching pathways in the brain, then manipulations to these networks could be compared to experimental manipulations achieved with optogenetic and chemogenetic strategies. Given that many brain disorders involve changes to specific neural populations, similar strategies could give insights into how these pathological changes alter brain functions. For example, very specific populations of neurons are disrupted in early stages of Alzheimer’s disease. By disrupting corresponding units in neural network models one could explore the resulting computational deficits and possible strategies for restoration of cognitive functions.

Branch Specialization

Chelsea Voss
Gabriel Goh
Nick Cammarata
Michael Petrov
Ludwig Schubert
Chris Olah

Summary: When a neural network layer is divided into multiple branches, neurons self-organize into coherent groupings.

Visualizing higher-layer features of a deep network

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Feature Visualization

Segregation of form, color, and stereopsis in primate area 18

Representation of Angles Embedded within Contour Stimuli in Area V2 of Macaque Monkeys


Multimodal Neurons in Artificial Neural Networks

Gabriel Goh
Nick Cammarata †
Chelsea Voss †
Shan Carter
Michael Petrov
Ludwig Schubert
Alec Radford
Chris Olah

Summary: We report the existence of multimodal neurons in artificial neural networks, similar to those found in the human brain.

Invariant visual representation by single neurons in the human brain

Learning Transferable Visual Models From Natural Language Supervision

Deep Residual Learning for Image Recognition

Visualizing and understanding recurrent networks

Object detectors emerge in deep scene cnns

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Geographical evaluation of word embeddings

Using Artificial Intelligence to Augment Human Intelligence

Visualizing Representations: Deep Learning and Human Beings

Intriguing properties of neural networks

Feature Visualization

Global Weighted Average Pooling Bridges Pixel-level Localization and Image-level Classification

Adversarial Patch

Synthesizing Robust Adversarial Examples

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space