Exploring Bayesian Optimization

 Many modern machine learning algorithms have a large number of

hyperparameters. To effectively use these algorithms, we need to pick

good hyperparameter values.

 In this article, we talk about Bayesian Optimization, a suite of

techniques often used to tune hyperparameters. More generally, Bayesian

Optimization can be used to optimize any black-box function.

Mining Gold!

============

 For now, we assume that the gold is distributed about a line. We

want to find the location along this line with the maximum gold while

only drilling a few times (as drilling is expensive).

![](Exploring%20Bayesian%20Optimization_files/GT.svg)

 Initially, we have no idea about the gold distribution. We can

learn the gold distribution by drilling at different locations. However,

 We now discuss two common objectives for the gold mining problem.

* **Problem 1: Best Estimate of Gold Distribution (Active Learning)**

 **Active Learning**.

* **Problem 2: Location of Maximum Gold (Bayesian Optimization)**

 In this problem, we want to find the location of the maximum

gold content. We, again, can not drill at every location. Instead, we

 **Bayesian Optimization**.

 We will soon see how these two problems are related, but not the same.

Active Learning

---------------

 For many machine learning problems, unlabeled data is readily

available. However, labeling (or querying) is often expensive. As an

example, for a speech-to-text task, the annotation requires expert(s) to

 label words and sentences manually. Similarly, in our gold mining

problem, drilling (akin to labeling) is expensive.

 Active learning minimizes labeling costs while maximizing modeling

 accuracy. While there are various methods in active learning

literature, we look at **uncertainty reduction**. This

method proposes labeling the point whose model uncertainty is the

highest. Often, the variance acts as a measure of uncertainty.

 for the values our function takes elsewhere. This surrogate should be

flexible enough to model the true function. Using a Gaussian Process

(GP) is a common choice, both because of its flexibility and its ability

 to give us uncertainty estimates

 Gaussian Process supports setting of priors by using specific

kernels and mean functions. One might want to look at this excellent

Distill article on Gaussian Processes to learn more.

 .

 .

![](Exploring%20Bayesian%20Optimization_files/prior2posterior.png)

 Each new data point updates our surrogate model, moving it closer

to the ground truth. The black line and the grey shaded region indicate

 However, we want to minimize the number of evaluations. Thus, we

should choose the next query point "smartly" using active learning.

Although there are many ways to pick smart points, we will be picking

the most uncertain one.

 This gives us the following procedure for Active Learning:

2. Train on the new training set

3. Go to #1 till convergence or budget elapsed

![](Exploring%20Bayesian%20Optimization_files/0_004.png)

The visualization shows that one can estimate the true

distribution in a few iterations. Furthermore, the most uncertain

positions are often the farthest points from the current evaluation

Bayesian Optimization

---------------------

 In the previous section, we picked points in order to determine an

 accurate model of the gold content. But what if our goal is simply to

find the location of maximum gold content? Of course, we could do active

 learning to estimate the true function accurately and then find its

maximum. But that seems pretty wasteful — why should we use evaluations

improving our estimates of regions where the function expects low gold

content when we only care about the maximum?

 This is the core question in Bayesian Optimization: "Based on what

 we know so far, which point should we evaluate next?" Remember that

evaluating each point is expensive, so we want to pick carefully! In the

 active learning case, we picked the most uncertain point, exploring the

 function. But in Bayesian Optimization, we need to balance exploring

uncertain regions, which might unexpectedly have high gold content,

against focusing on regions we already know have higher gold content (a

kind of exploitation).

 We make this decision with something called an acquisition

function. Acquisition functions are heuristics for how desirable it is

 This brings us to how Bayesian Optimization works. At every step,

we determine what the best point to evaluate next is according to the

acquisition function by optimizing it. We then update our model and

repeat this process to determine the next point to evaluate.

 You may be wondering what's "Bayesian" about Bayesian Optimization

 if we're just optimizing these acquisition functions. Well, at every

step we maintain a model describing our estimates and uncertainty at

### Formalizing Bayesian Optimization

 We present the general constraints in Bayesian Optimization and

 * [Youtube talk](https://www.youtube.com/watch?v=c4KKvyWW_Xk),

.

| General Constraints | Constraints in Gold Mining example |

| --- | --- |

| fff's feasible set AAA is simple,

| fff is continuous but lacks special structure,

| fff is derivative-free:

| fff is expensive to evaluate:

 the number of times we can evaluate it

 is severely limited. | Drilling is costly. |

it is easy to incorporate normally distributed noise for GP regression). |

 To solve this problem, we will follow the following algorithm:

### Acquisition Functions

#### Probability of Improvement (PI)

xt+1=argmax(αPI(x))=argmax(P(f(x)≥(f(x+)+ϵ)))

 x\_{t+1} = argmax(lpha\_{PI}(x)) = argmax(P(f(x) \geq (f(x^+) +\epsilon)))

 xt+1=argmax(αPI(x))=argmax(P(f(x)≥(f(x+)+ϵ)))

xt+1=argmax(αPI(x))=argmax(P(f(x)≥(f(x+)+ϵ)))

 egin{aligned}

 x\_{t+1} & = argmax(lpha\_{PI}(x))\

 & = argmax(P(f(x) \geq (f(x^+) +\epsilon)))

 \end{aligned}

 xt+1=argmax(αPI(x))=argmax(P(f(x)≥(f(x+)+ϵ)))

 where,

* P(⋅)P(\cdot)P(⋅) indicates probability

* ϵ\epsilonϵ is a small positive number

 Looking closely, we are just finding the upper-tail probability

(or the CDF) of the surrogate posterior. Moreover, if we are using a GP

as a surrogate the expression above converts to,

 where,

* Φ(⋅)\Phi(\cdot)Φ(⋅) indicates the CDF

 The violet region shows the probability density at each point. The grey

 regions show the probability density below the current max. The "area"

of the violet region at each point represents the "probability of

improvement over current maximum". The next point to evaluate via the PI

 criteria (shown in dashed blue line) is x=6x = 6x=6.

![](Exploring%20Bayesian%20Optimization_files/density_pi.png)

##### Intuition behind ϵ\epsilonϵ in PI

 PI uses ϵ\epsilonϵ to strike a balance between exploration and exploitation.

![](Exploring%20Bayesian%20Optimization_files/0_009.png)

 This observation also shows that we do not need to construct an

accurate estimate of the black-box function to find its maximum.

![](Exploring%20Bayesian%20Optimization_files/0_013.png)

 What happens if we increase ϵ\epsilonϵ a bit more?

![](Exploring%20Bayesian%20Optimization_files/0_012.png)

 We see that we made things worse! Our model now uses ϵ=3\epsilon = 3ϵ=3,

 and we are unable to exploit when we land near the global maximum.

Moreover, with high exploration, the setting becomes similar to active

learning.

#### Expected Improvement (EI)

 The idea is fairly simple — choose the next query point as the one

xt+1=argminxE(∣∣ht+1(x)−f(x⋆)∣∣ ∣ Dt)

 xt+1=argminxE(∣∣ht+1(x)−f(x⋆)∣∣ ∣ Dt)

 In essence, we are trying to select the point that minimizes the

distance to the objective evaluated at the maximum. Unfortunately, we do

 not know the ground truth function, fff. Mockus proposed

 the following acquisition function to overcome the issue.

xt+1=argmaxxE(max{0, ht+1(x)−f(x+)} ∣ Dt)

 xt+1=argmaxxE(max{0, ht+1(x)−f(x+)} ∣ Dt)

xt+1= argmaxxE(max{0, ht+1(x)−f(x+)} ∣ Dt)

 egin{aligned}

 x\_{t+1} = \ & argmax\_x \mathbb{E} \

 & \left( {max} \{ 0, \ h\_{t+1}(x) - f(x^+) \} \ | \ \mathcal{D}\_t
ight)

 \end{aligned}

 xt+1= argmaxxE(max{0, ht+1(x)−f(x+)} ∣ Dt)

EI(x)={(μt(x)−f(x+)−ϵ)Φ(Z)+σt(x)ϕ(Z),if σt(x)>00,if σt(x)=0

 EI(x)=

 egin{cases}

 0, & 	ext{if}\ \sigma\_t(x) = 0

 \end{cases}

 EI(x)={(μt(x)−f(x+)−ϵ)Φ(Z)+σt(x)ϕ(Z),0,if σt(x)>0if σt(x)=0

EI(x)={[(μt(x)−f(x+)−ϵ) σt(x)>0∗Φ(Z)]+σt(x)ϕ(Z),0, σt(x)=0

 EI(x)= egin{cases}

 [(\mu\_t(x) - f(x^+) - \epsilon) & \ \sigma\_t(x) > 0 \

 \quad \* \Phi(Z)] + \sigma\_t(x)\phi(Z),\

 0, & \ \sigma\_t(x) = 0

 \end{cases}

 EI(x)=⎩⎪⎨⎪⎧[(μt(x)−f(x+)−ϵ)∗Φ(Z)]+σt(x)ϕ(Z),0, σt(x)>0 σt(x)=0

 where Φ(⋅)\Phi(\cdot)Φ(⋅) indicates CDF and ϕ(⋅)\phi(\cdot)ϕ(⋅) indicates pdf.

![](Exploring%20Bayesian%20Optimization_files/0_002.png)

 We now increase ϵ\epsilonϵ to explore more.

![](Exploring%20Bayesian%20Optimization_files/0_010.png)

 As we expected, increasing the value to ϵ=0.3\epsilon = 0.3ϵ=0.3

 makes the acquisition function explore more. Compared to the earlier

evaluations, we see less exploitation. We see that it evaluates only two

 points near the global maxima.

 Let us increase ϵ\epsilonϵ even more.

![](Exploring%20Bayesian%20Optimization_files/0_005.png)

#### PI vs Ei

![](Exploring%20Bayesian%20Optimization_files/0.svg)

 dot is a point in the search space. Additionally, the training set used

### Thompson Sampling

 Another common acquisition function is Thompson Sampling .

 At every step, we sample a function from the surrogate's posterior and

optimize it. For example, in the case of gold mining, we would sample a

plausible distribution of the gold given the evidence and evaluate

(drill) wherever it peaks.

 Below we have an image showing three sampled functions from the

learned surrogate posterior for our gold mining problem. The training

![](Exploring%20Bayesian%20Optimization_files/thompson.svg)

 We can understand the intuition behind Thompson sampling by two observations:

* Locations with high uncertainty (σ(x) \sigma(x) σ(x))

 will show a large variance in the functional values sampled from the

surrogate posterior. Thus, there is a non-trivial probability that a

sample can take high value in a highly uncertain region. Optimizing such

 samples can aid **exploration**.

* The sampled functions must pass through the current max

value, as there is no uncertainty at the evaluated locations. Thus,

![](Exploring%20Bayesian%20Optimization_files/0.png)

The visualization above uses Thompson sampling for optimization. Again,

we can reach the global optimum in relatively few iterations.

### Random

 We have been using intelligent acquisition functions until now.

 We can create a random acquisition function by sampling xxx

 randomly.

![](Exploring%20Bayesian%20Optimization_files/0_014.png)

 The visualization above shows that the performance of the random

acquisition function is not that bad! However, if our optimization was

more complex (more dimensions), then the random acquisition might

perform poorly.

### Summary of Acquisition Functions

 Let us now summarize the core ideas associated with acquisition

functions: i) they are heuristics for evaluating the utility of a point;

 ii) they are a function of the surrogate posterior; iii) they combine

exploration and exploitation; and iv) they are inexpensive to evaluate.

#### Other Acquisition Functions

We have seen various acquisition functions until now. One

trivial way to come up with acquisition functions is to have a

explore/exploit combination.

###  Upper Confidence Bound (UCB)

 One such trivial acquisition function that combines the

exploration/exploitation tradeoff is a linear combination of the mean

and uncertainty of our surrogate model. The model mean signifies

exploitation (of our model's knowledge) and model uncertainty signifies

exploration (due to our model's lack of observations).

 α(x)=μ(x)+λ×σ(x)lpha(x) = \mu(x) + \lambda 	imes \sigma(x)α(x)=μ(x)+λ×σ(x)

 The intuition behind the UCB acquisition function is weighing

of the importance between the surrogate's mean vs. the surrogate's

 We can further form acquisition functions by combining the

existing acquisition functions though the physical interpretability of

such combinations might not be so straightforward. One reason we might

want to combine two methods is to overcome the limitations of the

individual methods.

 One such combination can be a linear combination of PI and EI.

 We know PI focuses on the probability of improvement, whereas

EI focuses on the expected improvement. Such a combination could help in

 having a tradeoff between the two based on the value of λ\lambdaλ.

### Gaussian Process Upper Confidence Bound (GP-UCB)

 a−ba -

 GP-UCB's formulation is given by:

αGP−UCB(x)=μt(x)+βtσt(x)

 lpha\_{GP-UCB}(x) = \mu\_t(x) + \sqrt{eta\_t}\sigma\_t(x)

 αGP−UCB(x)=μt(x)+βt

σt(x)

 Where ttt is the timestep.

### Comparison

 slides from Nando De Freitas. We have used the

optimum hyperparameters for each acquisition function.

 We ran the random acquisition function several times with

different seeds and plotted the mean gold sensed at every iteration.

![](Exploring%20Bayesian%20Optimization_files/comp.svg)

 strategy grows slowly. In comparison, the other acquisition functions

can find a good solution in a small number of iterations. In fact, most

acquisition functions reach fairly close to the global maxima in as few

as three iterations.

Hyperparameter Tuning

---------------------

Before we talk about Bayesian optimization for hyperparameter tuning,

 we will quickly differentiate between hyperparameters and parameters:

hyperparameters are set before learning and the parameters are learned

from the data. To illustrate the difference, we take the example of

Ridge regression.

θ^ridge=argminθ ∈ Rp∑i=1n(yi−xiTθ)2+λ∑j=1pθj2

 \hat{	heta}\_{ridge} = argmin\_{	heta\ \in \ \mathbb{R}^p}

\sum\limits\_{i=1}^{n} \left(y\_i - x\_i^T	heta
ight)^2 + \lambda

\sum\limits\_{j=1}^{p} 	heta^2\_j

 θ^ridge=argminθ ∈ Rpi=1∑n(yi−xiTθ)2+λj=1∑pθj2

θ^ridge=argminθ ∈ Rp∑i=1n(yi−xiTθ)2+λ∑j=1pθj2

 egin{aligned}

 \hat{	heta}\_{ridge} = & argmin\_{	heta\ \in \ \mathbb{R}^p}

 \sum\limits\_{i=1}^{n} \left(y\_i - x\_i^T	heta
ight)^2 \

 & + \lambda \sum\limits\_{j=1}^{p} 	heta^2\_j

 \end{aligned}

 θ^ridge=argminθ ∈ Rpi=1∑n(yi−xiTθ)2+λj=1∑pθj2

 If we solve the above regression problem via gradient descent

optimization, we further introduce another optimization parameter, the

learning rate αlphaα.

When training a model is not expensive and time-consuming, we

can do a grid search to find the optimum hyperparameters. However, grid

search is not feasible if function evaluations are costly, as in the

case of a large neural network that takes days to train. Further, grid

search scales poorly in terms of the number of hyperparameters.

### Example 1 — Support Vector Machine (SVM)

In this example, we use an SVM to classify on sklearn's moons

dataset and use Bayesian Optimization to optimize SVM hyperparameters.

 Let us have a look at the dataset now, which has two classes and two features.

![](Exploring%20Bayesian%20Optimization_files/moons.svg)

 the surface plots you see for the Ground Truth Accuracies below were

calculated for each possible hyperparameter for showcasing purposes

only. We do not have these values in real applications.

![](Exploring%20Bayesian%20Optimization_files/0_003.png)

![](Exploring%20Bayesian%20Optimization_files/0_006.png)

### Comparison

![](Exploring%20Bayesian%20Optimization_files/comp3d.svg)

 method seemed to perform much better initially, but it could not reach

the global optimum, whereas Bayesian Optimization was able to get fairly

 close. The initial subpar performance of Bayesian Optimization can be

attributed to the initial exploration.

#### Other Examples

### Example 2 — Random Forest

 Using Bayesian Optimization in a Random Forest Classifier.

 We will continue now to train a Random Forest on the moons

dataset we had used previously to learn the Support Vector Machine

model. The primary hyperparameters of Random Forests we would like to

optimize our accuracy are the  **number** of

 We will be again using Gaussian Processes with Matern kernel

to estimate and predict the accuracy function over the two

hyperparameters.

![](Exploring%20Bayesian%20Optimization_files/0_007.png)

![](Exploring%20Bayesian%20Optimization_files/0_015.png)

![](Exploring%20Bayesian%20Optimization_files/0_008.png)

![](Exploring%20Bayesian%20Optimization_files/0_011.png)

Let us now use the Random acquisition function.

![](Exploring%20Bayesian%20Optimization_files/RFcomp3d.svg)

 The optimization strategies seemed to struggle in this

example. This can be attributed to the non-smooth ground truth. This

shows that the effectiveness of Bayesian Optimization depends on the

surrogate's efficiency to model the actual black-box function. It is

interesting to notice that the Bayesian Optimization framework still

beats the *random* strategy using various acquisition functions.

### Example 3 — Neural Networks

 Let us take this example to get an idea of how to apply

 which also provides us support for optimizing function with a search

space of categorical, integral, and real variables. We will not be

plotting the ground truth here, as it is extremely costly to do so.

Below are some code snippets that show the ease of using Bayesian

Optimization packages for hyperparameter tuning.

 The code initially declares a search space for the

optimization problem. We limit the search space to be the following:

* batch\_size — This hyperparameter sets the number of

training examples to combine to find the gradients for a single step in

gradient descent.

* learning rate — This hyperparameter sets the stepsize with

 which we will perform gradient descent in the neural network.

* activation — We will have one categorical variable, i.e.

the activation to apply to our neural network layers. This variable can

take on values in the set {relu, sigmoid}\{ relu, \ sigmoid \}{relu, sigmoid}.

 log\_batch\_size = Integer(

 low=2,

 high=7,

 name='log\_batch\_size'

 )

 lr = Real(

 low=1e-6,

 high=1e0,

 prior='log-uniform',

 name='lr'

 )

 activation = Categorical(

 categories=['relu', 'sigmoid'],

 name='activation'

 )

 dimensions = [

 dim\_num\_batch\_size\_to\_base,

 dim\_learning\_rate,

 dim\_activation

 ]

 # initial parameters (1st point)

 default\_parameters =

 [4, 1e-1, 'relu']

 # bayesian optimization

 search\_result = gp\_minimize(

 func=train,

 dimensions=dimensions,

 acq\_func='EI', # Expctd Imprv.

 n\_calls=11,

 x0=default\_parameters

 )

![](Exploring%20Bayesian%20Optimization_files/conv.svg)

 Looking at the above example, we can see that incorporating

Bayesian Optimization is not difficult and can save a lot of time.

Optimizing to get an accuracy of nearly one in around seven iterations

 Let us get the numbers into perspective. If we had run this

 iterations. Whereas Bayesian Optimization only took seven iterations.

Each iteration took around fifteen minutes; this sets the time required

for the grid search to complete around seventeen hours!

Conclusion and Summary

======================

 In this article, we looked at Bayesian Optimization for

optimizing a black-box function. Bayesian Optimization is well suited

when the function evaluations are expensive, making grid or exhaustive

search impractical. We looked at the key components of Bayesian

Optimization. First, we looked at the notion of using a surrogate

function (with a prior over the space of objective functions) to model

our black-box function. Next, we looked at the "Bayes" in Bayesian

Optimization — the function evaluations are used as data to obtain the

surrogate posterior. We look at acquisition functions, which are

functions of the surrogate posterior and are optimized sequentially.

This new sequential optimization is in-expensive and thus of utility of

us. We also looked at a few acquisition functions and showed how these

different functions balance exploration and exploitation. Finally, we

looked at some practical examples of Bayesian Optimization for

optimizing hyper-parameters for machine learning models.

A statistical approach to some basic mine valuation problems on the Witwatersrand

Active Learning Literature Survey

Active learning: theory and applications

Taking the Human Out of the Loop: A Review of Bayesian Optimization

A
 Tutorial on Bayesian Optimization of Expensive Cost Functions, with
Application to Active User Modeling and Hierarchical Reinforcement
Learning

A Visual Exploration of Gaussian Processes

Gaussian Processes in Machine Learning

Bayesian approach to global optimization and application to multiobjective and constrained problems

On The Likelihood That One Unknown Probability Exceeds Another In View Of The Evidence Of Two Samples

Using Confidence Bounds for Exploitation-Exploration Trade-Offs

Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design

Practical Bayesian Optimization of Machine Learning Algorithms

Algorithms for Hyper-Parameter Optimization

Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures

Scikit-learn: Machine Learning in {P}ython

Bayesian Optimization with Gradients

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

Safe Exploration for Optimization with Gaussian Processes

Scalable Bayesian Optimization Using Deep Neural Networks

Portfolio Allocation for Bayesian Optimization

Bayesian Optimization for Sensor Set Selection

Constrained Bayesian Optimization with Noisy Experiments

Parallel Bayesian Global Optimization of Expensive Functions


Multimodal Neurons in Artificial Neural Networks

Gabriel Goh
Nick Cammarata †
Chelsea Voss †
Shan Carter
Michael Petrov
Ludwig Schubert
Alec Radford
Chris Olah

Abstract: In  2005, a letter published in Nature described human neurons responding  to specific people, such as Jennifer Aniston or Halle Berry .  The exciting thing wasn’t just that they selected for particular  people, but that they did so regardless of whether they were shown  photographs, drawings, or even images of the person’s name. The neurons  were multimodal. As the lead author would put it: "You are looking at  the far end of the transformation from metric, visual shapes to  conceptual… information." Quiroga's full quote, from New Scientist  reads: "I think that’s the excitement to these results. You are looking  at the far end of the transformation from metric, visual shapes to  conceptual memory-related information. It is that transformation that  underlies our ability to understand the world. It’s not enough to see  something familiar and match it. It’s the fact that you plug visual  information into the rich tapestry of memory that brings it to life." We  elided the portion discussing memory since it was less relevant.

#### Contents

* [Emotion Neurons](#emotion-neurons)

* [Region Neurons](#region-neurons)

* [Feature Properties](#feature-properties)

* [Understanding Language](#understanding-language)

* [Emotion Composition](#emotional-intelligence)

* [Faceted Feature Visualization](#faceted-feature-visualization)

* [CLIP Training](#clip)

In

 2005, a letter published in Nature described human neurons responding

to specific people, such as Jennifer Aniston or Halle Berry .

 The exciting thing wasn't just that they selected for particular

people, but that they did so regardless of whether they were shown

photographs, drawings, or even images of the person's name. The neurons

were multimodal. As the lead author would put it: "You are looking at

the far end of the transformation from metric, visual shapes to

 reads: "I think that's the excitement to these results. You are looking

 at the far end of the transformation from metric, visual shapes to

conceptual memory-related information. It is that transformation that

underlies our ability to understand the world. It's not enough to see

something familiar and match it. It's the fact that you plug visual

information into the rich tapestry of memory that brings it to life." We

 elided the portion discussing memory since it was less relevant.

We

 report the existence of similar multimodal neurons in artificial neural

 networks. This includes neurons selecting for prominent public figures

 important to note that the vast majority of people these models

recognize don't have a specific neuron, but instead are represented by a

 combination of neurons. Often, the contributing neurons are

conceptually related. For example, we found a Donald Trump neuron which

fires (albeit more weakly) for Mike Pence, contributing to representing

him.Some of the

neurons we found seem strikingly similar to those described in

neuroscience. A Donald Trump neuron we found might be seen as similar to

 And although we don't find an exact Jennifer Aniston neuron, we do find

 a neuron for the TV show "Friends" which fires for her.

Like the biological multimodal neurons, these artificial neurons respond

 to the same subject in photographs, drawings, and images of their name:

 neurons only scratch the surface of the highly abstract neurons we've

found. Some neurons seem like topics out of a kindergarten curriculum:

weather, seasons, letters, counting, or primary colors. All of these

features, even the trivial-seeming ones, have rich multimodality, such

We find these multimodal neurons in the recent CLIP models ,

 although it's possible similar undiscovered multimodal neurons may

 authors also kindly shared an alternative version from earlier

experiments, where the training objective was an autoregressive language

 modelling objective, instead of a contrastive objective. The features

seem pretty similar. There are several CLIP models of

varying sizes; we find multimodal neurons in all of them, but focus on

studying the mid-sized RN50-x4 model. We

found it challenging to make feature visualization work on the largest

CLIP models. The reasons why remain unclear. See faceted feature

 for more detailed discussion of CLIP's architecture and performance.

Our analysis will focus on CLIP's vision side, so when we talk about a

multimodal neuron responding to text we mean the model "reading" text in

 images. The alignment with the text side

of the model might be seen as an additional form of multimodality,

perhaps analogous to a human neuron responding to hearing a word rather

than seeing it (see Quiroga's later work). But since that is an expected

 result of the training objective, it seems less interesting.

CLIP's

 abstract visual features might be seen as the natural result of

aligning vision and text. We expect word embeddings (and language models

 generally) to learn abstract "topic" features .

 Either the side of the model which processes captions (the "language

side") needs to give up those features, or its counterpart, the "vision

side", needs to build visual analogues. Many

 researchers are interested in "grounding" language models by training

them on tasks involving another domain, in the hope of them learning a

more real world understanding of language. The abstract features we find

 in vision models can be seen as a kind of "inverse grounding": vision

taking on more abstract features by connection to language. This

 includes some of the classic kinds of bias we see in word embeddings,

such as a "terrorism"/"Islam" neuron, or an "Immigration"/"Mexico"

neuron. See discussion in the [region neurons section](#region-neurons).

 But even if these features seem natural in retrospect, they are

qualitatively different from neurons previously studied in vision models

 (eg. ).

 They also have real world implications: these models are vulnerable to a

 kind of "typographic attack" where adding adversarial text to images

can cause them to be systematically misclassified.

 A typographic attack.

---

A Guided Tour of Neuron Families

--------------------------------

What

 features exist in CLIP models? In this section, we examine neurons

found in the final convolutional layer of the vision side across four

models. A majority of these neurons seem to be interpretable.We

 checked a sample of 50 neurons from this layer and classified them as

interpretable, polysemantic, or uninterpretable. We found that 76% of

the sampled neurons were interpretable. (As a 95% confidence interval,

that's between 64% and 88%.) A further 18% were polysemantic but with

interpretable facets, and 6% were as yet uninterpretable.

Each layer consists of thousands of neurons, so for our preliminary

analysis we looked at feature visualizations, the dataset examples that

most activated the neuron, and the English words which most activated

the neuron when rastered as images. This revealed an incredible

diversity of features, a sample of which we share below:

 neurons respond to content associated with with a geographic region,

with neurons ranging in scope from entire hemispheres to individual

 this, we mean both that it responds to people presenting as this

gender, as well as that it responds to concepts associated with that

 neurons detect features that an image might contain, whether it's

normal object recognition or detection of more exotic features such as

 despite being able to "read" words and map them to semantic features,

the model keeps a handful of more typographic features in its high-level

 representations. Like a child spelling out a word they don't know, we

 many of the neurons in the model contribute to recognizing an

incredible diversity of abstract concepts that cannot be cleanly

 neurons respond to any visual information that contextualizes the image

 in a particular time – for some it's a season, for others it's a

 This diagram presents selected neurons from the final layer of four

CLIP models, hand organized into "families" of similar neurons. Each

neuron is represented by a feature visualization (selected from regular

or [faceted feature visualization](#faceted-feature-visualization)

 to best illustrate the neuron) with human-chosen labels to help quickly

 provide a sense of each neuron. Labels were picked after looking at

hundreds of stimuli that activate the neuron, in addition to feature

visualizations.

 You can click on any neuron to open it up in OpenAI Microscope to

see feature visualizations, dataset examples that maximally activate the

 neuron, and more.

These neurons don't just select for a single

object. They also fire (more weakly) for associated stimuli, such as a

Barack Obama neuron firing for Michelle Obama or a morning neuron firing

 for images of breakfast. They also tend to be maximally inhibited by

stimuli which could be seen, in a very abstract way, as their opposite. Some

 neurons seem less abstract. For example, typographic features like the

"-ing" detector seem to roughly fire based on how far a string is away

in Levenshtein distance. Although, even these show remarkable

generalization, such as responding to different font sizes and rotated

text.

How should we think of these neurons? From an

interpretability perspective, these neurons can be seen as extreme

examples of "multi-faceted neurons" which respond to multiple distinct

 is a classic example in neuroscience of a hypothetical neuron that

responds in a highly specific way to some complex concept or stimulus –

 but this framing might encourage people to overinterpret these

artificial neurons. Instead, the authors generally think of these

neurons as being something like the visual version of a topic feature,

activating for features we might expect to be similar in a word

embedding.

Many of these neurons deal with sensitive topics, from

political figures to emotions. Some neurons explicitly represent or are

closely related to protected characteristics: age, gender, race,

religion, sexual orientation,There's a

neuron we conceptualize as an LGBT neuron, which responds to the Pride

flag, rainbows, and images of words like "LGBT". Previous work (Wang

& Kosinski) has suggested that neural networks might be able to

determine sexual orientation from facial structure. This work has since

been thoroughly rebutted and we wish to emphasize that we see no

 neurons related to age and gender, see "person trait neurons." Region

neurons seem closely related to race and national origin, responding to

ethnicities associated with given regions of the world. For sexual

 These neurons may reflect prejudices in the "associated" stimuli they

respond to, or be used downstream to implement biased behavior. There

are also a small number of people detectors for individuals who have

committed crimes against humanity, and a "toxic" neuron which responds

to hate speech and sexual content. Having neurons corresponding to

sensitive topics doesn't necessarily mean a network will be prejudiced.

You could even imagine explicit representations helping in some cases:

the toxic neuron might help the model match hateful images with captions

 that refute them. But they are a warning sign for a wide range of

possible biases, and studying them may help us find potential biases

which might be less on our radar.Examples

 of bias in AI models, and work drawing attention to it, has helped the

research community to become somewhat "alert" to potential bias with

regards to gender and race. However, CLIP could easily have biases which

 we are less alert to, such as biased behavior towards parents when

there's a child's drawing in the background.

CLIP

contains a large number of interesting neurons. To allow detailed

examination we'll focus on three of the "neuron families" shown above:

people neurons, emotion neurons, and region neurons. We invite you to

explore others in Microscope.

### Person Neurons

This

 section will discuss neurons representing present and historical

figures. Our discussion is intended to be descriptive and frank about

what the model learned from the internet data it was trained on, and is

not endorsement of associations it makes or of the figures discussed,

who include political figures and people who committed crimes against

humanity. This content may be disturbing to some readers.To

caption images on the Internet, humans rely on cultural knowledge. If

you try captioning the popular images of a foreign place, you'll quickly

 find your object and scene recognition skills aren't enough. You can't

caption photos at a stadium without recognizing the sport, and you may

even need to know specific players to get the caption right. Pictures of

 politicians and celebrities speaking are even more difficult to caption

 if you don't know who's talking and what they talk about, and these are

 some of the most popular pictures on the Internet. Some public figures

elicit strong reactions, which may influence online discussion and

captions regardless of other content.

With this in mind, perhaps

it's unsurprising that the model invests significant capacity in

representing specific public and historical figures — especially those

 detects Christian symbols like crosses and crowns of thorns, paintings

of Jesus, his written name, and feature visualization shows him as a

 recognizes the masked hero and knows his secret identity, Peter Parker.

 It also responds to images, text, and drawings of heroes and villians

 learns to detect his face and body, symbols of the Nazi party, relevant

 historical documents, and other loosely related concepts like German

food. Feature visualization shows swastikas and Hitler seemingly doing a

 Nazi salute.

Which

 people the model develops dedicated neurons for is stochastic, but

seems correlated with the person's prevalence across the datasetThe

 model's dataset was collected in 2019 and likely emphasizes content

from around that time. In the case of the Donald Trump neuron, it seems

likely there would have also been a Hillary Clinton neuron if data had

 and the intensity with which people respond to them. The one person

we've found in every CLIP model is Donald Trump. It strongly responds to

 images of him across a wide variety of settings, including effigies and

 caricatures in many artistic mediums, as well as more weakly activating

 for people he's worked closely with like Mike Pence and Steve Bannon.

It also responds to his political symbols and messaging (eg. "The Wall"

and "Make America Great Again" hats). On the other hand, it most

\*negatively\* activates to musicians like Nicky Minaj and Eminem, video

games like Fortnite, civil rights activists like Martin Luther King Jr.,

 and LGBT symbols like rainbow flags.

To understand the Trump neuron in more depth, we collected about 650

images that cause it to fire different amounts and labeled them by hand

into categories we created. This lets us estimate the conditional

 for details. As the black / LGBT category contains only a few images,

since they don't occur frequently in the dataset, we validated they

cause negative activations with a futher experimentAs

 we were labeling images for the conditional probability plot in Figure 2

 we were surprised that images related to black and gay rights

consistently caused strong negative activations. However, since there

were four images in that category, we decided to do a follow-up

experiment on more images.

We searched Google Images for the

terms "black rights" and "gay rights" and took ten top images for each

term without looking at their activations. Then we validated these

images reliably cause the Trump neuron to fire in the range of roughly

negative ~3-6 standard deviations from zero. The images that cause less

strong negative activations near -3 standard deviations tend to have

broad symbols such as an image of several black teenagers raising their

arm and fist that causes a -2.5 standard deviations. Conversely, images

of more easy to recognize and specific symbols such as rainbow flags or

photos of Martin Luther King Jr consistently cause activations of at

least -4 standard deviations. In Figure 3 we also show activations

related to photos of Martin Luther King Jr..

 Across

 all categories, we see that higher activations of the Trump neuron are

highly selective, as more than 90% of the images with a standard

deviation greater than 30 are related to Donald Trump.While

 labeling images for the previous experiment it became clear the neuron

activates different amounts for specific people. We can study this more

by searching the Internet for pictures of specific people and measuring

how the images of each person makes the neuron fire.

To see how the Trump neuron responds to different individuals, we

searched the query "X giving a speech at a microphone" for various

individuals on Google Images. We cleaned the data by hand, excluding

photos that are not clear photos of the individual's face. The bar

length for each individual shows the median activation of the person's

photos in standard deviations of the neuron over the dataset, and the

range over the bar shows the standard deviation of the activations of

the person's photos.Presumably, person

neurons also exist in other models, such as facial recognition models.

What makes these neurons unique is that they respond to the person

across modalities and associations, situating them in a cultural

context. In particular, we're struck by how the neuron's response tracks

 an informal intuition with how associated people are. In this sense,

person neurons can be thought of as a landscape of person-associations,

with the person themself as simply the tallest peak.

### Emotion Neurons

This

 section will discuss neurons representing emotions, and a neuron for

"mental illness." Our discussion is intended to be descriptive and frank

 about what the model learned from the internet data it was trained on

and is not endorsement. This content may be disturbing to some readers.Since

 a small change in someone's expression can radically change the meaning

 of a picture, emotional content is essential to the task of captioning.

 The model dedicates dozens of neurons to this task, each representing a

 different emotion.

These emotion neurons don't just respond to

facial expressions associated with an emotion -- they're flexible,

responding to body language and facial expressions in humans and

 activates even when the majority of the face is obscured. It responds

to slang like "OMG!" and "WTF", and text feature visualization produces

similar words of shock and surprise. There are even some emotion neurons

 in discussion of emotions because (1) they are sometimes included in

emotion wheels, (2) they seem to play a role in captioning emotions and

feelings, and (3) being more inclusive in our discussion allow us to

explore more of the model. Of course, these neurons simply

respond to cues associated with an emotion and don't necessarily

correspond to the mental state of subjects in an image.In

 addition to CLIP neurons potentially incorrectly recognizing cues, they

 cues themselves don't necessarily reflect people's mental states. For

example, facial expressions don't reliably correspond to someone

 addition to these emotion neurons, we also find which neurons respond

to an emotion as a secondary role, but mostly respond to something else.

 which primarily responds to jail and incarceration helps represent

emotions such as "persecuted." Similarly, a neuron that primarily

detects pornographic content seems to have a secondary function of

to see some of these different facets. In particular, the face facet

shows facial expressions corresponding to different emotions, such as

smiling, crying, or wide-eyed shock. Click on any neuron to open it in

Microscope to see more information, including dataset examples.While

 most emotion neurons seem to be very abstract, there are also some

neurons which simply respond to specific body and facial expressions,

 It activates most to the internet-born duckface expression and peace

signs, and we'll see later that both words show up in the maximally

corresponding captions.

One

 neuron that doesn't represent a single emotion but rather a high level

 This neuron activates when images contain words associated with

negative mental states (eg. "depression," "anxiety," "lonely,"

"stressed"), words associated with clinical mental health treatment

("psychology", "mental," "disorder", "therapy") or mental health

pejoratives ("insane," "psycho"). It also fires more weakly for images

of drugs, and for facial expressions that look sad or stressed, and for

the names of negative emotions.

 we wouldn't think of mental illness as a dimension of emotion. However,

 a couple things make this neuron important to frame in the emotion

context. First, in its low-mid range activations, it represents common

negative emotions like sadness. Secondly, words like "depressed" are

often colloquially used to describe non-clinical conditions. Finally,

we'll see in a later section that this neuron plays an important role in

 captioning emotions, composing with other emotion neurons to

differentiate "healthy" and "unhealthy" versions of an emotion.

To

 better understand this neuron we again estimated the conditional

probabilities of various categories by activation magnitude. The

strongest positive activations are concepts related to mental illness.

Conversely, the strongest negative activations correspond to activities

like exercise, sports, and music events.

 To understand the "mental illness neuron" in more depth, we collected

images that cause it to fire different amounts and labeled them by hand

into categories we created. This lets us estimate the conditional

 for details. During the labeling process we couldn't see how much it

made the neuron fire. We see that the strongest activations all belong

to labels corresponding to low-valence mental states. On the other hand,

 many images with a negative pre-ReLU activation are of scenes we may

typically consider high-valence, like photos with pets, travel photos,

and or pictures of sporting events.### Region Neurons

This

 section will discuss neurons representing regions of the world, and

indirectly ethnicity. The model's representations are learned from the

internet, and may reflect prejudices and stereotypes, sensitive regional

 situations, and colonialism. Our discussion is intended to be

descriptive and frank about what the model learned from the internet

data it was trained on, and is not endorsement of the model's

representations or associations. This content may be disturbing to some

readers.From local weather and food, to travel and immigration,

 to language and race: geography is an important implicit or explicit

context in a great deal of online discourse. Blizzards are more likely

 They respond to a wide variety of modalities and facets associated with

 a given region: country and city names, architecture, prominent public

figures, faces of the most common ethnicity, distinctive clothing,

wildlife, and local script (if not the Roman alphabet). If shown a world

 map, even without labels, these neurons fire selectively for the

relevant region on the map.Map responses

seem to be strongest around distinctive geographic landmarks, such as

the Gulf Of Carpentaria and Cape York Peninsula for Australia, or the

Gulf of Guinea for Africa.

 which responds to bears, moose, coniferous forest, and the entire

Northern third of a world map — down to sub-regions of countries, such

as the US West Coast.One interesting

property of the regional neuron "hierarchy" is that the parent neuron

often doesn't fire when a child is uniquely implicated. So while the

Europe neuron fires for the names of European cities, the general United

 States neuron generally does not, and instead lets neurons like the

West Coast neuron fire. See also another example of a neuron "hierarchy"

 Some region neurons seem to form more consistently than others. Which

neurons form doesn't seem to be fully explained by prevalence in the

 but not all models seem to have a UK neuron. Why is that? One intuition

 is that there's more variance in neurons when there's a natural

supercategory they can be grouped into. For example, when an individual

UK neuron doesn't exist, it seems to be folded into a Europe neuron. In

Africa, we sometimes see multiple different Africa neurons (in

particular a South/West Africa neuron and an East Africa neuron), while

other times there seems to be a single unified Africa neuron. In

contrast, Australia is perhaps less subdividable, since it's both a

continent and country.

Geographical Activation of Region Neurons  **Unlabeled map activations**:

 Spatial activations of neurons in response to unlabeled geographical

world map. Activations averaged over random crops. Note that neurons for

 Countries colored by activations of neurons in response to rastered

images of country names. Activations averaged over font sizes, max over

word positions.  **City name activations**:

 Cities colored by activations of neurons in response to rastered images

 of city names. Activations averaged over font sizes, max over word

positions.      Selected Region Neurons Most Activating Words

Hover on a neuron to isolate activations. Click to open in Microscope.

 This diagram contextualizes region neurons with a map.

 Each neuron is mapped to a hue, and then regions where it

activates are colored in that hue, with intensity proportional to

activiation. If multiple neurons of opposing hues fire, the region will

be colored in a desaturated gray.

 It can show their response

 to an unlabeled geographical map,

 to country names,

 and to city names.

 "large region neurons"

 (such as the "Northern Hemisphere" neuron)

 and at

 "secondarily regional neurons"

 "entrepreneurship"

 or

 may not. This means that visualizing behavior on a global map

underrepresents the sheer number of region neurons that exist in CLIP.

Using the top-activating English words as a heuristic, we estimate

around 4% of neurons are regional.To

estimate the fraction of neurons that are regional, we looked at what

fraction of each neuron's top-activating words (ie. words it responds to

 when rastered as images) were explicitly linked to geography, and used

this as a heuristic for whether a neuron was regional. To do this, we

created a list of geographic words consisting of continent / country /

 We found 2.5% (64) of RN50-x4 neurons had geographic words for all of

the five maximally activating words. This number varied between 2-4% in

other CLIP models. However, looking only at neurons for which all top

five words are explicitly geographic misses many region neurons which

respond strongly to words with implicit regional connotations (eg.

"hockey" for a Canada neuron, "volkswagen" for a German neuron, "palm"

for an equatorial neuron). We bucketed neurons by fraction of five most

activating words that are geographic, then estimated the fraction of

each bucket that were regional. With many neurons, the line was quite

blurry (should we include polysemantic neurons where one case is

regional? What about "secondarily regional neurons"?). For a relatively

conservative definition, this seems to get us about 4%, but with a more

liberal one you might get as high as 8%.

 caution is needed in interpreting these neurons as truly regional,

rather than spuriously weakly firing for part of a world map. Important

validations are that they fire for the same region on multiple different

 maps, and if they respond to words for countries or cities in that

region. These neurons don't have a region as the primary

focus, but have some kind of geographic information baked in, firing

 also find that the linear combination of neurons that respond to Russia

 on a map strongly responds to Pepe the frog, a symbol of white

nationalism in the United States allegedly promoted by Russia. Our

impression is that Russians probably wouldn't particularly see this as a

 symbol of Russia, suggesting it is more "Russia as understood by the

US."

#### Case Study: Africa Neurons

Despite these

 examples of neurons learning Americentric caricatures, there are some

areas where the model seems slightly more nuanced than one might fear,

especially given that CLIP was only trained on English language data.

For example, rather than blurring all of Africa into a monolithic

entity, the RN50-x4 model develops neurons for three regions within

Africa. This is significantly less detailed than its representation of

many Western countries, which sometimes have neurons for individual

countries or even sub-regions of countries, but was still striking to

us.It's important to keep in mind that

the model can represent many more things using combinations of neurons.

Where the model dedicates neurons may give us some sense of the level of

 nuance, but we shouldn't infer, for example, that it doesn't somehow

represent individual African countries. To

 contextualize this numerically, the model seems to dedicate ~4% of its

regional neurons to Africa, which accounts for ~20% of the world's

landmass, and ~15% of the world's population.

 early explorations it quickly became clear these neurons "know" more

about Africa than the authors. For example, one of the first feature

visualizations of the South African regional neuron drew the text

 Learning about a TV drama might not be the kind of deep insights one

might have envisioned, but it is a charming proof of concept.

We

 chose the East Africa neuron for more careful investigation, again

using a conditional probability plot. It fires most strongly for flags,

 activations tend to follow an exponential distribution in their tails, a

 point that was made to us by Brice Menard. This means that strong

activations are more common than you'd expect in a Gaussian (where the

tail decays at exp(-x^2)), but are much less common than weaker

activations. — have a significantly different distribution

and seems to be mostly about ethnicity. Perhaps this is because

ethnicity is implicit in all images of people, providing weak evidence

for a region, while features like flags are far less frequent, but

provide strong evidence when they do occur. This is the first neuron

we've studied closely with a distinct regime change between medium and

strong activations.

We labeled more than 400 images that causes a neuron that most strongly

responds to the word "Ghana" to fire at different levels of activation,

without access to how much each image caused the neuron to fire while

labeling. See [the appendix](#conditional-probability) for details.

 It fires most strongly for people of African descent as well as African

 words like country names. It's pre-ReLU activation is negative for

symbols associated with other countries, like the Tesla logo or British

flag, as well as people of non-African descent. Many of its strongest

negative activations are for weaponry such as military vehicles and

handguns. Ghana, the country name it responds to most strongly, has a

Global Peace Index rating higher than most African countries, and

perhaps it learns this anti-association.We

 also looked at the activations of the other two Africa neurons. We

suspect they have interesting differences beyond their detection of

different country names and flags — why else would the model dedicate

three neurons — but we lacked the cultural knowledge to appreciate their

 subtleties.

### Feature properties

So

 far, we've looked at particular neurons to give a sense of the kind of

features that exist in CLIP models. It's worth noting several properties

 that might be missed in the discussion of individual features:

**Image-Based Word Embedding:**

 Despite being a vision model, one can produce "image-based word

embeddings" with the visual CLIP model by rastering words into images

and then feeding these images into the model, and then subtracting off

the average over words. Like normal word embeddings, the nearest

Original

Word

Nearest Neighbors

Collobert embeddings

Nearest Neighbors

CLIP image-based embeddings

France

French, Francis, Paris, Les, Des, Sans, Le, Pairs, Notre, Et

Jesus

God, Sati, Christ, Satan, Indra, Vishnu, Ananda, Parvati, Grace

Christ, God, Bible, Gods, Praise, Christians, Lord, Christian, Gospel, Baptist

xbox

Amiga, Playstation, Msx, Ipod, Sega, Ps#, Hd, Dreamcast, Geforce, Capcom

*V(Img(*"King"*)) - V(Img(*"Man"*)) + V(Img(*"Woman"*)) = V(Img(*"Queen"*))*

 work in some cases if we mask non-semantic lexicographic neurons (eg.

"-ing" detectors). It seems likely that mixed arithmetic of words and

images should be possible.

**Limited Multilingual Behavior:**

Although CLIP's training data was filtered to be English, many features

 responds to images of English "Thank You", French "Merci", German

"Danke", and Spanish "Gracias," and also to English "Congratulations",

German "Gratulieren", Spanish "Felicidades", and Indonesian "Selamat".

As the example of Indonesian demonstrates, the model can recognize some

words from non Romance/Germanic languages. However, we were unable to

find any examples of the model mapping words in non-Latin script to

semantic meanings. It can recognize many scripts (Arabic, Chinese,

Japanese, etc) and will activate the corresponding regional neurons, but

 doesn't seem to be able to map words in those scripts to their

meanings.One interesting question is why

the model developed reading abilities in latin alphabet languages, but

not others. Was it because more data of that type slipped into the

training data, or (the more exciting possibility) because it's easier to

 learn a language from limited data if you already know the alphabet?

 The most striking examples are likely racial and religious bias. As

 which responds to images of words such as "Terrorism", "Attack",

"Horror", "Afraid", and also "Islam", "Allah", "Muslim". This isn't just

 an illusion from looking at a single neuron: the image-based word

embedding for "Terrorist" has a cosine similarity of 0.52 with

"Muslims", the highest value we observe for a word that doesn't include

"terror."By "image-based word embedding",

 we mean the activation for an image of that word, with the average

activation over images of 10,000 English words subtracted off. The

intuition is that this removes generic "black text on white background"

fatures. If one measures the cosine similarity between "Terrorism" and

"Muslim" without subtracting off the average, it's much higher at about

0.98, but that's because all values are shifted up due to sharing the

**Polysemanticity and Conjoined Neurons:**

 Our qualitative experience has been that individual neurons are more

interpretable than random directions; this mirrors observations made in

previous work.Although

 we've focused on neurons which seem to have a single clearly defined

concept they respond to, many CLIP neurons are "polysemantic" ,

 responding to multiple unrelated features. Unusually, polysemantic

neurons in CLIP often have suspicious links between the different

 The concepts in these neurons seem "conjoined", overlapping in a

superficial way in one facet, and then generalizing out in multiple

directions. We haven't ruled out the possibility that these are just

coincidences, given the large number of facets that could overlap for

each concept. But if conjoined features genuinely exist, they hint at

new potential explanations of polysemanticity.In

 the past, when we've observed seemingly polysemantic neurons, we've

considered two possibilities: either it is responding to some shared

feature of the stimuli, in which case it isn't really polysemantic, or

it is genuinely responding to two unrelated cases. Usually we

distinguish these cases with feature visualization. For example,

InceptionV1 4e:55 responds to cars and cat heads. One could imagine it

being the case that it's responding to some shared feature — perhaps cat

 eyes and car lights look similar. But feature visualization establishes

 a facet selecting for a globally coherent cat head, whiskers and all,

as well as the metal chrome and corners of a car. We concluded that it

was genuinely *OR(cat, car)*.

Conjoined features can be seen

 as a kind of mid-point between detecting a shared low-level feature and

 detecting independent cases. Detecting Santa Claus and "turn" are

clearly true independent cases, but there was a different facet where

they share a low-level feature.

Why would models have conjoined

features? Perhaps they're a vestigial phenomenon from early in training

when the model couldn't distinguish between the two concepts in that

facet. Or perhaps there's a case where they're still hard to

distinguish, such as large font sizes. Or maybe it just makes concept

packing more efficient, as in the superposition hypothesis.

---

Using Abstractions

------------------

We

 typically care about features because they're useful, and CLIP's

features are more useful than most. These features, when ensembled,

allow direct retrieval on a variety of queries via the dot product

alone.

Untangling the image into its semantics

 enables the model to perform a wide variety of downstream tasks

including imagenet classification, facial expression detection,

geolocalization and more. How do they do this? Answering these questions

 will require us to look at how neurons work in concert to represent a

broader space of concepts.

### The Imagenet Challenge

To

 study how CLIP classifies Imagenet, it helps to look at the simplest

case. We use a sparse linear model for this purpose, following the

methodology of Radford et al .

 With each class using only 3 neurons on average, it is easy to look at

all of the weights. This model, by any modern standard, fares poorly

with a top-5 accuracy of 56.4%, but the surprising thing is that such a

miserly model can do anything at all. How is each weight carrying so

much weight?

ImageNet

 organizes images into categories borrowed from another project called

WordNet.

Neural networks typically classify images treating ImageNet classes as

structureless labels. But WordNet actually gives them a rich structure

of higher level nodes. For example, a Labrador Retriever is a Canine

which is a Mammal which is an Animal.

We find that the weights and

 neurons of CLIP reflect some of this structure. At the highest levels

we find conventional categories such as

 This diagram visualizes a submatrix of the full weight matrix that

takes neurons in the penultimate layer of Resnet 4x to the imagenet

classes. Each grey circle represents a positive weight. We see the model

 fails in ways that close but incorrect, such as its labeling of

scorpion as a fish.

 arrive at a surprising discovery: it seems as though the neurons appear

 to arrange themselves into a taxonomy of classes that appear to mimic,

very approximately, the imagenet hierarchy. While there have been

attempts to explicitly integrate this information ,

 CLIP was not given this information as a training signal. The fact that

 these neurons naturally form a hierarchy — form a hierarchy without

even being trained on ImageNet — suggests that such hierarchy may be a

universal feature of learning systems.We've

 seen hints of similar structure in region neurons, with a whole world

neuron, a northern hemisphere neuron, a USA neuron, and then a West

Coast neuron.

### Understanding Language

The

 most exciting aspect of CLIP is its ability to do zero-shot

classification: it can be "programmed" with natural language to classify

 images into new categories, without fitting a model. Where linear

probes had fixed weights for a limited set of classes, now we have

dynamic weight vectors that can be generated automatically from text.

Indeed, CLIP makes it possible for end-users to 'roll their own

classifier' by programming the model via intuitive, natural language

commands - this will likely unlock a broad range of downstream uses of

CLIP-style models.

Recall that CLIP has two sides, a vision side

(which we've discussed up to this point) and a language side. The two

sides meet at the end, going through some processing and then performing

 a dot product to create a logit. If we ignore spatial structureIn

 order to use a contrastive loss, the 3d activation tensor of the last

convolutional layer must discard spatial information and be reduced to a

 single vector which can be dot producted with the language embedding.

CLIP does this with an attention layer, first generating attention

 is the text embedding. We focus on the bilinear interaction term, which

 governs local interactions in most directions. Although this

We'll

 mostly be focusing on using text to create zero-shot weights for

images. But it's worth noting one tool that the other direction gives

us. If we fix a neuron on the vision side, we can search for the text

that maximizes the logit. We do this with a hill climbing algorithm to

find what amounts to the text maximally corresponding to that neuron.

### Emotion Composition

As

 we see above, English has far more descriptive words for emotions than

the vision side has emotion neurons. And yet, the vision side recognizes

 these more obscure emotions. How can it do that?

We can see what

different emotion words correspond to on the vision side by taking

attribution, as described in the previous section, to "I feel X" on the

language side. This gives us a vector of image neurons for each emotion

word.Since the approximations we made in

the previous section aren't exact, we double-checked these attribution

vectors for all of the "emotion equations" shown by taking the top image

 neuron in each one, artificially increasing its activation at the last

layer on the vision side when run on a blank image, and confirming that

the logit for the corresponding emotion word increases on the language

 for the prompts "i am feeling {emotion}, "Me feeling {emotion} on my

face", "a photo of me with a {emotion} expression on my face" on each

one of the emotion-words on the emotion-wheel. We assign each prompt a

label corresponding to the emotion-word, and then we then run sparse

logistic regression to find the neurons that maximally discriminate

between the attribution vectors. For the purposes of this article, these

 vectors are then cleaned up by hand by removing neurons that respond to

 bigrams. This may relate to a line of thinking in

psychology where combinations of basic emotions form the "complex

emotions" we experience.The theory of constructed emotion.

For

 example, the jealousy emotion is success + grumpy. Bored is relaxed +

grumpy. Intimate is soft smile + heart - sick. Interested is question

mark + heart and inquisitive is question mark + shocked. Surprise is

celebration + shock.

 physical objects contribute to representing emotions.

For example, part of "powerful" is a lightning neuron, part of

"creative" is a painting neuron, part of "embarrassed" is a neuron

 also see concerning use of sensitive topics in these emotion vectors,

suggesting that problematic spurious correlations are used to caption

expressions of emotion. For instance, "accepted" detects LGBT.

"Confident" detects overweight. "Pressured" detects Asian culture.

 can also search for examples where particular neurons are used, to

explore their role in complex emotions. We see the mental illness neuron

 contributes to emotions like "stressed," "anxious," and "mad."

 far, we've only looked at a subset of these emotion words. We can also

see a birds-eye view of this broader landscape of emotions by

visualizing every attribution vector together.

 of complex emotions by applying non-negative matrix factorization to

the emotion attribution vectors and using the factors to color each

cell. The atlas resembles common feeling wheels

 hand-crafted by psychologists to explain the space of human emotions,

indicating that the vectors have a high-level structure that resembles

emotion research in psychology.This atlas has a

 few connections to classical emotion research. When we use just 2

factors, we roughly reconstruct the canonical mood-axes used in much of

psychology: valence and arousal. If we increase to 7 factors, we nearly

reconstruct a well known categorization of these emotions into happy,

surprised, sad, bad, disgusted, fearful, and angry, except with

"disgusted" switched for a new category related to affection that

includes "valued," "loving," "lonely," and "insignificant."

---

Typographic Attacks

-------------------

As

 we've seen, CLIP is full of multimodal neurons which respond to both

images and text for a given concept. Given how strongly these neurons

react to text, we wonder: can we perform a kind of non-programmatic

adversarial attack – a *typographic attack* – simply using handwriting?

To

 test this hypothesis, we took several common items and deliberately

mislabeled them. We then observed how this affects ImageNet

#### No label

#### Labeled "ipod"

#### Labeled "library"

#### Labeled "pizza"

 Physical typographic attacks.

 Above we see the CLIP RN50-4x model's classifications of

objects labeled with incorrect ImageNet classes. Each row corresponds to

 an object, and each column corresponds to a labeling. Some attacks are

more effective than others, and some objects are more resilient to

attack.

 Expand more examples

 Recall that there are two ways to use CLIP for ImageNet

classification: zero-shot and linear probes.

 For this style of attack, we observe that the zero-shot

methodology is somewhat consistently effective, but that the linear

probes methodology is ineffective. Later on, we show an attack style

 Displayed ImageNet classification method:

 Zero-shot

 Linear probes

 Adversarial patches are stickers that can be placed on real-life

objects in order to cause neural nets to misclassify those objects as

something else – for example, as toasters. Physical adversarial examples

 are complete 3D objects that are reliably misclassified from all

perspectives, such as a 3D-printed turtle that is reliably misclassified

 as a rifle. Typographic attacks are both weaker and stronger than

these. On the one hand, they only work for models with multimodal

neurons. On the other hand, once you understand this property of the

### Evaluating Typographic Attacks

Our

 physical adversarial examples are a proof of concept, but they don't

give us a very good sense of how frequently typographic attacks succeed.

 Duct tape and markers don't scale, so we create an automated setup to

measure the attack's success rate on the ImageNet validation set.

Target class:

`pizza`

Attack text:

We found text snippets for our attacks in

two different ways. Firstly, we manually looked through the multimodal

model's neurons for those that appear sensitive to particular kinds of

 attacks. Secondly, we brute-force searched through all of the ImageNet

class names looking for short class names which are, in and of

 this setup, we found several attacks to be reasonably effective. The

most successful attacks achieve a 97% attack success rate with only

around 7% of the image's pixels changed. These results are competitive

with the results found in *Adversarial Patch*, albeit on a different model.

| Target class | Attack text | Pixel cover | Success Linear probes |

| --- | --- | --- | --- |

| `waste container` | *trash* | 7.59% | 95.4% |

| `iPod` | *iPod* | 6.8% | 94.7% |

| `rifle` | *rifle* | 6.41% | 91% |

| `pizza` | *pizza* | 8.11% | 92.3% |

| `radio` | *radio* | 7.73% | 77% |

| `great white shark` | *shark* | 8.33% | 62.2% |

| `library` | *library* | 9.95% | 75.9% |

| `Siamese cat` | *meow* | 8.44% | 46.5% |

| `piggy bank` | *$\$\$\$$* | 6.99% | 36.4% |

 **Pixel cover** measures the attack's impact on the original

image: the average percentage of pixels that were changed by any amount

(an L0-norm) in order to add the attack.

 **Success rate** is measured over 1000 ImageNet validation

images with an attack considered to have succeeded if the attack class

is the most likely. We do not consider an attack to have succeeded if

the attack-free image was already classified as the attack class.

### Comparison with the Stroop Effect

 Just as our models make errors when adversarial text is added to

images, humans are slower and more error prone when images have

incongruent labels.

 is harder than normal. To compare CLIP's behavior to these human

experiments, we had CLIP classify these stimuli by color, using its

zero-shot classification. Unlike humans, CLIP can't slow down to

compensate for the harder task. Instead of taking a longer amount of

time for the incongruent stimuli, it has a very high error rate.

 A Stroop effect experiment.

 Above we see the CLIP RN50-4x model's classifications of

various words colored with various colors. Activations were gathered

 Expand more examples

---

Appendix: Methodological Details

--------------------------------

### Conditional Probability Plots

If

 we really want to understand the behavior of a neuron, it's not enough

to look at the cases where it maximally fires. We should look at the

full spectrum: the cases where it weakly fired, the cases where it was

on the border of firing, and the cases where it was strongly inhibited

from firing. This seems especially true for highly abstract neurons,

where weak activations can reveal "associated stimuli," such as a Donald

 Trump neuron firing for Mike Pence.

Since we have access to a

validation set from the same distribution the model was trained on, we

can sample the distribution of stimuli that cause a certain level of

activation by iterating through the validation set until we find an

image that causes that activation.

To more rigorously characterize

 this, we create a plot showing the conditional probability of various

categories as a function of neuron activation, following the example of

Curve Detectors . To do

 this, we defined uniformly spaced buckets between the maximally

inhibitory and maximally excitatory activation values, and sampled a

fixed number of stimuli for each activation range. Filling in the most

extreme buckets requires checking the neuron activations for millions of

 stimuli. Once we have a full set of stimuli in each bucket, we blind a

labeler to the activation of each stimuli, and have them select salient

categories they observed, informed by the hypothesis we have for the

neuron. The human labeler then categorized each stimuli into these

categories, while blinded to the activation.

We plot the

activation axis in terms of standard deviations of activation from zero,

 since activations have an arbitrary scale. But keep in mind that

activations aren't Gaussian distributed, and have much thicker tails.

In

 reading these graphs, it's important to keep in mind that different

activation levels can have many orders of magnitude differences in

probability density. In particular, probability density peaks around

zero and decays exponentially to the tails. This means that false

negatives for a rare category will tend to not be very visible, because

they'll be crowded out at zero: these graphs show a neuron's precision,

but not recall. Curve Detectors  discusses these issues in more detail.

An

 alternative possibility is to look at the distribution of activations

conditioned on a category. We take this approach in our second plot for

the Trump neuron. These plots can help characterize how the neuron

responds to rare categories in regions of higher density, and can help

resolve concerns about recall. However, one needs some way to get

samples conditioned on a category for these experiments, and it's

possible that your process may not be representative. For our purposes,

since these neurons are so high-level, we used a popular image search to

 sample images in a category.

### Faceted Feature Visualization

A neuron is said to have multiple facets

 if it responds to multiple, distinct categories of images. For example,

 a pose-invariant dog-head detector detects dog heads tilted to the

 look for a difference in texture from one side to the other but doesn't

 care which is which. A neuron may even fire for many different,

unrelated categories of images . We refer to these as polysemantic neurons.

[Feature visualization](https://distill.pub/2017/feature-visualization/)is

 a technique where the input to a neural network is optimized to create a

 stimuli demonstrating some behavior, typically maximizing the

activation of a neuron. Neurons that possess multiple facets present

particular challenges to feature visualization as the multiple facets

are difficult to represent as a single image. When such neurons are

encountered, feature visualization often tries to draw both facets at

once (making it nonsensical), or just reveal one facet The

 difference between the two is believed to be related to the phenomena

of mutual inhibition, see the InceptionV1 pose invariant dog head

circuit.. Both cases are inadequate.

We

 are aware of two past approaches to improving feature visualization for

 multi-faceted neurons. The first approach is to find highly diverse

images that activate a given neuron, and use them as seeds for the

feature visualization optimization process.

 The second tries to combine feature visualization together with a term

that encourages diversity of the activations on earlier layers.

 that allows us to steer the feature visualization towards a particular

theme (e.g. text, logos, facial features, etc), defined by a collection

of images. The procedure works as follows: first we collect examples of

images in this theme, and train a linear probe on the lower layers of

the model to discriminate between those images and generic natural

images. We then do feature visualization by maximizing the penalized

The reader may be curious why we do not maximize f(g(x)) + w^Tg(x)

 instead. We have found that, in practice, the former objective produces

 far higher quality feature visualizations; we believe this is because

Invariant visual representation by single neurons in the human brain

Explicit encoding of multimodal percepts by single neurons in the human brain

Learning Transferable Visual Models From Natural Language Supervision

Deep Residual Learning for Image Recognition

Attention is all you need

Improved deep metric learning with multi-class n-pair loss objective

Contrastive multiview coding

Linear algebraic structure of word senses, with applications to polysemy

Visualizing and understanding recurrent networks

Object detectors emerge in deep scene cnns

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Zoom In: An Introduction to Circuits

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Sparse but not ‘grandmother-cell’ coding in the medial temporal lobe

Concept cells: the building blocks of declarative memory functions

Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements

Geographical evaluation of word embeddings

Using Artificial Intelligence to Augment Human Intelligence

Visualizing Representations: Deep Learning and Human Beings

Natural language processing (almost) from scratch

Linguistic regularities in continuous space word representations

Man is to computer programmer as woman is to homemaker? debiasing word embeddings

Intriguing properties of neural networks

Visualizing higher-layer features of a deep network

Feature Visualization

How does the brain solve visual object recognition?

Imagenet: A large-scale hierarchical image database

BREEDS: Benchmarks for Subpopulation Shift

Global Weighted Average Pooling Bridges Pixel-level Localization and Image-level Classification

Separating style and content with bilinear models

The feeling wheel: A tool for expanding awareness of emotions and increasing spontaneity and intimacy

Activation atlas

Adversarial Patch

Synthesizing Robust Adversarial Examples

Studies of interference in serial verbal reactions.

Curve Detectors

An overview of early vision in inceptionv1

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Sun database: Large-scale scene recognition from abbey to zoo

The pascal visual object classes (voc) challenge

Fairface: Face attribute dataset for balanced race, gender, and age

A style-based generator architecture for generative adversarial networks