Using Artificial Intelligence to Augment Human Intelligence

Abstract: Historically, different answers to this question – that is,     different visions of computing – have helped inspire and     determine the computing systems humanity has ultimately     built. Consider the early electronic computers.  ENIAC, the     world’s first general-purpose electronic computer, was     commissioned to compute artillery firing tables for the United     States Army.  Other early computers were also used to solve     numerical problems, such as simulating nuclear explosions,     predicting the weather, and planning the motion of rockets. The     machines operated in a batch mode, using crude input and output     devices, and without any real-time interaction.  It was a vision     of computers as number-crunching machines, used to speed up     calculations that would formerly have taken weeks, months, or more     for a team of humans.

 What are computers for?

-------------------------

 Historically, different answers to this question – that is,

 different visions of computing – have helped inspire and

 determine the computing systems humanity has ultimately

 built. Consider the early electronic computers. ENIAC, the

 world's first general-purpose electronic computer, was

 commissioned to compute artillery firing tables for the United

 States Army. Other early computers were also used to solve

 numerical problems, such as simulating nuclear explosions,

 predicting the weather, and planning the motion of rockets. The

 machines operated in a batch mode, using crude input and output

 devices, and without any real-time interaction. It was a vision

 of computers as number-crunching machines, used to speed up

 calculations that would formerly have taken weeks, months, or more

 for a team of humans.

 In the 1950s a different vision of what computers are for began to

 develop. That vision was crystallized in 1962, when Douglas

 Engelbart proposed that computers could be used as a way

 of augmenting human

 intellect. In this view, computers weren't primarily

 tools for solving number-crunching problems. Rather, they were

 real-time interactive systems, with rich inputs and outputs, that

 humans could work with to support and expand their own

 problem-solving process. This vision of intelligence augmentation

 (IA) deeply influenced many others, including researchers such as

 Alan Kay at Xerox PARC, entrepreneurs such as Steve Jobs at Apple,

 and led to many of the key ideas of modern computing systems. Its

 ideas have also deeply influenced digital art and music, and

 fields such as interaction design, data visualization,

 computational creativity, and human-computer interaction.

 Research on IA has often been in competition with research on

 artificial intelligence (AI): competition for funding, competition

 for the interest of talented researchers. Although there has

 always been overlap between the fields, IA has typically focused

 on building systems which put humans and machines to work

 together, while AI has focused on complete outsourcing of

 intellectual tasks to machines. In particular, problems in AI are

 often framed in terms of matching or surpassing human performance:

 beating humans at chess or Go; learning to recognize speech and

 images or translating language as well as humans; and so on.

 This essay describes a new field, emerging today out of a

 synthesis of AI and IA. For this field, we suggest the

 name *artificial intelligence augmentation* (AIA): the use

 of AI systems to help develop new methods for intelligence

 augmentation. This new field introduces important new fundamental

 questions, questions not associated to either parent field. We

 believe the principles and systems of AIA will be radically

 different to most existing systems.

 Our essay begins with a survey of recent technical work hinting at

 artificial intelligence augmentation, including work

 on *generative interfaces* – that is, interfaces

 which can be used to explore and visualize generative machine

 learning models. Such interfaces develop a kind of cartography of

 generative models, ways for humans to explore and make meaning

 from those models, and to incorporate what those models

 "know" into their creative work.

 Our essay is not just a survey of technical work. We believe now

 is a good time to identify some of the broad, fundamental

 questions at the foundation of this emerging field. To what

 extent are these new tools enabling creativity? Can they be used

 to generate ideas which are truly surprising and new, or are the

 ideas cliches, based on trivial recombinations of existing ideas?

 Can such systems be used to develop fundamental new interface

 primitives? How will those new primitives change and expand the

 way humans think?

 Using generative models to invent meaningful creative operations

------------------------------------------------------------------

 Let's look at an example where a machine learning model makes a

 new type of interface possible. To understand the interface,

 imagine you're a type designer, working on creating a new

 fontWe shall egregiously abuse the distinction between

 a font and a typeface. Apologies to any type designers who may be

 reading.. After sketching some initial designs, you

 wish to experiment with bold, italic, and condensed variations.

 Let's examine a tool to generate and explore such variations, from

 any initial design. For reasons that will soon be explained the

 quality of results is quite crude; please bear with us.

#### Starting Font

 …

 …

 …

 …

 …

 …

#### Modifications

Bold

Italic

Condensed

#### Result

 Of course, varying the bolding (i.e., the weight), italicization

 and width are just three ways you can vary a font. Imagine that

 instead of building specialized tools, users could build their own

 tool merely by choosing examples of existing fonts. For instance,

 suppose you wanted to vary the degree of serifing on a font. In

 the following, please select 5 to 10 sans-serif fonts from the top

 box, and drag them to the box on the left. Select 5 to 10 serif

 fonts and drag them to the box on the right. As you do this, a

 machine learning model running in your browser will automatically

 infer from these examples how to interpolate your starting font in

 either the serif or sans-serif direction:

Drag **sans-serif** examples here

Drag **serif** examples here

#### Starting font

 …

 …

 …

 …

 …

 …

#### Apply your new tool

#### Result

 In fact, we used this same technique to build the earlier bolding

 italicization, and condensing tool. To do so, we used the

 following examples of bold and non-bold fonts, of italic and

 non-italic fonts, and of condensed and non-condensed fonts:

#### Bold

#### Italic

#### Condensed

 To build these tools, we used what's called a *generative

 model*; the particular model we use was trained

 by James Wexler. To

 understand generative models, consider that *a priori*

 describing a font appears to require a lot of data. For

 instance, if the font is 646464 by 646464 pixels, then we'd expect

 glyph. But we can use a generative model to find a much simpler

 description.

 We do this by building a neural network which takes a small number

 of input variables, called *latent variables*, and produces

 as output the entire glyph. For the particular model we use, we

 have 404040 latent space dimensions, and map that into the

 4,0964,0964,096-dimensional space describing all the pixels in the glyph.

 In other words, the idea is to map a low-dimensional space into a

 higher-dimensional space:

Neural network

 The generative model we use is a type of neural network known as

 a variational autoencoder

 (VAE). For our purposes, the details of the generative

 model aren't so important. The important thing is that by

 changing the latent variables used as input, it's possible to get

 different fonts as output. So one choice of latent variables will

 give one font, while another choice will give a different font:

…

…

…

 You can think of the latent variables as a compact, high-level

 representation of the font. The neural network takes that

 high-level representation and converts it into the full pixel

 data. It's remarkable that just 404040 numbers can capture the

 apparent complexity in a glyph, which originally required 4,0964,0964,096

 variables.

 The generative model we use is learnt from a training set of more

 than 505050 thousand

 fonts Bernhardsson

 scraped from the open web. During training, the weights and

 biases in the network are adjusted so that the network can output

 a close approximation to any desired font from the training set,

 provided a suitable choice of latent variables is made. In some

 sense, the model is learning a highly compressed representation of

 all the training fonts.

 In fact, the model doesn't just reproduce the training fonts. It

 can also generalize, producing fonts not seen in training. By

 being forced to find a compact description of the training

 examples, the neural net learns an abstract, higher-level model of

 what a font is. That higher-level model makes it possible to

 generalize beyond the training examples already seen, to produce

 realistic-looking fonts.

 Ideally, a good generative model would be exposed to a relatively

 small number of training examples, and use that exposure to

 generalize to the space of all possible human-readable fonts.

 That is, for any conceivable font – whether existing or

 perhaps even imagined in the future – it would be possible

 to find latent variables corresponding exactly to that font. Of

 course, the model we're using falls far short of this ideal

 – a particularly egregious failure is that many fonts

 generated by the model omit the tail on the capital

 "Q" (you can see this in the examples above). Still,

 it's useful to keep in mind what an ideal generative model would

 do.

 Such generative models are similar in some ways to how scientific

 theories work. Scientific theories often greatly simplify the

 description of what appear to be complex phenomena, reducing large

 numbers of variables to just a few variables from which many

 aspects of system behavior can be deduced. Furthermore, good

 scientific theories sometimes enable us to generalize to discover

 new phenomena.

 As an example, consider ordinary material objects. Such objects

 have what physicists call a *phase* – they may be a

 liquid, a solid, a gas, or perhaps something more exotic, like a

 superconductor

 or [Bose-Einstein

 complex, involving perhaps 102310^{23}1023 or so molecules. But the

 laws of thermodynamics and statistical mechanics enable us to find

 a simpler description, reducing that complexity to just a few

 variables (temperature, pressure, and so on), which encompass much

 of the behavior of the system. Furthermore, sometimes it's

 possible to generalize, predicting unexpected new phases of

 matter. For example, in 1924, physicists used thermodynamics and

 statistical mechanics to predict a remarkable new phase of matter,

 Bose-Einstein condensation, in which a collection of atoms may all

 occupy identical quantum states, leading to surprising large-scale

 quantum interference effects. We'll come back to this predictive

 ability in our later discussion of creativity and generative

 models.

 Returning to the nuts and bolts of generative models, how can we

 use such models to do example-based reasoning like that in the

 tool shown above? Let's consider the case of the bolding tool. In

 that instance, we take the average of all the latent vectors for

 the user-specified bold fonts, and the average for all the

 user-specified non-bold fonts. We then compute the difference

 between these two average vectors:

bolding vectoraverage ofnon-bold fontsaverage ofbold fonts

 We'll refer to this as the *bolding vector*. To make some

 given font bolder, we simply add a little of the bolding vector to

 the corresponding latent vector, with the amount of bolding vector

 added controlling the boldness of the resultIn

 practice, sometimes a slightly different procedure is used. In

 some generative models the latent vectors satisfy some constraints

 – for instance, they may all be of the same length. When

 that's the case, as in our model, a more sophisticated

 "adding" operation must be used, to ensure the length

 remains the same. But conceptually, the picture of adding the

 bolding vector is the right way to think.:

 This technique was introduced

 by Larsen *et al*, and

 vectors like the bolding vector are sometimes called

 *attribute vectors*. The same idea is use to implement all

 the tools we've shown. That is, we use example fonts to creating

 a bolding vector, an italicizing vector, a condensing vector, and

 a user-defined serif vector. The interface thus provides a way of

 exploring the latent space in those four directions.

 The tools we've shown have many drawbacks. Consider the following

 example, where we start with an example glyph, in the middle, and

 either increase or decrease the bolding (on the right and left,

 respectively):

original

 Examining the glyphs on the left and right we see many unfortunate

 artifacts. Particularly for the rightmost glyph, the edges start to get

 rough, and the serifs begin to disappear. A better generative

 model would reduce those artifacts. That's a good long-term

 research program, posing many intriguing problems. But even with

 the model we have, there are also some striking benefits to the

 use of the generative model.

 To understand these benefits, consider a naive approach to

 bolding, in which we simply add some extra pixels around a glyph's

 edges, thickening it up. While this thickening perhaps matches a

 non-expert's way of thinking about type design, an expert does

 something much more involved. In the following we show the

 results of this naive thickening procedure versus what is actually

 done, for Georgia and Helvetica:

####

#### Naive bolding

#### Actual bolding

Georgia

Helvetica

 As you can see, the naive bolding procedure produces quite

 different results, in both cases. For example, in Georgia, the

 left stroke is only changed slightly by bolding, while the right

 stroke is greatly enlarged, but only on one side. In both

 fonts, bolding doesn't change the height of the font, while the

 naive approach does.

 As these examples show, good bolding is *not* a trivial

 process of thickening up a font. Expert type designers have many

 heuristics for bolding, heuristics inferred from much previous

 experimentation, and careful study of historical

 examples. Capturing all those heuristics in a conventional program

 would involve immense work. The benefit of using the generative

 model is that it automatically learns many such heuristics.

 For example, a naive bolding tool would rapidly fill in the

 enclosed negative space in the enclosed upper region of the letter

 "A". The font tool doesn't do this. Instead, it goes

 to some trouble to preserve the enclosed negative space, moving

 the A's bar down, and filling out the interior strokes more slowly

 than the exterior. This principle is evident in the examples

 shown above, especially Helvetica, and it can also be seen in the

 operation of the font tool:

original

original

Height is constant

original

Even non-standard fonts are properly bolded

 The heuristic of preserving enclosed negative space is not *a

 priori* obvious. However, it's done in many professionally

 designed fonts. If you examine examples like those shown above

 it's easy to see why: it improves legibility. During training,

 our generative model has automatically inferred this principle

 from the examples it's seen. And our bolding interface then makes

 this available to the user.

 In fact, the model captures many other heuristics. For instance,

 in the above examples the heights of the fonts are (roughly)

 preserved, which is the norm in professional font design. Again,

 what's going on isn't just a thickening of the font, but rather

 the application of a more subtle heuristic inferred by the

 generative model. Such heuristics can be used to create fonts

 with properties which would otherwise be unlikely to occur to

 users. Thus, the tool expands ordinary people's ability to

 explore the space of meaningful fonts.

 The font tool is an example of a kind of cognitive technology. In

 particular, the primitive operations it contains can be

 internalized as part of how a user thinks. In this it resembles a

 program such as *Photoshop* or a spreadsheet or 3D graphics

 programs. Each provides a novel set of interface primitives,

 primitives which can be internalized by the user as fundamental

 new elements in their thinking. This act of internalization of new

 primitives is fundamental to much work on intelligence

 augmentation.

 The ideas shown in the font tool can be extended to other domains.

 Using the same interface, we can use a generative model to

 manipulate images of human faces using qualities such as

 expression, gender, or hair color. Or to manipulate sentences

 using length, sarcasm, or tone. Or to manipulate molecules using

 chemical properties:

 #alternate-uses .root {

 max-width: 300px;

 margin: 0 auto;

 padding: 0 20px;

 }

 @media(min-width: 600px) {

 #alternate-uses .root {

 max-width: 760px;

 display: grid;

 grid-template-columns: 1fr 1fr 1fr;

 grid-gap: 50px;

 }

 }

 .note {

 margin-top: 12px;

 grid-column-end: span 2;

 font-size: 10px;

 line-height: 1.5em;

 text-align: left;

 color: rgba(0, 0, 0, 0.4);

 }

#### Faces

Smiling

 Images from *Sampling Generative Networks* by White.

#### Sentences

"Everyone knows that a rich, single man must want a wife."

"A rich, single man obviously must want a wife."

"A single man must want a wife."

Length

#### Molecules

Delayed fluorescence decay rate

 Such generative interfaces provide a kind of cartography of

 generative models, ways for humans to explore and make meaning

 using those models.

 We saw earlier that the font model automatically infers relatively

 deep principles about font design, and makes them available to

 users. While it's great that such deep principles can be

 inferred, sometimes such models infer other things that are wrong,

 or undesirable. For example, White

 points out the addition of a smile vector in some face

 models will make faces not just smile more, but also appear more

 feminine. Why? Because in the training data more women than men

 were smiling. So these models may not just learn deep facts about

 the world, they may also internalize prejudices or erroneous

 beliefs. Once such a bias is known, it is often possible to make

 corrections. But to find those biases requires careful auditing

 of the models, and it is not yet clear how we can ensure such

 audits are exhaustive.

 More broadly, we can ask why attribute vectors work, when they

 work, and when they fail? At the moment, the answers to these

 questions are poorly understood.

 For the attribute vector to work requires that taking any starting

 font, we can construct the corresponding bold version by adding

 the *same* vector in the latent space. However, *a

 priori* there is no reason using a single constant vector to

 displace will work. It may be that we should displace in many

 different ways. For instance, the heuristics used to bold serif

 and sans-serif fonts are quite different, and so it seems likely

 that very different displacements would be involved:

#### Assumption

#### Reality

 Of course, we could do something more sophisticated than using a

 single constant attribute vector. Given pairs of example fonts

 (unbold, bold) we could train a machine learning algorithm to take

 as input the latent vector for the unbolded version and output the

 latent vector for the bolded version. With additional training

 data about font weights, the machine learning algorithm could

 learn to generate fonts of arbitrary weight. Attribute vectors

 are just an extremely simple approach to doing these kinds of

 operation.

 For these reasons, it seems unlikely that attribute vectors will

 last as an approach to manipulating high-level features. Over the

 next few years much better approaches will be developed. However,

 we can still expect interfaces offering operations broadly similar

 to those sketched above, allowing access to high-level and

 potentially user-defined concepts. That interface pattern doesn't

 depend on the technical details of attribute vectors.

 Interactive Generative Adversarial Models

-------------------------------------------

 Let's look at another example using machine learning models to

 augment human creativity. It's the interactive generative

 adversarial networks, or iGANs, introduced

 by Zhu *et al* in 2016.

 One of the examples of Zhu *et al* is the use of iGANs in

 an interface to generate images of consumer products such as

 shoes. Conventionally, such an interface would require the

 programmer to write a program containing a great deal of knowledge

 about shoes: soles, laces, heels, and so on. Instead of doing

 this, Zhu *et al* train a generative model using 505050

 thousand images of shoes, downloaded from Zappos. They then use

 that generative model to build an interface that lets a user

 roughly sketch the shape of a shoe, the sole, the laces, and so

 on:

Excerpted from Zhu *et

 al*.

 The visual quality is low, in part because the generative model

 Zhu *et al* used is outdated by modern (2017) standards

 – with more modern models, the visual quality would be much

 higher.

 But the visual quality is not the point. Many interesting things

 are going on in this prototype. For instance, notice how the

 overall shape of the shoe changes considerably when the sole is

 filled in – it becomes narrower and sleeker. Many small

 details are filled in, like the black piping on the top of the

 white sole, and the red coloring filled in everywhere on the

 shoe's upper. These and other facts are automatically deduced

 from the underlying generative model, in a way we'll describe

 shortly.

 The same interface may be used to sketch landscapes. The only

 difference is that the underlying generative model has been

 trained on landscape images rather than images of shoes. In this

 case it becomes possible to sketch in just the colors associated

 to a landscape. For example, here's a user sketching in some green

 grass, the outline of a mountain, some blue sky, and snow on the

 mountain:

Excerpted from Zhu *et

 al*.

 The generative models used in these interfaces are different than

 for our font model. Rather than using variational autoencoders,

 they're based on generative

 adversarial networks (GANs). But the underlying idea is

 still to find a low-dimensional latent space which can be used to

 represent (say) all landscape images, and map that latent space to

 a corresponding image. Again, we can think of points in the

 latent space as a compact way of describing landscape images.

 Roughly speaking, the way the iGANs works is as follows. Whatever

 the current image is, it corresponds to some point in the latent

 space:

 Suppose, as happened in the earlier video, the user now sketches

 in a stroke outlining the mountain shape. We can think of the

 stroke as a constraint on the image, picking out a subspace of the

 latent space, consisting of all points in the latent space whose

 image matches that outline:

Subspace of all images that satisfy the mountain constraint

 The way the interface works is to find a point in the latent space

 which is near to the current image, so the image is not changed

 too much, but also coming close to satisfying the imposed

 constraints. This is done by optimizing an objective function

 which combines the distance to each of the imposed constraints, as

 well as the distance moved from the current point. If there's

 just a single constraint, say, corresponding to the mountain

 stroke, this looks something like the following:

 We can think of this, then, as a way of applying constraints to

 the latent space to move the image around in meaningful ways.

 The iGANs have much in common with the font tool we showed

 earlier. Both make available operations that encode much subtle

 knowledge about the world, whether it be learning to understand

 what a mountain looks like, or inferring that enclosed negative

 space should be preserved when bolding a font. Both the iGANs and

 the font tool provide ways of understanding and navigating a

 high-dimensional space, keeping us on the natural space of fonts

 or shoes or landscapes. As Zhu *et al* remark:

>

>  [F]or most of us, even a simple image manipulation in Photoshop

>  presents insurmountable difficulties… any less-than-perfect

>  edit immediately makes the image look completely unrealistic. To

>  put another way, classic visual manipulation paradigm does not

>  prevent the user from "falling off" the manifold of

>  natural images.

>

 Like the font tool, the iGANs is a cognitive technology. Users

 can internalize the interface operations as new primitive elements

 in their thinking. In the case of shoes, for example, they can

 learn to think in terms of the difference they want to apply,

 adding a heel, or a higher top, or a special highlight. This is

 richer than the traditional way non-experts think about shoes

 ("Size 11, black" *etc*). To the extent that

 non-experts do think in more sophisticated ways –

 "make the top a little higher and sleeker" –

 they get little practice in thinking this way, or seeing the

 consequences of their choices. Having an interface like this

 enables easier exploration, the ability to develop idioms and the

 ability to plan, to swap ideas with friends, and so on.

 Two models of computation

---------------------------

 Let's revisit the question we began the essay with, the question

 of what computers are for, and how this relates to intelligence

 augmentation.

 One common conception of computers is that they're problem-solving

 machines: "computer, what is the result of firing this

 artillery shell in such-and-such a wind [and so on]?";

 "computer, what will the maximum temperature in Tokyo be in

 5 days?"; "computer, what is the best move to take

 when the Go board is in this position?"; "computer,

 how should this image be classified?"; and so on.

 This is a conception common to both the early view of computers as

 number-crunchers, and also in much work on AI, both historically

 and today. It's a model of a computer as a way of outsourcing

 cognition. In speculative depictions of possible future AI,

 this *cognitive outsourcing* model often shows up in the

 view of an AI as an oracle, able to solve some large class of

 problems with better-than-human performance.

 But a very different conception of what computers are for is

 possible, a conception much more congruent with work on

 intelligence augmentation.

 To understand this alternate view, consider our subjective

 experience of thought. For many people, that experience is verbal:

 they think using language, forming chains of words in their heads,

 similar to sentences in speech or written on a page. For other

 people, thinking is a more visual experience, incorporating

 representations such as graphs and maps. Still other people mix

 mathematics into their thinking, using algebraic expressions or

 diagrammatic techniques, such as Feynman diagrams and Penrose

 diagrams.

 In each case, we're thinking using representations invented by

 other people: words, graphs, maps, algebra, mathematical diagrams,

 and so on. We internalize these cognitive technologies as we grow

 up, and come to use them as a kind of substrate for our thinking.

 For most of history, the range of available cognitive technologies

 has changed slowly and incrementally. A new word will be

 introduced, or a new mathematical symbol. More rarely, a radical

 new cognitive technology will be developed. For example, in 1637

 Descartes published his "Discourse on Method",

 explaining how to represent geometric ideas using algebra, and

 vice versa:

-1.5-1-0.50.511.5-1.5-1-0.50.511.5

x2 + y2 = 1

While it may seem obvious now, it is remarkable that a

geometric shape and an equation can represent the same underlying

mathematical object.

 This enabled a radical change and expansion in how we think about

 both geometry and algebra.

 Historically, lasting cognitive technologies have been invented

 only rarely. But modern computers are a meta-medium enabling the

 rapid invention of many new cognitive technologies. Consider a

 relatively banal example, such

 as *Photoshop*. Adept *Photoshop* users routinely

 have formerly impossible thoughts such as: "let's apply the

 clone stamp to the such-and-such layer.". That's an

 instance of a more general class of thought: "computer, [new

 type of action] this [new type of representation for a newly

 imagined class of object]". When that happens, we're using

 computers to expand the range of thoughts we can think.

 It's this kind of *cognitive transformation* model which

 underlies much of the deepest work on intelligence augmentation.

 Rather than outsourcing cognition, it's about changing the

 operations and representations we use to think; it's about

 changing the substrate of thought itself. And so while cognitive

 outsourcing is important, this cognitive transformation view

 offers a much more profound model of intelligence augmentation.

 It's a view in which computers are a means to change and expand

 human thought itself.

 Historically, cognitive technologies were developed by human

 inventors, ranging from the invention of writing in Sumeria and

 Mesoamerica, to the modern interfaces of designers such as Douglas

 Engelbart, Alan Kay, and others.

 Examples such as those described in this essay suggest that AI

 systems can enable the creation of new cognitive technologies.

 Things like the font tool aren't just oracles to be consulted when

 you want a new font. Rather, they can be used to explore and

 discover, to provide new representations and operations, which can

 be internalized as part of the user's own thinking. And while

 these examples are in their early stages, they suggest AI is not

 just about cognitive outsourcing. A different view of AI is

 possible, one where it helps us invent new cognitive technologies

 which transform the way we think.

 In this essay we've focused on a small number of examples, mostly

 involving exploration of the latent space. There are many other

 examples of artificial intelligence augmentation. To give some

 flavor, without being comprehensive:

 the sketch-rnn system, for neural

 network assisted drawing;

 the Wekinator, which enables

 users to rapidly build new musical instruments and artistic

 systems; TopoSketch, for developing

 animations by exploring latent spaces; machine learning models for

 designing overall typographic

 layout; and a generative model which enables

 interpolation between musical

 phrases. In each case, the systems use machine learning

 to enable new primitives which can be integrated into the user's

 thinking. More broadly, artificial intelligence augmentation will

 draw on fields such as computational

 creativity and interactive machine

 learning.

 Finding powerful new primitives of thought

--------------------------------------------

 We've argued that machine learning systems can help create

 representations and operations which serve as new primitives in

 human thought. What properties should we look for in such new

 primitives? This is too large a question to be answered

 comprehensively in a short essay. But we will explore it briefly.

 Historically, important new media forms often seem strange when

 introduced. Many such stories have passed into popular culture:

 the near riot at the premiere of Stravinsky and Nijinksy's

 "Rite of Spring"; the consternation caused by the

 early cubist paintings, leading

 *The New York Times* to

 comment: "What do they mean? Have those

 responsible for them taken leave of their senses? Is it art or

 madness? Who knows?"

 Another example comes from physics. In the 1940s, different

 formulations of the theory of quantum electrodynamics were

 developed independently by the physicists Julian Schwinger,

 Shin'ichirō Tomonaga, and Richard Feynman. In their work,

 Schwinger and Tomonaga used a conventional algebraic approach,

 along lines similar to the rest of physics. Feynman used a more

 radical approach, based on what are now known as Feynman diagrams,

 for depicting the interaction of light and matter:

Image by [Joel

 Attribution-Share Alike 3.0 Unported license

 Initially, the Schwinger-Tomonaga approach was easier for other

 physicists to understand. When Feynman and Schwinger presented

 their work at a 1948 workshop, Schwinger was immediately

 acclaimed. By contrast, Feynman left his audience mystified. As

 James Gleick put it in his biography of

 Feynman:

>

>  It struck Feynman that everyone had a favorite principle or

>  theorem and he was violating them all… Feynman knew he had

>  failed. At the time, he was in anguish. Later he said simply:

>  "I had too much stuff. My machines came from too far

>  away."

>

 Of course, strangeness for strangeness's sake alone is not

 useful. But these examples suggest that breakthroughs in

 representation often appear strange at first. Is there any

 underlying reason that is true?

 Part of the reason is because if some representation is truly new,

 then it will appear different than anything you've ever seen

 before. Feynman's diagrams, Picasso's paintings, Stravinsky's

 music: all revealed genuinely new ways of making meaning. Good

 representations sharpen up such insights, eliding the familiar to

 show that which is new as vividly as possible. But because of

 that emphasis on unfamiliarity, the representation will seem

 strange: it shows relationships you've never seen before. In some

 sense, the task of the designer is to identify that core

 strangeness, and to amplify it as much as possible.

 Strange representations are often difficult to understand. At

 first, physicists preferred Schwinger-Tomonaga to Feynman. But as

 Feynman's approach was slowly understood by physicists, they

 realized that although Schwinger-Tomonaga and Feynman were

 mathematically equivalent, Feynman was more powerful. As Gleick

 puts it:

>

>  Schwinger's students at Harvard were put at a competitive

>  disadvantage, or so it seemed to their fellows elsewhere, who

>  suspected them of surreptitiously using the diagrams anyway. This

>  was sometimes true… Murray Gell-Mann later spent a semester

>  staying in Schwinger's house and loved to say afterward that he

>  had searched everywhere for the Feynman diagrams. He had not

>  found any, but one room had been locked…

>

 These ideas are true not just of historical representations, but

 also of computer interfaces. However, our advocacy of strangeness

 in representation contradicts much conventional wisdom about

 interfaces, especially the widely-held belief that they should be

 "user friendly", i.e., simple and immediately useable

 by novices. That most often means the interface is cliched, built

 from conventional elements combined in standard ways. But while

 using a cliched interface may be easy and fun, it's an ease

 similar to reading a formulaic romance novel. It means the

 interface does not reveal anything truly surprising about its

 subject area. And so it will do little to deepen the user's

 understanding, or to change the way they think. For mundane tasks

 that is fine, but for deeper tasks, and for the longer term, you

 want a better interface.

 Ideally, an interface will surface the deepest principles

 underlying a subject, revealing a new world to the user. When you

 learn such an interface, you internalize those principles, giving

 you more powerful ways of reasoning about that world. Those

 principles are the diffs in your understanding. They're all you

 really want to see, everything else is at best support, at worst

 unimportant dross. The purpose of the best interfaces isn't to be

 user-friendly in some shallow sense. It's to be user-friendly in

 a much stronger sense, reifying deep

 principles about the world, making them the working

 conditions in which users live and create. At that point what once

 appeared strange can instead becomes comfortable and familiar,

 part of the pattern of thoughtA powerful instance of

 these ideas is when an interface reifies general-purpose

 principles. An example is an

 interface one of us developed

 based on the principle of conservation of energy. Such

 general-purpose principles generate multiple unexpected

 relationships between the entities of a subject, and so are a

 particularly rich source of insights when reified in an

 interface..

 What does this mean for the use of AI models for intelligence

 augmentation?

 Aspirationally, as we've seen, our machine learning models will

 help us build interfaces which reify deep principles in ways

 meaningful to the user. For that to happen, the models have to

 discover deep principles about the world, recognize those

 principles, and then surface them as vividly as possible in an

 interface, in a way comprehensible by the user.

 Of course, this is a tall order! The examples we've shown are just

 barely beginning to do this. It's true that our models do

 sometimes discover relatively deep principles, like the

 preservation of enclosed negative space when bolding a font. But

 this is merely implicit in the model. And while we've built a tool

 which takes advantage of such principles, it'd be better if the

 model automatically inferred the important principles learned, and

 found ways of explicitly surfacing them through the interface.

 (Encouraging progress toward this has been made

 by InfoGANs, which use

 information-theoretic ideas to find structure in the latent

 space.) Ideally, such models would start to get at true

 explanations, not just in a static form, but in a dynamic form,

 manipulable by the user. But we're a long way from that point.

 Do these interfaces inhibit creativity?

-----------------------------------------

 It's tempting to be skeptical of the expressiveness of the

 interfaces we've described. If an interface constrains us to

 explore only the natural space of images, does that mean we're

 merely doing the expected? Does it mean these interfaces can only

 be used to generate visual cliches? Does it prevent us from

 generating anything truly new, from doing truly creative work?

 To answer these questions, it's helpful to identify two different

 modes of creativity. This two-mode model is over-simplified:

 creativity doesn't fit so neatly into two distinct categories. Yet

 the model nonetheless clarifies the role of new interfaces in

 creative work.

 The first mode of creativity is the everyday creativity of a

 craftsperson engaged in their craft. Much of the work of a font

 designer, for example, consists of competent recombination of the

 best existing practices. Such work typically involves many

 creative choices to meet the intended design goals, but not

 developing key new underlying principles.

 For such work, the generative interfaces we've been discussing are

 promising. While they currently have many limitations, future

 research will identity and fix many deficiencies. This is

 happening rapidly with GANs: the original

 GANs had many limitations,

 but models soon appeared that were better adapted to

 images, improved the

 resolution, reduced artifactsSo much work has been

 done on improving resolution and reducing artifacts it seems

 unfair to single out any small set of papers, and to omit the many

 others., and so on. With enough iterations it's

 plausible these generative interfaces will become powerful tools

 for craft work.

 The second mode of creativity aims toward developing new

 principles that fundamentally change the range of creative

 expression. One sees this in the work of artists such as Picasso

 or Monet, who violated existing principles of painting, developing

 new principles which enabled people to see in new ways.

 Is it possible to do such creative work, while using a generative

 interface? Don't such interfaces constrain us to the space of

 natural images, or natural fonts, and thus actively prevent us

 from exploring the most interesting new directions in creative

 work?

 The situation is more complex than this.

 In part, this is a question about the power of our generative

 models. In some cases, the model can only generate recombinations

 of existing ideas. This is a limitation of an ideal GAN, since a

 perfectly trained GAN generator will reproduce the training

 distribution. Such a model can't directly generate an image based

 on new fundamental principles, because such an image wouldn't look

 anything like it's seen in its training data.

 Artists such as [Mario

 Klingemann](http://quasimondo.com/) and [Mike

 Tyka](http://www.miketyka.com/) are now using GANs to create interesting

 artwork. They're doing that using "imperfect" GAN

 models, which they seem to be able to use to explore interesting

 new principles; it's perhaps the case that bad GANs may be more

 artistically interesting than ideal GANs. Furthermore, nothing

 says an interface must only help us explore the latent space.

 Perhaps operations can be added which deliberately take us out

 of the latent space, or to less probable (and so more

 surprising) parts of the space of natural images.

 Of course, GANs are not the only generative models. In a

 sufficiently powerful generative model, the generalizations

 discovered by the model may contain ideas going beyond what humans

 have discovered. In that case, exploration of the latent space may

 enable us to discover new fundamental principles. The model would

 have discovered stronger abstractions than human experts. Imagine

 a generative model trained on paintings up until just before the

 time of the cubists; might it be that by exploring that model it

 would be possible to discover cubism? It would be an analogue to

 something like the prediction of Bose-Einstein condensation, as

 discussed earlier in the essay. Such invention is beyond today's

 generative models, but seems a worthwhile aspiration for future

 models.

 Our examples so far have all been based on generative models. But

 there are some illuminating examples which are not based on

 generative models. Consider the pix2pix system developed

 by Isola *et al*. This

 system is trained on pairs of images, e.g., pairs showing the

 edges of a cat, and the actual corresponding cat. Once trained,

 it can be shown a set of edges and asked to generate an image for

 an actual corresponding cat. It often does this quite well:

 .cat-grid .row {

 grid-template-columns: 1fr 1fr 0.5fr;

 align-items: center;

 }

 .cat-grid .row {

 border-bottom: 1px solid rgba(0, 0, 0, 0.1);

 }

 .cat-grid .row:last-child {

 border-bottom: none;

 }

#### Input

#### Output

[Live demo by Christopher Hesse](https://affinelayer.com/pixsrv/)

 When supplied with unusual constraints, pix2pix can produce

 striking images:

[Bread cat by Ivy Tsai](https://twitter.com/ivymyt/status/834174687282241537)

[Cat beholder by Marc Hesse](https://affinelayer.com/pixsrv/beholder.jpg)

Spiral cat

 This is perhaps not high creativity of a Picasso-esque level. But

 it is still surprising. It's certainly unlike images most of us

 have ever seen before. How does pix2pix and its human user achieve

 this kind of result?

 Unlike our earlier examples, pix2pix is not a generative model.

 This means it does not have a latent space or a corresponding

 space of natural images. Instead, there is a neural network,

 called, confusingly, a generator – this is not meant in the

 same sense as our earlier generative models – that takes as

 input the constraint image, and produces as output the filled-in

 image.

 The generator is trained adversarially against a discriminator

 network, whose job is to distinguish between pairs of images

 generated from real data, and pairs of images generated by the

 generator.

 While this sounds similar to a conventional GAN, there is a

 crucial difference: there is no latent vector input to the

 generatorActually, Isola *et

 al* experimented with adding such a latent vector to

 the generator, but found it made little difference to the

 resulting images.. Rather, there is simply an input

 constraint. When a human inputs a constraint unlike anything seen

 in training, the network is forced to improvise, doing the best it

 can to interpret that constraint according to the rules it has

 previously learned. The creativity is the result of a forced

 merger of knowledge inferred from the training data, together with

 novel constraints provided by the user. As a result, even

 relatively simple ideas – like the bread- and beholder-cats

 – can result in striking new types of images, images not

 within what we would previously have considered the space of

 natural images.

Conclusion

----------

 It is conventional wisdom that AI will change how we interact with

 computers. Unfortunately, many in the AI community greatly

 underestimate the depth of interface design, often regarding it as

 a simple problem, mostly about making things pretty or

 easy-to-use. In this view, interface design is a problem to be

 handed off to others, while the hard work is to train some machine

 learning system.

 This view is incorrect. At its deepest, interface design means

 developing the fundamental primitives human beings think and

 create with. This is a problem whose intellectual genesis goes

 back to the inventors of the alphabet, of cartography, and of

 musical notation, as well as modern giants such as Descartes,

 Playfair, Feynman, Engelbart, and Kay. It is one of the hardest,

 most important and most fundamental problems humanity grapples

 with.

 As discussed earlier, in one common view of AI our computers will

 continue to get better at solving problems, but human beings will

 remain largely unchanged. In a second common view, human beings

 will be modified at the hardware level, perhaps directly through

 neural interfaces, or indirectly through whole brain emulation.

 We've described a third view, in which AIs actually change

 humanity, helping us invent new cognitive technologies, which

 expand the range of human thought. Perhaps one day those

 cognitive technologies will, in turn, speed up the development of

 AI, in a virtuous feedback cycle:

 It would not be a Singularity in machines. Rather, it would be a

 Singularity in humanity's range of thought. Of course, this loop

 is at present extremely speculative. The systems we've described

 can help develop more powerful ways of thinking, but there's at

 most an indirect sense in which those ways of thinking are being

 used in turn to develop new AI systems.

 Of course, over the long run it's possible that machines will

 exceed humans on all or most cognitive tasks. Even if that's the

 case, cognitive transformation will still be a valuable end, worth

 pursuing in its own right. There is pleasure and value involved

 in learning to play chess or Go well, even if machines do it

 better. And in activities such as story-telling the benefit often

 isn't so much the artifact produced as the process of construction

 itself, and the relationships forged. There is intrinsic value in

 personal change and growth, apart from instrumental benefits.

 The interface-oriented work we've discussed is outside the

 narrative used to judge most existing work in artificial

 intelligence. It doesn't involve beating some benchmark for a

 classification or regression problem. It doesn't involve

 impressive feats like beating human champions at games such as

 Go. Rather, it involves a much more subjective and

 difficult-to-measure criterion: is it helping humans think and

 create in new ways?

 This creates difficulties for doing this kind of work,

 particularly in a research setting. Where should one publish?

 What community does one belong to? What standards should be

 applied to judge such work? What distinguishes good work from

 bad?

 We believe that over the next few years a community will emerge

 which answers these questions. It will run workshops and

 conferences. It will publish work in venues such as Distill. Its

 standards will draw from many different communities: from the

 artistic and design and musical communities; from the mathematical

 community's taste in abstraction and good definition; as well as

 from the existing AI and IA communities, including work on

 computational creativity and human-computer interaction. The

 long-term test of success will be the development of tools which

 are widely used by creators. Are artists using these tools to

 develop remarkable new styles? Are scientists in other fields

 using them to develop understanding in ways not otherwise

 possible? These are great aspirations, and require an approach

 that builds on conventional AI work, but also incorporates very

 different norms.

Augmenting Human Intellect: A Conceptual Framework

deeplearn.js font demo

Auto-encoding variational Bayes

Analyzing 50k fonts using deep neural networks

Autoencoding beyond pixels using a learned similarity metric

Sampling Generative Networks

Writing with the Machine

Automatic chemical design using a data-driven continuous representation of molecules

Generative visual manipulation on the natural image manifold

Generative adversarial nets

A Neural Representation of Sketch Drawings

Real-time human interaction with supervised learning algorithms for music composition and performance

TopoSketch: Drawing in Latent Space

Taking The Robots To Design School, Part 1

Hierarchical Variational Autoencoders for Music

Computational creativity: the final frontier?

Interactive machine learning: letting users build classifiers

Eccentric School of Painting Increased Its Vogue in the Current Art Exhibition — What Its Followers Attempt to Do

Genius: The Life and Science of Richard Feynman

Thought as a Technology

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Image-to-Image Translation with Conditional Adversarial Networks


Activation Atlas

Shan Carter
Zan Armstrong
Ludwig Schubert
Ian Johnson
Chris Olah

Introduction

------------

Neural networks can learn to classify images more accurately than

any system humans directly design. This raises a natural question: What

have these networks learned that allows them to classify images so well?

Feature visualization is a thread of research that tries to

answer this question by letting us "see through the eyes" of the network

 and trying to determine what they respond to. Because neurons don't

 But there was still a problem — what combinations of neurons should we

These approaches are exciting because they can make the hidden

layers of networks comprehensible. These layers are the heart of how

neural networks outperform more traditional approaches to machine

learning and historically, we've had little understanding of what

Unfortunately, visualizing activations has a major

weakness — it is limited to seeing only how the network sees a single

input. Because of this, it doesn't give us a big picture view of the

network. When what we want is a map of an entire forest, inspecting one

tree at a time will not suffice.

 gives a global view of a dataset by taking each image and organizing

 them by their activation values from a neural network.

 Showing which images the model sees as similar does help us infer

some ideas about what features the network is responding to,

 but feature visualization makes those connections much more

explicit.

 Nguyen, et al  use t-SNE to make more diverse neuron visualizations,

In this article we introduce *activation atlases* to this

quiver of techniques. (An example is shown at the top of this article.)

Broadly speaking, we use a technique similar to the one in CNN codes,

but instead of showing input data, we show feature visualizations of

averaged activations. By combining these two techniques, we can get the

advantages of each in one view — a global map seen through the eyes of

the network.

 would give us the global view of a network that we are seeking. In

practice, however, neurons are rarely used by the network in isolation,

and it may be difficult to understand them that way. As an analogy,

while the 26 letters in the alphabet provide a basis for English, seeing

 how letters are commonly combined to make words gives far more insight

into the concepts that can be expressed than the letters alone.

Similarly, activation atlases give us a bigger picture view by showing

common combinations of neurons.

#### Individual Neurons

![](Activation%20Atlas_files/overview-neuron.jpg)

![diagram](Activation%20Atlas_files/manifold-1.jpg)

 make hidden layers somewhat meaningful, but misses interactions between

 neurons — it only shows us one-dimensional, orthogonal probes of the

high-dimensional activation space.

#### Pairwise Interactions

![](Activation%20Atlas_files/overview-pairwise.jpg)

![diagram](Activation%20Atlas_files/manifold-2.jpg)

 reveal interaction effects, but they only show two-dimensional slices

of a space that has hundreds of dimensions, and many of the combinations

 are not realistic.

#### Spatial Activations

![](Activation%20Atlas_files/activation-grid.png)

![diagram](Activation%20Atlas_files/manifold-3.jpg)

 show us important combinations of many neurons by sampling the

sub-manifold of likely activations, but they are limited to those that

occur in the given example image.

#### Activation Atlas

![](Activation%20Atlas_files/overview-atlas.jpg)

![diagram](Activation%20Atlas_files/manifold-4.jpg)

 These atlases not only reveal visual abstractions within a model,

but later in the article we will show that they can reveal high-level

misunderstandings in a model that can be exploited. For example, by

looking at an activation atlas we will be able to see why a picture of a

 baseball can switch the classification of an image from "grey whale" to

 "great white shark".

![great white shark / grey whale](Activation%20Atlas_files/whale-baseball.png)

|  |  |  |

| --- | --- | --- |

| 1. | grey whale | 91.0% |

| 2. | killer whale | 7.5% |

| 3. | great white shark | 0.7% |

| 4. | gar | 0.4% |

|  |  |  |

| --- | --- | --- |

| 1. | great white shark | 66.7% |

| 2. | baseball | 7.4% |

| 3. | grey whale | 4.1% |

| 4. | sombrero | 3.2% |

Of course, activation atlases do have limitations. In particular,

they're dependent on the distribution of the data we choose to sample

activations from (in our examples, we use one million images chosen at

random from the ImageNet dataset

 training data). As a result, they will only show the activations that

exist within the distribution of the sample data. However, while it's

important to be aware of these limitations — we'll discuss them in much

more depth later! — Activation Atlases still give us a new kind of

overview of what neural networks can represent.

Looking at a Single Image

-------------------------

Before we dive into Activation Atlases, let's briefly review how we

 use feature visualization to make activation vectors meaningful ("see

 (also known as "GoogLeNet").

 InceptionV1 consists of a number of layers, which we refer to as

"mixed3a", "mixed3b", "mixed4a", etc., and sometimes shortened to just

"3a". Each layer successively builds on the previous layers.

![model](Activation%20Atlas_files/model.svg)

To visualize how InceptionV1 sees an image, the first step is to

feed the image into the network and run it through to the layer of

interest. Then we collect the activations — the numerical values of how

much each neuron fired. If a neuron is excited by what it is shown, its

activation value will be positive.

Unfortunately these vectors of activation values are just

vectors of unitless numbers and not particularly interpretable by

 comes in.

 Roughly speaking, we can think of feature visualization as creating an

 idealized image of what the network thinks would produce a particular

activation vector. Whereas we normally use a network to transform an

image into an activation vector, in feature visualization we go in the

opposite direction. Starting with an activation vector at a particular

layer, we create an image through an iterative optimization process.

 there is not just one activation vector per layer per image.

 This means that the same neurons are run on each patch of the previous

 layer.

 Thus, when we pass an entire image through the network, each neuron

will be evaluated hundreds of times, once for each overlapping patch of

the image. We can consider the vectors of how much each neuron fired for

 each patch separately.

The result is a grid of feature visualizations, one for each

 patch. This shows us how the network sees different parts of the input

image.

![input image](Activation%20Atlas_files/dogcat.jpg)

Input image from ImageNet.

![activation grid](Activation%20Atlas_files/dogcat-grid.jpg)

Activation grid from InceptionV1, layer mixed4d.

Aggregating Multiple Images

---------------------------

Activation grids show how the network sees a single image, but what

 if we want to see more? What if we want to understand how it reacts to

millions of images?

Of course, we could look at individual activation grids for

those images one by one. But looking at millions of examples doesn't

scale, and human brains aren't good at comparing lots of examples

without structure. In the same way that we need a tool like a histogram

in order to understand millions of numbers, we need a way to aggregate

and organize activations if we want to see meaningful patterns in

millions of them.

 This gives us one million activation vectors. Each of the vectors is

high-dimensional, perhaps 512 dimensions! With such a complex set of

data, we need to organize and aggregate it if we want a big picture

view.

 can project high-dimensional data like our collection of activation

vectors into useful 2D layouts, preserving some of the local structure

of the original space. This takes care of organizing our activation

vectors, but we also need to aggregate into a more manageable number of

elements — one million dots would be hard to interpret. We'll do this by

 drawing a grid over the 2D layout we created with dimensionality

reduction. For each cell in our grid, we average all the activations

that lie within the boundaries of that cell, and use feature

visualization to create an iconic representation.

[0.700, 0.498, …]

 [0.818, 0.437, …]

 [0.421, 0.027, …]

The activations are fed through UMAP to

reduce them to two dimensions. They are then plotted, with similar

activations placed near each other.

We then draw a grid and average the

activations that fall within a cell and run feature inversion on the

averaged activation. We also optionally size the grid cells according to

 the density of the number of activations that are averaged within.

 However, we use a slightly non-standard objective.

 We find it helpful to use an objective that emphasizes angle more

heavily by multiplying the dot product by cosine similarity, leading to

objectives of the following form:

 We don't yet fully understand this phenomenon.

 For each activation vector, we also compute an *attribution*

vector.

 The attribution vector has an entry for each class, and approximates

 the amount that the activation vector influenced the logit for each

class.

 Attribution vectors generally depend on the surrounding context.

 This is similar to Grad-CAM ,

 but without the spatial averaging of the gradient.

 Instead, we reduce noise in the gradient by using a continuous

relaxation of the gradient for max pooling in computing the gradient (as

 in ).

 This average attribution can be thought of as showing what classes

that cell tends to support, marginalizing over contexts.

 At early layers, the average attribution is very small and the top

classes are fairly arbitrary because low-level visual features like

textures tend to not be very discriminative without context.

Grid size:

 20x20

 40x40

 80x80

 160x160

 attribution labels

This atlas can be a bit overwhelming at first

glance — there's a lot going on! This diversity is a reflection of the

variety of abstractions and concepts the model has developed. Let's take

 a tour to examine this atlas in more depth.

If we look at the top-left of the atlas, we see images which

look like animal heads. There is some differentiation between different

types of animals, but it seems to be more a collection of elements of

generic mammals — eyes, fur, noses — rather than a collection of

different classes of animals. We've also added labels that show which

class each averaged activation most contributes to. Please note, in some

 areas of a layer this early in the network these attribution labels can

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

Below the feet we start to lose any identifiable parts of

animals, and see isolated grounds and floors. We see attribution toward

environments like "sandbar" and also toward things that are found on the

 ground, like "doormat" or "ant".

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

These sandy, rocky backgrounds slowly blend into beaches and

 bodies of water. Here we see lakes and oceans, both above and below

water. Though the network does have certain classes like "seashore", we

see attribution toward many sea animals, without any visual references

to the animals themselves. While not unexpected, it is reassuring to see

 that the activations that are used to identify the sea for the class

"seashore" are the same ones used when classifying "starfish" or "sea

lion". There is also no real distinction at this point between lakes and

 ocean — "lakeside" and "hippopotamus" attributions are intermingled

with "starfish" and "stingray".

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

Now let's jump to the other side of the atlas, where we can

see many variations of text detectors. These will be useful when

identifying classes such as "menu", "web site" or "book jacket".

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

Moving upward, we see many variations of people. There are

very few classes that specifically identify people in ImageNet, but

people are present in lots of the images. We see attribution toward

things people use ("hammer", "flute"), clothes that people wear ("bow

tie", "maillot") and activities that people participate in

("basketball"). There is a uniformity to the skin color in these

visualizations which we suspect is a reflection of the distribution of

the data used for training. (You can browse the ImageNet training data

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

And finally, moving back to the left, we can see round food

and fruit organized mostly by colors — we see attribution toward

"lemon", "orange" and "fig".

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

We can also trace curved paths through this manifold that

we've created. Not only are regions important, but certain movements

through the space seem to correspond to human interpretable qualities.

With the fruit, we can trace a path that seems to correlate with the

size and number of fruits in the frame.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

Similarly, with people, we can trace a path that seems to

correspond to how many people are in the frame, whether it's a single

person or a crowd.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

In the plants region, we can trace a path that seems to

correspond to how blurry the plant is. This could possibly be used to

determine relative size of objects because of the typical focal lengths

of cameras. Close up photos of small insects have more opportunity for

blurry background foliage than photos of larger animals, like monkeys.

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

It is important to note that these paths are constructed

after the fact in the low-dimensional projection. They are smooth paths

in this reduced projection but we don't necessarily know how the paths

operate in the original higher-dimensional activation space.

Looking at Multiple Layers

--------------------------

In the previous section we focused on one layer of the network,

mixed4c, which is in the middle of our network. Convolutional networks

are generally deep, consisting of many layers that progressively build

up more powerful abstractions. In order to get a holistic view, we must

look at how the model's abstractions develop over several layers.

![model](Activation%20Atlas_files/model2.svg)

To start, let's compare three layers from different areas of

 the network to try to get a sense for the different personalities of

 .vertical-layer-comparison {

 display: grid;

 grid-column: page;

 grid-template-columns: 1fr 1fr 1fr;

 grid-gap: 16px 20px;

 }

 .vertical-layer-comparison h4 {

 margin-bottom: 0;

 padding-bottom: 4px;

 width: 100%;

 border-bottom: solid #CCC 1px;

 }

 .vertical-layer-comparison.progress {

 display: block;

 max-width: 704px;

 padding: 0 20px;

 margin: 0 auto;

 }

 .vertical-layer-comparison.progress > div {

 display: grid;

 grid-template-columns: 2.5fr 1fr;

 grid-gap: 20px;

 margin-bottom: 16px;

 }

 .vertical-layer-comparison.progress h4 {

 margin-top: 0;

 margin-bottom: 16px;

 }

 .vertical-layer-comparison.progress .figcaption {

 margin-top: 8px;

 }

#### Mixed3b

#### Mixed4c

#### Mixed5b

![](Activation%20Atlas_files/layers-plant-0.jpg)

![thumbnail for mixed3b](Activation%20Atlas_files/thumbnail-mixed3b.jpg)

You'll immediately notice that

the early layer is very nonspecific in comparison to the others. The

icons that emerge are of patterns and splotches of color. It is

suggestive of the final class, but not particularly evocative.

![](Activation%20Atlas_files/layers-plant-1.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

By the middle layer, icons

definitely resemble leaves, but they could be any type of plant.

Attributions are focused on plants, but are a little all over the board.

![](Activation%20Atlas_files/layers-plant-2.jpg)

![thumbnail for mixed5b](Activation%20Atlas_files/thumbnail-mixed5b.jpg)

Here we see foliage with

textures that are specific to cabbage, and curved into rounded balls.

There are full heads of cabbage rather than individual leaves.

As you move through the network, the later layers seem to

get much more specific and complex. This is to be expected, as each

layer builds its activations on top of the preceding layer's

activations. The later layers also tend to have larger receptive fields

than the ones that precede them (meaning they are shown larger subsets

of the image) so the concepts seem to encompass more of the whole of

objects.

There is another phenomenon worth noting: not only are concepts

 being refined, but new concepts are appearing out of combinations of

old ones. Below, you can see how sand and water are distinct concepts in

 a middle layer, mixed4c, both with strong attributions to the

classification of "sandbar". Contrast this with a later layer, mixed5b,

where the two ideas seem to be fused into one activation.

#### Mixed4c

#### Mixed5b

![](Activation%20Atlas_files/layers-water-0.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

![](Activation%20Atlas_files/layers-water-1.jpg)

![thumbnail for mixed4c](Activation%20Atlas_files/thumbnail-mixed4c.jpg)

…and a separate area of **watery textures**.

![](Activation%20Atlas_files/layers-water-2.jpg)

![thumbnail for mixed5b](Activation%20Atlas_files/thumbnail-mixed5b.jpg)

Finally, if we zoom out a little, we can see how the broader

 shape of the activation space changes from layer to layer. By looking

at similar regions in several consecutive layers, we can see concepts

getting refined and differentiated — In mixed4a we see very vague,

generic blob, which gets refined into much more specific "peninsulas" by

 mixed4e.

![mixed4a](Activation%20Atlas_files/progress-mixed4a.jpg)

#### Mixed4a

 In mixed4a there is a vague "mammalian" area.

![mixed4b](Activation%20Atlas_files/progress-mixed4b.jpg)

#### Mixed4b

![mixed4c](Activation%20Atlas_files/progress-mixed4c.jpg)

#### Mixed4c

![mixed4d](Activation%20Atlas_files/progress-mixed4d.jpg)

#### Mixed4d

 The specialization continues in mixed4d.

![mixed4e](Activation%20Atlas_files/progress-mixed4e.jpg)

#### Mixed4e

 And further still in mixed4e.

Below you can browse many more of the layers of InceptionV1. You can compare the

 [curved edge detectors of mixed4a](#) with the

 [bowls and cups of mixed5b](#). Mixed4b has some

 [fabrics](#). In later layers, you'll see

 [specific types of clothing](#).

Class Filter

------------

show all

fireboat

streetcar

speedboat

scuba diver

snorkel

dalmatian

Labrador retriever

Siberian husky

English setter

Doberman

Tibetan terrier

sea lion

great white shark

goldfish

lionfish

oil filter

digital clock

digital watch

comic book

spider web

barn spider

head cabbage

artichoke

yellow lady's slipper

croquet ball

volleyball

parachute

sleeping bag

lakeside

coral reef

seashore

vine snake

water snake

Indian cobra

green mamba

goldfinch

house finch

vulture

lorikeet

peacock

Layer

-----

mixed3a

mixed3b

mixed4a

mixed4b

mixed4c

mixed4d

mixed4e

mixed5a

mixed5b

![thumbnail for mixed4e](Activation%20Atlas_files/thumbnail-mixed4e.jpg)

 attribution labels

 show tooltip

 scroll to zoom

### Grid size

 20x20

 40x40

 80x80

 160x160

 320x320

 auto

auto threshold: 0.5

### Icon density: 1

### Class filter

 positive influence

 negative influence

Intensity: 1

### Location

x: 0.500

y: 0.500

scale: 1.000

fewer options

Focusing on a Single Classification

-----------------------------------

Looking at an atlas of all activations can be a little

overwhelming, especially when you're trying to understand how the

network goes about ranking one particular class. For instance, let's

investigate how the network classifies a "fireboat".

![fireboat](Activation%20Atlas_files/fireboat-01.jpg)

An image labeled "fireboat" from ImageNet.

We'll start by looking at an atlas for the last layer,

mixed5b. Instead of showing all the activations, however, we'll

calculate the amount that each activation contributes toward a

classification of "fireboat" and then map that value to the opacity of

the activation icon.In the case of

mixed5b, determining this contribution is fairly straightforward because

 the relationship between activations at mixed5b and the logit values is

 linear. When there are multiple layers between our present one and the

output — and as a result, the relationship is non-linear — it's a little

 less clear what to do. In this article, we take the simple approach of

forming a linear approximation of these future layers and use it to

approximate the effect of our activations. The areas that

contribute a lot toward a classification of "fireboat" will be clearly

visible, whereas the areas that contribute very little (or even

contribute negatively) will be completely transparent.

![](Activation%20Atlas_files/focus-1-1.jpg)

![](Activation%20Atlas_files/focus-1-2.jpg)

When we map opacity to the

amount that each activation contributes to "fireboat" in the layer

mixed5b, we see a main cluster of icons showing red boats and splashing,

 spraying water. While there are some stray areas elsewhere, it seems

that this is region of the atlas that is dedicated specifically to

classifying red boats with splashing water nearby.

The layer we just looked at, mixed5b, is located just before

 the final classification layer so it seems reasonable that it would be

closely aligned with the final classes. Let's look at a layer a little

earlier in the network, say mixed4d, and see how it differs.

![](Activation%20Atlas_files/focus-2-1.jpg)

3

2

1

![](Activation%20Atlas_files/focus-2-4.jpg)

1

![](Activation%20Atlas_files/focus-2-3.jpg)

2

![](Activation%20Atlas_files/focus-2-2.jpg)

3

In mixed4d we see we see the

attribution toward "fireboat" is high in several clusters located in

different positions around the atlas. One is very focused on windows,

another on geysers and splashing water, and yet another on crane-like

objects.

Here we see a much different pattern. If we look at some

more input examples, this seems entirely reasonable. It's almost as if

we can see a collection of the component concepts the network will use

in later layers to classify "fireboat". Windows + crane + water =

"fireboat".

![fireboat](Activation%20Atlas_files/fireboat-10.jpg)

![fireboat](Activation%20Atlas_files/fireboat-02.jpg)

![fireboat](Activation%20Atlas_files/fireboat-03.jpg)

One of the clusters, the one with windows, has strong

attribution to "fireboat", but taken on its own, it has an even stronger

 attribution toward "streetcar". So, let's go back to the atlas at

mixed4d, but isolate "streetcar" and compare it to the patterns seen for

 "fireboat". Let's look more closely at the four highlighted areas: the

three areas we highlighted for fireboat plus one additional area that is

 highly activated for streetcars.

#### Activations for Fireboat

![](Activation%20Atlas_files/focus-2-1.jpg)

1234

#### Activations for Streetcar

![](Activation%20Atlas_files/focus-3-1.jpg)

1234

If we zoom in, we can get a better look at what

distinguishes the two classifications at this layer. (We've

cherry-picked these examples for brevity, but you can explore all the

layers and activations in detail in a explorable playground below.)

#### Fireboat

![](Activation%20Atlas_files/focus-2-2.jpg)

1![](Activation%20Atlas_files/focus-2-3.jpg)

2![](Activation%20Atlas_files/focus-2-4.jpg)

3![](Activation%20Atlas_files/focus-2-5.jpg)

4

#### Streetcar

![](Activation%20Atlas_files/focus-3-2.jpg)

1![](Activation%20Atlas_files/focus-3-3.jpg)

2![](Activation%20Atlas_files/focus-3-4.jpg)

3![](Activation%20Atlas_files/focus-3-5.jpg)

4

"Fireboat" activations have

much stronger attributions from water than "streetcar", where there is

virtually no positive evidence.

If we look at a couple of input examples, we can see how

buildings and water backgrounds are an easy way to differentiate between

 a "fireboat" and a "streetcar".

![fireboat](Activation%20Atlas_files/fireboat-01.jpg)

![streetcar](Activation%20Atlas_files/streetcar-01.jpg)

Images from ImageNet

By isolating the activations that contribute strongly to one

 class and comparing it to other class activations, we can see which

activations are conserved among classes and which are recombined to form

 more complex activations in later layers. Below you can explore the

activation patterns of many classes in ImageNet through several layers

of InceptionV1. You can even explore negative attributions, which we

ignored in this discussion.

Class Filter

------------

show all

fireboat

streetcar

speedboat

scuba diver

snorkel

dalmatian

Labrador retriever

Siberian husky

English setter

Doberman

Tibetan terrier

sea lion

great white shark

goldfish

lionfish

oil filter

digital clock

digital watch

comic book

spider web

barn spider

head cabbage

artichoke

yellow lady's slipper

croquet ball

volleyball

parachute

sleeping bag

lakeside

coral reef

seashore

vine snake

water snake

Indian cobra

green mamba

goldfinch

house finch

vulture

lorikeet

peacock

Layer

-----

mixed3a

mixed3b

mixed4a

mixed4b

mixed4c

mixed4d

mixed4e

mixed5a

mixed5b

![thumbnail for mixed4d](Activation%20Atlas_files/thumbnail-mixed4d.jpg)

 attribution labels

 show tooltip

 scroll to zoom

### Grid size

 20x20

 40x40

 80x80

 160x160

 320x320

 auto

auto threshold: 0.5

### Icon density: 1

### Class filter

 positive influence

 negative influence

Intensity: 1

### Location

x: 0.500

y: 0.500

scale: 1.000

fewer options

Further Isolating Classes

-------------------------

Highlighting the class-specific activations in situ of a full atlas

 is helpful for seeing how that class relates to the full space of what a

 network "can see." However, if we want to really isolate the

activations that contribute to a specific class we can remove all the

other activations rather than just dimming them, creating what we'll

 class activations we generally have better results from using t-SNE for

 the dimensionality reduction step rather than UMAP. We suspect it is

#### Class Activation Atlas for "snorkel" from mixed5b

 scuba diver

 snorkel

 red fox

 ibex

 giant panda

 lesser panda

 zebra

 tiger

 fireboat

 dalmatian

 English setter

 great white shark

 goldfish

 lionfish

 comic book

 plunger

 knot

 tile roof

 pill bottle

 measuring cup

 crossword puzzle

 cauliflower

 head cabbage

 artichoke

 green mamba

 goldfinch

 house finch

 vulture

 chainlink fence

 wok

 grey whale

 waffle iron

 conch

 snowmobile

 sax

 French horn

 stage

 spider web

 barn spider

 grasshopper

 monarch

A class activation atlas gives us a much clearer view of

which detectors the network is using to rank a specific class. In the

"snorkel" example we can clearly see ocean, underwater, and colorful

masks.

In the previous example, we are only showing those activations

whose strongest attribution is toward the class in question. This will

show us activations that contribute mostly to our class in question,

even if their overall strength is low (like in background detectors). In

 some cases, though, there are strong correlations that we'd like to see

 (like fish with snorkelers). These activations on their own might

contribute more strongly to a different class than the one we're

interested in, but their existence can also contribute strongly to our

class of interest. For these we need to choose a different filtering

method.

#### "snorkel" filtered by top rank

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

snorkel

We pluck only those

activations whose top attribution is toward the class in question. The

results are often much more focused and isolated, exclusive to the

class. Some are low magnitude, like backgrounds, and we miss

correlations or concepts that are shared among many classes.

#### "snorkel" filtered by overall magnitude

drake

red-breasted merganser

swimming trunks

maillot

balance beam

dugong

killer whale

snorkel

snorkel

snorkel

canoe

rock beauty

puffer

dugong

snorkel

snorkel

bathing cap

lifeboat

canoe

rock beauty

vine snake

agama

scuba diver

snorkel

snorkel

green mamba

snorkel

snorkel

bathing cap

neck brace

snorkel

oxygen mask

gasmask

football helmet

neck brace

water bottle

oxygen mask

oxygen mask

sunglass

ballpoint

swab

sunglass

sunglasses

Here we sort all the

activations by the magnitude toward the class in question (independent

of other classes) and take the top 2,000 activations. We see more

correlated activations that could, on their own, contribute to another

classification (We label each activation with the class that it

contributes toward most). Notice that certain fish that are commonly

seen while snorkeling now show up with that class, and a "coffee mug"

now shows up with "crossword puzzle". Some of them appear spurious,

however.

 scuba diver

 snorkel

 red fox

 ibex

 giant panda

 lesser panda

 zebra

 tiger

 fireboat

 dalmatian

 English setter

 great white shark

 goldfish

 lionfish

 comic book

 plunger

 knot

 tile roof

 pill bottle

 measuring cup

 crossword puzzle

 cauliflower

 head cabbage

 artichoke

 green mamba

 goldfinch

 house finch

 vulture

 chainlink fence

 wok

 grey whale

 waffle iron

 conch

 snowmobile

 sax

 French horn

 stage

 spider web

 barn spider

 grasshopper

 monarch

Using the magnitude filtering method, let's try to compare two

related classes and see if we can more easily see what distinguishes

them. (We could have instead used rank, or a combination of the two, but

 magnitude will suffice to show us a good variety of concepts).

#### "snorkel"

#### "scuba diver"

It can be a little hard to immediately understand all the

differences between classes. To help make the comparison easier, we can

combine the two views into one. We'll plot the difference between the

attributions of the "snorkel" and "scuba diver" horizontally, and use

t-SNE  to cluster similar activations vertically.

In this comparison we can see some bird-like creatures and

clear tubes on the left, implying a correlation with "snorkel", and some

 shark-like creatures and something round, shiny, and metallic on the

right, implying correlation with "scuba diver" (This activation has a

strong attribution toward the class "steam locomotive"). Let's take an

image from the ImageNet dataset labeled as "snorkel" and add something

that resembles this icon to see how it affects the classification

scores.

![snorkel / scuba diver](Activation%20Atlas_files/snorkel-train.png)

|  |  |  |

| --- | --- | --- |

| 1. | snorkel | 55.6% |

| 2. | coral reef | 18.6% |

| 3. | scuba diver | 13.5% |

| 4. | loggerhead | 5.5% |

| 5. | lionfish | 1.7% |

| 6. | sea snake | 1.4% |

![snorkel / scuba diver](Activation%20Atlas_files/snorkel-train-medium.png)

|  |  |  |

| --- | --- | --- |

| 1. | scuba diver | 71.3% |

| 2. | coral reef | 14.7% |

| 3. | snorkel | 4.5% |

| 4. | lionfish | 3.2% |

| 5. | sea snake | 2.4% |

| 6. | loggerhead | 0.9% |

By adding a picture of one

of the concepts seen in the visualization above we can change the

classification. With an added picture of a steam train the confidence of

 "scuba diver" classification rises and "snorkel" drops significantly.

![snorkel / scuba diver](Activation%20Atlas_files/snorkel-train-large.png)

|  |  |  |

| --- | --- | --- |

| 1. | steam locomotive | 89.6% |

| 2. | snowplow | 6.2% |

| 3. | jeep | 0.8% |

| 4. | tractor | 0.5% |

| 5. | scuba diver | 0.4% |

| 6. | passenger car | 0.3% |

The failure mode here seems to be that the model is using

its detectors for the class "steam locomotive" to identify air tanks to

help classify "scuba diver". We'll call these "multi-use"

features — detectors that react to very different concepts that are

nonetheless visually similar. Let's look at the differences between a

"grey whale" and a "great white shark" to see another example of this

issue.

In this example we see another detector that seems to be

playing two roles: detecting red stitching on a baseball and a sharks's

![great white shark / grey whale](Activation%20Atlas_files/whale-baseball.png)

|  |  |  |

| --- | --- | --- |

| 1. | grey whale | 91.0% |

| 2. | killer whale | 7.5% |

| 3. | great white shark | 0.7% |

| 4. | gar | 0.4% |

| 5. | sea lion | 0.1% |

| 6. | tiger shark | 0.1% |

|  |  |  |

| --- | --- | --- |

| 1. | great white shark | 66.7% |

| 2. | baseball | 7.4% |

| 3. | grey whale | 4.1% |

| 4. | sombrero | 3.2% |

| 5. | sea lion | 3.1% |

| 6. | killer whale | 2.7% |

|  |  |  |

| --- | --- | --- |

| 1. | baseball | 100.0% |

| 2. | rugby ball | 0.0% |

| 3. | golf ball | 0.0% |

| 4. | ballplayer | 0.0% |

| 5. | drum | 0.0% |

| 6. | sombrero | 0.0% |

The results follow the pattern in previous examples pretty

closely.

 Adding a small-sized baseball does change the top classification to

"great white shark", and as it gets bigger it overpowers the

classification, so the top slot goes to "baseball".

Let's look at one more example: "frying pan" and "wok".

One difference stands out here — the type of related foods

present. On the right we can clearly see something resembling noodles

(which have a strong attribution toward the class "carbonara"). Let's

take a picture from ImageNet labeled as "frying pan" and add an inset of

 some noodles.

![frying pan / wok](Activation%20Atlas_files/frying-pan-noodles.png)

|  |  |  |

| --- | --- | --- |

| 1. | frying pan | 76.5% |

| 2. | wok | 15.8% |

| 3. | stove | 5.4% |

| 4. | spatula | 1.0% |

| 5. | Dutch oven | 0.5% |

| 6. | mixing bowl | 0.2% |

![frying pan / wok](Activation%20Atlas_files/frying-pan-noodles-medium.png)

|  |  |  |

| --- | --- | --- |

| 1. | wok | 63.2% |

| 2. | frying pan | 35.1% |

| 3. | spatula | 0.6% |

| 4. | hot pot | 0.5% |

| 5. | mixing bowl | 0.1% |

| 6. | stove | 0.1% |

![frying pan / wok](Activation%20Atlas_files/frying-pan-noodles-large.png)

|  |  |  |

| --- | --- | --- |

| 1. | carbonara | 96.4% |

| 2. | plate | 2.3% |

| 3. | wok | 0.6% |

| 4. | meat loaf | 0.1% |

| 5. | dishwasher | 0.1% |

| 6. | rotisserie | 0.1% |

As we make the picture of

noodles larger, its influence overpowers the other classifications, but

"wok" remains ranked above "frying pan".

Here the patch was not as effective at lowering the initial

classification, which makes sense since the noodle-like icons were

plotted more toward the center of the visualization thus having less of a

 difference in attribution. We suspect that the training set simply

contained more images of woks with noodles than frying pans with

noodles.

### Testing dozens of patches on thousands of images

So far we've only shown single examples of these patches. Below we

show the result of ten sample patches (each set includes the one example

 we explored above), run on 1,000 images from the ImageNet training set

for the class in question. While they aren't effective in all cases,

they do flip the image classification to the target class in about 2 in 5

 images. The success rate reaches about 1 in 2 images if we also allow

to position the patch in the best of the four corners of the image (top

left, top right, bottom left, bottom right) at the most effective size.

To ensure our attack isn't just blocking out evidence for the original

class, we also compare each attack to a random noise image patch.

 #adversarial-analysis {

 display: none;

 grid-column: text;

 margin-bottom: 0;

 }

 #adversarial-analysis table {

 width: 100%;

 text-align: right;

 }

 #adversarial-analysis table th {

 vertical-align: initial;

 }

 #adversarial-analysis table th .figcaption {

 display: block;

 font-weight: initial;

 }

 #adversarial-analysis table th img {

 width: initial;

 }

 #adversarial-analysis table .detail {

 /\* padding-left: 1em; \*/

 font-size: 85%;

 color: rgba(0, 0, 0, 0.6);

 }

 #adversarial-analysis table .synset {

 font-family: monospace;

 white-space: nowrap;

 }

 #adversarial-analysis table .from,

 #adversarial-analysis table .to {

 white-space: nowrap;

 }

 #adversarial-analysis table .overall {

 font-weight: bold;

 }

 @supports (font-variant-numeric: tabular-nums) {

 #adversarial-analysis table {

 font-variant-numeric: tabular-nums;

 }

 }

#### Dynamic corner position

| icon

Choosing the strongest size and corner position for each example. | Success rate

Mean probability assigned to the target class before and after patch. |

| --- | --- | --- |

| snorkel → scuba diver | 85.0% | 7,031 / 8,270 | +56.8% | 3.7% → 60.5% |

| frying pan → wok | 44.9% | 2,799 / 6,240 | +24.1% | 10.3% → 34.4% |

#### Consistent corner position

| icon

| --- | --- | --- |

| snorkel → scuba diver | 65.4% | 5,409 / 8,270 | +43.0% | 3.7% → 46.7% |

| frying pan → wok | 25.6% | 1,595 / 6,240 | +12.7% | 10.3% → 23.0% |

#### details for each patch tested

| snorkel → scuba divergrey whale → great white sharkfrying pan → wok

| --- | --- | --- |

| patch 0

Patch 01 | 88.1% | 729 / 827 | +60.3% | 3.7% → 64.0% |

| patch 1

Patch 02 | 83.4% | 690 / 827 | +53.0% | 3.7% → 56.7% |

| patch 2

Patch 03 | 92.4% | 764 / 827 | +66.9% | 3.7% → 70.6% |

| patch 3

Patch 04 | 93.0% | 769 / 827 | +66.7% | 3.7% → 70.4% |

| patch 4

Patch 05 | 88.0% | 728 / 827 | +58.8% | 3.7% → 62.5% |

| patch 5

Patch 06 | 85.5% | 707 / 827 | +52.0% | 3.7% → 55.7% |

| patch 6

Patch 07 | 86.2% | 713 / 827 | +57.4% | 3.7% → 61.2% |

| patch 7

Patch 08 | 56.1% | 464 / 827 | +30.7% | 3.7% → 34.5% |

| patch 8

Patch 09 | 92.0% | 761 / 827 | +64.8% | 3.7% → 68.5% |

| patch 9

Patch 10 | 85.4% | 706 / 827 | +57.6% | 3.7% → 61.3% |

| patch undefined

Random noise | 13.9% | 115 / 827 | +9.5% | 3.7% → 13.2% |

 Our "attacks" can be seen as part of a larger trend (eg. ) of researchers

 In many ways, our attacks are most similar to adversarial patches ,

 which also add a small patch to the input image.

 From this perspective, adversarial patches are far more effective,

working much more reliably. Instead, we see our attacks as interesting

because they are synthesized by humans from their understanding of the

model,

 and seem to be attacking the model at a higher level of abstraction.

 We also want to emphasize that not all class comparisons reveal

these type of patches and not all icons in the visualization have the

same (or any) effectiveness and we've only tested them on one model. If

we wanted to find these patches more systematically, a different

approach would most likely be more effective. However, the class

activation atlas technique was what revealed the existence of these

patches before we knew to look for them. If you'd like to explore your

own comparisons and search for your own patches, we've provided a

Conclusion and Future Work

--------------------------

Activation atlases give us a new way to peer into convolutional

vision networks. They give us a global, hierarchical, and

human-interpretable overview of concepts within the hidden layers. Not

only does this allow us to better see the inner workings of these

complicated systems, but it's possible that it could enable new

interfaces for working with images.

### Surfacing Inner Properties of Models

The vast majority of neural network research focuses on

quantitative evaluations of network behavior. How accurate is the model?

 What's the precision-recall curve?

 it behaves the way it does. To truly understand why a network behaves

the way it does, we would need to fully understand the rich inner world

of the network — it's hidden layers. For example, understanding better

how InceptionV1 builds up a classifier for a fireboat from component

parts in mixed4d can help us build confidence in our models and can

surface places where it isn't doing what we want.

Engaging with this inner world also invites us to do deep

learning research in a new way. Normally, each neural network experiment

 gives only a few bits of feedback — whether the loss went up or

down — to inform the next round of experiments. We design architectures

by almost blind trial and error, guided by vague intuitions that we

build up over years. In the future, we hope that researchers will get

rich feedback on what each layer in their model is doing in a way that

will make our current approach seem like stumbling in the dark.

Activation atlases, as they presently stand, are inadequate to

really help researchers iterate on models, in part because they aren't

comparable. If you look at atlases for two slightly different models,

it's hard to take away anything. In future work, we will explore how

similar visualizations can compare models, showing similarities and

differences beyond error rates.

### New interfaces

Machine learning models are usually deployed as black boxes that

automate a specific task, executing it on their own. But there's a

growing sense that there might be an alternate way for us to relate to

them: that instead of increasingly automating a task, they could be used

 more directly by a person. One vision of this augmentation that we find

 particularly compelling is the idea that the internal representations

 and music

  .

We think of activation atlases as revealing a machine-learned

alphabet for images — a collection of simple, atomic concepts that are

combined and recombined to form much more complex visual ideas. In the

same way that we use word processors to turn letters into words, and

words into sentences, we can imagine a tool that would allow us to

create images from a machine-learned language system for images. Similar

 to GAN painting , imagine

using something like an activation atlas as a palette — one could dip a

brush into a "tree" activation and paint with it. A palette of concepts

rather than colors.

 has shown that this is entirely possible. In this particular instance,

we imagine constructing a grid of activations by selecting them from an

atlas (or some derivation), then optimizing an output image that would

correspond to the user's constructed activation matrix.

 have shown us that we can use these vision networks to create nuanced

visual expression outside the explicit distribution of visual data it

was trained on. We speculate that activation atlases could be helpful in

 manipulating artistic styles without having to find an existing

reference image, or they could help in guiding and modifying automated

style transfer techniques.

We could also use these atlases to query large image datasets.

In the same way that we probe large corpuses of text with words, we

could, too, use activation atlases to find types of images in large

image datasets. Using words to search for something like a "tree" is

quite powerful, but as you get more specific, human language is often

ill-suited to describe specific visual characteristics. In contrast, the

 hidden layers of neural networks are a language optimized for the sole

purpose of representing visual concepts. Instead of using the proverbial

 thousand words to uniquely specify the image one is seeking, we can

imagine someone using the language of the activation atlas.

And lastly, we can also liken activation atlases to histograms.

 In the same way that traditional histograms give us good summaries of

large datasets, activation atlases can be used to summarize large

numbers of images.

In the examples in this article we used the same dataset for

training the model as we did for collecting the activations. But, if we

use a different dataset to collect the activations, we could use the

atlas as a way of inspecting an unknown dataset. An activation atlas

could show us a histogram of *learned concepts* that exist within

 the images. Such a tool could show us the semantics of the data and not

 just visual similarities, like showing histograms of common pixel

values.

While we are excited about the potential of activation atlases,

 we are even more excited at the possibility of developing similar

techniques for other types of models. Imagine having an array of machine

 learned, but human interpretable, languages for images, audio and text.

Visualizing higher-layer features of a deep network

Feature Visualization

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Understanding deep image representations by inverting them

The Building Blocks of Interpretability

t-SNE visualization of CNN codes

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Imagenet: A large-scale hierarchical image database

Going deeper with convolutions

Imagenet large scale visual recognition challenge

Visualizing data using t-SNE

UMAP: Uniform Manifold Approximation and Projection

Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization

Adversarial Patch

Adversarial examples in the physical world

A rotation and a translation suffice: Fooling cnns with simple transformations

Unrestricted adversarial examples

Intriguing properties of neural networks

Using Artificial Intelligence to Augment Human Intelligence

Visualizing Representations: Deep Learning and Human Beings

Machine Learning for Visualization

Image-to-image translation with conditional adversarial networks

TopoSketch: Drawing in Latent Space

Generative visual manipulation on the natural image manifold

ML as Collaborator: Composing Melodic Palettes with Latent Loops

A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

A neural algorithm of artistic style

Exploring Histograms