 With the growing success of neural networks, there is a corresponding need to be able to explain their decisions — including building confidence about how they will behave in the real-world, detecting model bias, and for scientific curiosity.

 In order to do so, we need to both construct deep abstractions and reify (or instantiate) them in rich interfaces .

 With a few exceptions , existing work on interpretability fails to do these in concert.

 The machine learning community has primarily focused on developing powerful methods, such as [feature visualization](https://distill.pub/2017/feature-visualization/) , attribution , and dimensionality reduction , for reasoning about neural networks.

 However, these techniques have been studied as isolated threads of research, and the corresponding work of reifying them has been neglected.

 On the other hand, the human-computer interaction community has begun to explore rich user interfaces for neural networks , but they have not yet engaged deeply with these abstractions.

 To the extent these abstractions have been used, it has been in fairly standard ways.

 As a result, we have been left with impoverished interfaces (e.g., saliency maps or correlating abstract neurons) that leave a lot of value on the table.

 Worse, many interpretability techniques have not been fully actualized into abstractions because there has not been pressure to make them generalizable or composable.

 In this article, we treat existing interpretability methods as fundamental and composable building blocks for rich user interfaces.

 We find that these disparate techniques now come together in a unified grammar, fulfilling complementary roles in the resulting interfaces.

 Moreover, this grammar allows us to systematically explore the space of interpretability interfaces, enabling us to evaluate whether they meet particular goals.

 We will present interfaces that show *what* the network detects and explain *how* it develops its understanding, while keeping the amount of information *human-scale*.

 For example, we will see how a network looking at a labrador retriever detects floppy ears and how that influences its classification.

 Our interfaces are speculative and one might wonder how reliable they are.

 Rather than address this point piecemeal, we dedicate a section to it at the end of the article.

 In this article, we use GoogLeNet, an image classification model, to demonstrate our interface ideas because its neurons seem unusually semantically meaningful.We’re actively investigating why this is, and hope to uncover principles for designing interpretable models. In the meantime, while we demonstrate our techniques on GoogLeNet, we provide code for you to try them on other models.

 Although here we’ve made a specific choice of task and network, the basic abstractions and patterns for combining them that we present can be applied to neural networks in other domains.

Making Sense of Hidden Layers
-----------------------------

 Much of the recent work on interpretability is concerned with a neural network’s input and output layers.
 Arguably, this focus is due to the clear meaning these layers have: in computer vision, the input layer represents values for the red, green, and blue color channels for every pixel in the input image, while the output layer consists of class labels and their associated probabilities.

 However, the power of neural networks lies in their hidden layers — at every layer, the network discovers a new representation of the input.

 In computer vision, we use neural networks that run the same feature detectors at every position in the image.

 We can think of each layer’s learned representation as a three-dimensional cube. Each cell in the cube is an *activation*, or the amount a neuron fires.

 The x- and y-axes correspond to positions in the image, and the z-axis is the channel (or detector) being run.

 The cube of activations that a neural network for computer vision develops at each hidden layer.

 Different slices of the cube allow us to target the activations of individual neurons, spatial positions, or channels.

 To make a semantic dictionary, we pair every neuron activation with a visualization of that neuron and sort them by the magnitude of the activation.

 This marriage of activations and feature visualization changes our relationship with the underlying mathematical object.

 Activations now map to iconic representations, instead of abstract indices, with many appearing to be similar to salient human ideas, such as “floppy ear,” “dog snout,” or “fur.”

 We use optimization-based feature visualization to avoid spurious correlation, but one could use other methods.

 Semantic dictionaries are powerful not just because they move away from meaningless indices, but because they express a neural network’s learned abstractions with canonical examples.

 With image classification, the neural network learns a set of visual abstractions and thus images are the most natural symbols to represent them.

 Were we working with audio, the more natural symbols would most likely be audio clips.

 This is important because when neurons appear to correspond to human ideas, it is tempting to reduce them to words.

 Doing so, however, is a lossy operation — even for familiar abstractions, the network may have learned a deeper nuance.

 For instance, GoogLeNet has multiple floppy ear detectors that appear to detect slightly different levels of droopiness, length, and surrounding context to the ears.

 There also may exist abstractions which are visually familiar, yet that we lack good natural language descriptions for: for example, take the particular column of shimmering light where sun hits rippling water.

 Moreover, the network may learn new abstractions that appear alien to us — here, natural language would fail us entirely!

 In general, canonical examples are a more natural way to represent the foreign abstractions that neural networks learn than native human language.

 By bringing meaning to hidden layers, semantic dictionaries set the stage for our existing interpretability techniques to be composable building blocks.

 As we shall see, just like their underlying vectors, we can apply dimensionality reduction to them.

 In other cases, semantic dictionaries allow us to push these techniques further.

 For example, besides the one-way attribution that we currently perform with the input and output layers, semantic dictionaries allow us to attribute to-and-from specific hidden layers.

 In principle, this work could have been done without semantic dictionaries but it would have been unclear what the results meant.

 While we introduce semantic dictionaries in terms of neurons, they can be used with any basis of activations. We will explore
 this more later.

What Does the Network See?
--------------------------

 Applying this technique to all the activation vectors allows us to not only see what the network detects at each position, but also what the network understands of the input image as a whole.

 And, by working across layers (eg. “mixed3a”, “mixed4d”), we can observe how the network’s understanding evolves: from detecting edges in earlier layers, to more sophisticated shapes and object parts in the latter.

 These visualizations, however, omit a crucial piece of information: the magnitude of the activations.

 By scaling the area of each cell by the magnitude of the activation vector, we can indicate how strongly the network detected features at that position:

How Are Concepts Assembled?
---------------------------

 Feature visualization helps us answer *what* the network detects, but it does not answer *how* the network assembles these individual pieces to arrive at later decisions, or *why* these decisions were made.

 Attribution is a set of techniques that answers such questions by explaining the relationships between neurons.

 There are a wide variety of approaches to attribution  but, so far, there doesn’t seem to be a clear right answer.

 In fact, there’s reason to think that all our present answers aren’t quite right .

 We think there’s a lot of important research to be done on attribution methods, but for the purposes of this article the exact approach taken to attribution doesn’t matter.

 We use a fairly simple method, linearly approximating the relationshipWe do attribution by linear approximation in all of our interfaces. That is, we estimate the effect of a neuron on the output is its activation times the rate at which increasing its activation increases the output. When we talk about a linear combination of activations, the attribution can be thought of as the linear combination of the attributions of the units, or equivalently as the dot product between the activation of that combination and the gradient.

 For spatial attribution, we do an additional trick. GoogLeNet’s strided max pooling introduces a lot of noise and checkerboard patterns to it’s gradients. To avoid our interface demonstrations being dominated by this noise, we (a) do a relaxation of the gradient of max pooling, distributing gradient to inputs proportional to their activation instead of winner takes all and (b) cancel out the checkerboard patterns.

The notebooks attached to diagrams provide reference implementations., but could easily substitute in essentially any other technique.

 Future improvements to attribution will, of course, correspondingly improve the interfaces built on top of them.

### Spatial Attribution with Saliency Maps

 The most common interface for attribution is called a *saliency map* — a simple heatmap that highlights pixels of the input image that most caused the output classification.

 We see two weaknesses with this current approach.

 First, it is not clear that individual pixels should be the primary unit of attribution.

 The meaning of each pixel is extremely entangled with other pixels, is not robust to simple visual transforms (e.g., brightness, contrast, etc.), and is far-removed from high-level concepts like the output class.

 Second, traditional saliency maps are a very limited type of interface — they only display the attribution for a single class at a time, and do not allow you to probe into individual points more deeply.

 As they do not explicitly deal with hidden layers, it has been difficult to fully explore their design space.

 We instead treat attribution as another user interface building block, and apply it to the hidden layers of a neural network.

 In doing so, we change the questions we can pose.

 Rather than asking whether the color of a particular pixel was important for the “labrador retriever” classification, we instead ask whether the *high-level idea* detected at that position (such as “floppy ear”) was important.

 This approach is similar to what Class Activation Mapping (CAM) methods do but, because they interpret their results back onto the input image, they miss the opportunity to communicate in terms of the rich behavior of a network’s hidden layers.

 The above interface affords us a more flexible relationship with attribution.

 To start, we perform attribution from each spatial position of each hidden layer shown to all 1,000 output classes.

 In order to visualize this thousand-dimensional vector, we use dimensionality reduction to produce a multi-directional saliency map.

 Overlaying these saliency maps on our magnitude-sized activation grids provides an information scent over attribution space.

 The activation grids allow us to anchor attribution to the visual vocabulary our semantic dictionaries first established.

 On hover, we update the legend to depict attribution to the output classes (i.e., which classes does this spatial position most contribute to?).

 Perhaps most interestingly, this interface allows us to interactively perform attribution *between hidden layers*.

 On hover, additional saliency maps mask the hidden layers, in a sense shining a light into their black boxes.

 This type of layer-to-layer attribution is a prime example of how carefully considering interface design drives the generalization of our existing abstractions for interpretability.

 With this diagram, we have begun to think of attribution in terms of higher-level concepts.

 However, at a particular position, many concepts are being detected together and this interface makes it difﬁcult to split them apart.

 By continuing to focus on spatial positions, these concepts remain entangled.

### Channel Attribution

 Saliency maps implicitly slice our cube of activations by applying attribution to the spatial positions of a hidden layer.

 This aggregates over all channels and, as a result, we cannot tell which specific detectors *at each position* most contributed to the final output classification.

 An alternate way to slice the cube is by channels instead of spatial locations.

 Doing so allows us to perform *channel attribution*: how much did each detector contribute to the final output?

 (This approach is similar to contemporaneous work by Kim et al., who do attribution to learned combination of channels.)

 This diagram is analogous to the previous one we saw: we conduct layer-to-layer attribution but this time over channels rather than spatial positions.

 Once again, we use the icons from our semantic dictionary to represent the channels that most contribute to the final output classification.

 Hovering over an individual channel displays a heatmap of its activations overlaid on the input image.

 The legend also updates to show its attribution to the output classes (i.e., what are the top classes this channel supports?).

 Clicking a channel allows us to drill into the layer-to-layer attributions, identifying the channels at lower layers that most contributed as well as the channels at higher layers that are most supported.

 While these diagrams focus on layer-to-layer attribution, it can still be valuable to focus on a single hidden layer.

 For example, the teaser figure allows us to evaluate hypotheses for why one class succeeded over the other.

 Attribution to spatial locations and channels can reveal powerful things about a model, especially when we combine them together.

 Unfortunately, this family of approaches is burdened by two significant problems.

 On the one hand, it is very easy to end up with an overwhelming amount of information: it would take hours of human auditing to understand the long-tail of channels that slightly impact the output.

 On the other hand, both the aggregations we have explored are extremely lossy and can miss important parts of the story.

 And, while we could avoid lossy aggregation by working with individual neurons, and not aggregating at all, this explodes the first problem combinatorially.

Making Things Human-Scale
-------------------------

 In previous sections, we’ve considered three ways of slicing the cube of activations: into spatial activations, channels, and individual neurons.
 Each of these has major downsides.
 If one only uses spatial activations or channels, they miss out on very important parts of the story.
 For example it’s interesting that the floppy ear detector helped us classify an image as a Labrador retriever, but it’s much more interesting when that’s combined with the locations that fired to do so.
 One can try to drill down to the level of neurons to tell the whole story, but the tens of thousands of neurons are simply too much information.
 Even the hundreds of channels, before being split into individual neurons, can be overwhelming to show users!

 If we want to make useful interfaces into neural networks, it isn’t enough to make things meaningful.
 We need to make them human scale, rather than overwhelming dumps of information.
 The key to doing so is finding more meaningful ways of breaking up our activations.
 There is good reason to believe that such decompositions exist.
 Often, many channels or spatial positions will work together in a highly correlated way and are most useful to think of as one unit.
 Other channels or positions will have very little activity, and can be ignored for a high-level overview.
 So, it seems like we ought to be able to find better decompositions if we had the right tools.

 There is an entire field of research, called matrix factorization, that studies optimal strategies for breaking up matrices.
 By flattening our cube into a matrix of spatial locations and channels, we can apply these techniques to get more meaningful groups of neurons.
 These groups will not align as naturally with the cube as the groupings we previously looked at.
 Instead, they will be combinations of spatial locations and channels.
 Moreover, these groups are constructed to explain the behavior of a network on a particular image.
 It would not be effective to reuse the same groupings on another image; each image requires calculating a unique set of groups.

 In addition to naturally slicing a hidden layer’s cube of activations into neurons, spatial locations, or channels, we can also consider more arbitrary groupings of locations and channels.

 The groups that come out of this factorization will be the atoms of the interface a user works with. Unfortunately, any grouping is inherently a tradeoff between reducing things to human scale and, because any aggregation is lossy, preserving information. Matrix factorization lets us pick what our groupings are optimized for, giving us a better tradeoff than the natural groupings we saw earlier.

 The goals of our user interface should influence what we optimize our matrix factorization to prioritize. For example, if we want to prioritize what the network detected, we would want the factorization to fully describe the activations. If we instead wanted to prioritize what would change the network’s behavior, we would want the factorization to fully describe the gradient. Finally, if we want to prioritize what caused the present behavior, we would want the factorization to fully describe the attributions. Of course, we can strike a balance between these three objectives rather than optimizing one to the exclusion of the others.

 In the following diagram, we’ve constructed groups that prioritize the activations, by factorizing the activations
 Most matrix factorization algorithms and libraries are set up to minimize the mean squared error of the reconstruction of a matrix you give them. There are ways to hack such libraries to achieve more general objectives through clever manipulations of the provided matrix, as we will see below. More broadly, matrix factorization is an optimization problem, and with custom tools you can achieve all sorts of custom factorizations.
 with non-negative matrix factorization
 As the name suggests, non-negative matrix factorization (NMF) constrains its factors to be positive. This is fine for the activations of a ReLU network, which must be positive as well. Our experience is that the groups we get from NMF seem more independent and semantically meaningful than those without this constraint. Because of this constraints, groups from NMF are a less efficient at representing the activations than they would be without, but our experience is that they seem more independent and semantically meaningful.
  .
 Notice how the overwhelmingly large number of neurons has been reduced to a small set of groups, concisely summarizing the story of the neural network.

 This figure only focuses at a single layer but, as we saw earlier, it can be useful to look across multiple layers to understand how a neural network assembles together lower-level detectors into higher-level concepts.

 The groups we constructed before were optimized to understand a single layer independent of the others. To understand multiple layers together, we would like each layer’s factorization to be “compatible” — to have the groups of earlier layers naturally compose into the groups of later layers. This is also something we can optimize the factorization for

 We formalize this “compatibility” in a manner described below, although we’re not confident it’s the best formalization and won’t be surprised if it is superseded in future work.

 Consider the attribution from every neuron in the layer to the set of *N* groups we want it to be compatible with.
 The basic idea is to split each entry in the activation matrix into *N* entries on the channel dimension, spreading the values proportional to the absolute value of its attribution to the corresponding group.
 Any factorization of this matrix induces a factorization of the original matrix by collapsing the duplicated entries in the column factors.
 However, the resulting factorization tries to create separate factors when the activation of the same channel has different attributions in different places.

  .

 In this section, we recognize that the way in which we break apart the cube of activations is an important interface decision. Rather than resigning ourselves to the natural slices of the cube of activations, we construct more optimal groupings of neurons. These improved groupings are both more meaningful and more human-scale, making it less tedious for users to understand the behavior of the network.

 Our visualizations have only begun to explore the potential of alternate bases in providing better atoms for understanding neural networks.
 For example, while we focus on creating smaller numbers of directions to explain individual examples, there’s recently been exciting work finding “globally” meaningful directions  — such bases could be especially helpful when trying to understand multiple examples at a time, or in comparing models.
 The recent [NIPS disentangling workshop](https://sites.google.com/corp/view/disentanglenips2017) provides other promising directions. We’re excited to see a venue for this developing area of research.

The Space of Interpretability Interfaces
----------------------------------------

 The interface ideas presented in this article combine building blocks such as feature visualization and attribution.

 Composing these pieces is not an arbitrary process, but rather follows a structure based on the goals of the interface.

 For example, should the interface emphasize *what* the network recognizes, prioritize *how* its understanding develops, or focus on making things *human-scale*.

 To evaluate such goals, and understand the tradeoffs, we need to be able to *systematically* consider possible alternatives.

We can think of an interface as a union of individual elements.

 Each element displays a specific type of *content* (e.g., activations or attribution) using a particular style of *presentation* (e.g., feature visualization or traditional information visualization).

 This content lives on substrates defined by how given *layers* of the network are broken apart into *atoms*, and may be *transformed* by a series of operations (e.g., to filter it or project it onto another substrate).

 For example, our semantic dictionaries use feature visualization to display the activations of a hidden layer's neurons.

 One way to represent this way of thinking is with a formal grammar, but we find it helpful to think about the space visually.

 We can represent the network’s substrate (which layers we display, and how we break them apart) as a grid, with the content and style of presentation plotted on this grid as points and connections.

![](images/design_space/empty.svg)

 This setup gives us a framework to begin exploring the space of interpretability interfaces step by step.

 For instance, let us consider our teaser figure again.

 Its goal is to help us compare two potential classifications for an input image.

![](images/design_space/teaser.svg)

**1. Feature visualization**
![](images/teaser-1.png)

 To understand a classification, we focus on the channels of the `mixed4d` layer. Feature visualization makes these channels meaningful.

**2. Filter by output attribution**
![](images/teaser-2.png)

 Next, we filter for specific classes by calculating the output attribution.

**3. Drill down on hover**
![](images/teaser-3.png)

 Hovering over channels, we get a heatmap of spatial activations.

 figure .teaser-thumb {
 width: 100%;
 border: 30px solid rgb(248, 248, 248);
 border-radius: 10px;
 box-sizing: border-box;
 margin: 12px 0 10px;
 }

 In this article, we have only scratched the surface of possibilities.

 There are lots of combinations of our building blocks left to explore, and the design space gives us a way to do so systematically.

 Moreover, each building block represents a broad class of techniques.

 Our interfaces take only one approach but, as we saw in each section, there are a number of alternatives for feature visualization, attribution, and matrix factorization.

 An immediate next step would be to try using these alternate techniques, and research ways to improve them.

 Finally, this is not the complete set of building blocks; as new ones are discovered, they expand the space.

 For example, Koh & Liang. suggest ways of understanding the influence of dataset examples on model behavior .

 We can think of dataset examples as another substrate in our design space, thus becoming another building block that fully composes with the others.

 In doing so, we can now imagine interfaces that not only allow us to inspect the influence of dataset examples on the final output classification (as Koh & Liang proposed), but also how examples influence the features of hidden layers, and how they influence the relationship between these features and the output.

 For example, if we consider our “Labrador retriever” image, we can not only see which dataset examples most influenced the model to arrive at this classification, but also which dataset examples most caused the “floppy ear” detectors to fire, and which dataset examples most caused these detectors to increase the “Labrador retriever” classification.

![](images/design_space/dataset.svg)

 A new substrate.

 An interface to understand how dataset examples influence the output classification, as presented by Koh & Liang

 An interface showing how examples influence the channels of hidden layers.

 An interface for identifying which dataset examples most caused particular detectors to increase the output classification.

 Beyond interfaces for analyzing model behavior, if we add model *parameters* as a substrate, the design space now allows us to consider interfaces for *taking action* on neural networks.Note that essentially all our interpretability techniques are differentiable, so you can backprop through them.

 While most models today are trained to optimize simple objective functions that one can easily describe, many of the things we’d like models to do in the real world are subtle, nuanced, and hard to describe mathematically.

 An extreme example of the subtle objective problem is something like “creating interesting art”, but much more mundane examples arise more or less whenever humans are involved.

 One very promising approach to training models for these subtle objectives is learning from human feedback .

 However, even with human feedback, it may still be hard to train models to behave the way we want if the problematic aspect of the model doesn’t surface strongly in the training regime where humans are giving feedback.

 There are lots of reasons why problematic behavior may not surface or may be hard for an evaluator to give feedback on.

 For example, discrimination and bias may be subtly present throughout the model’s behavior, such that it’s hard for a human evaluator to critique.

 Or the model may be making a decision in a way that has problematic consequences, but those consequences never play out in the problems we’re training it on.

 Human feedback on the model’s decision making process, facilitated by interpretability interfaces, could be a powerful solution to these problems.

 It might allow us to train models not just to make the *right decisions*, but to make them *for the right reasons*.

 (There is however a danger here: we are optimizing our model to look the way we want in our interface — if we aren’t careful, this may lead to the model fooling us!Related ideas have occasionally been discussed under the term “cognitive steganography.”)

 Another exciting possibility is interfaces for comparing multiple models.

 For instance, we might want to see how a model evolves during training, or how it changes when you transfer it to a new task.

 Or, we might want to understand how a whole family of models compares to each other.

 Existing work has primarily focused on comparing the output behavior of models but more recent work is starting to explore comparing their internal representations as well.

 One of the unique challenges of this work is that we may want to align the atoms of each model; if we have completely different models, can we find the most analogous neurons between them?

 Zooming out, can we develop interfaces that allow us to evaluate large spaces of models at once?

How Trustworthy Are These Interfaces?
-------------------------------------

 In order for interpretability interfaces to be effective, we must trust the story they are telling us.

 We perceive two concerns with the set of building blocks we currently use.

 First, do neurons have a relatively consistent meaning across different input images, and is that meaning accurately reified by feature visualization?

 Semantic dictionaries, and the interfaces that build on top of them, are premised off this question being true.

 Second, does attribution make sense and do we trust any of the attribution methods we presently have?

 Much prior research has found that directions in neural networks are semantically meaningful.

 One particularly striking example of this is “semantic arithmetic” (eg. “king” - “man” + “woman” = “queen”).

 We explored this question, in depth, for GoogLeNet in our previous article and found that many of its neurons seem to correspond to meaningful ideas.We validated this in a number of ways: we visualized them without a generative model prior, so that the content of the visualizations
 was causally linked to the neuron firing; we inspected the spectrum of examples that cause the neuron to fire; and used diversity
 visualizations to try to create different inputs that cause the neuron to fire.

For more details, see
 [the article’s appendix](https://distill.pub/2017/feature-visualization/appendix/) and the guided tour in
 [@ch402′s Twitter thread](https://twitter.com/ch402/status/927968700384153601). We’re actively investigating why GoogLeNet’s neurons seem more meaningful.

 Besides these neurons, however, we also found many neurons that do not have as clean a meaning including “poly-semantic” neurons that respond to a mixture of salient ideas (e.g., “cat” and “car”).

 There are natural ways that interfaces could respond to this: we could use diversity visualizations to reveal the variety of meanings the neuron can take, or rotate our semantic dictionaries so their components are more disentangled.

 Of course, just like our models can be fooled, the features that make them up can be too — including with adversarial examples .

 In our view, features do not need to be flawless detectors for it to be useful for us to think about them as such.

 In fact, it can be interesting to identify when a detector misfires.

 With regards to attribution, recent work suggests that many of our current techniques are unreliable.

 One might even wonder if the idea is fundamentally flawed, since a function’s output could be the result of non-linear interactions between its inputs.

 One way these interactions can pan out is as attribution being “path-dependent”.

 A natural response to this would be for interfaces to explicitly surface this information: how path-dependent is the attribution?

 A deeper concern, however, would be whether this path-dependency dominates the attribution.

 Clearly, this is not a concern for attribution between adjacent layers because of the simple (essentially linear) mapping between them.

 While there may be technicalities about correlated inputs, we believe that attribution is on firm grounding here.

 And even with layers further apart, our experience has been that attribution between high-level features at the output is much more consistent than attribution to the input — we believe that path-dependence is not a dominating concern here.

 Model behavior is extremely complex, and our current building blocks force us to show only speciﬁc aspects of it.

 An important direction for future interpretability research will be developing techniques that achieve broader coverage of model behavior.

 But, even with such improvements, we anticipate that a key marker of trustworthiness will be interfaces that do not mislead.

 Interacting with the explicit information displayed should not cause users to implicitly draw incorrect assessments about the model (we see a similar principle articulated by Mackinlay for data visualization).

 Undoubtedly, the interfaces we present in this article have room to improve in this regard.

 Fundamental research, at the intersection of machine learning and human-computer interaction, is necessary to resolve these issues.

 Trusting our interfaces is essential for many of the ways we want to use interpretability.

 This is both because the stakes can be high (as in safety and fairness) and also because ideas like training models with interpretability feedback put our interpretability techniques in the middle of an adversarial setting.

Conclusion & Future Work
------------------------

 There is a rich design space for interacting with enumerative algorithms, and we believe an equally rich space exists for interacting with neural networks.

 We have a lot of work left ahead of us to build powerful and trustworthy interfaces for interpretability.

 But, if we succeed, interpretability promises to be a powerful tool in enabling meaningful human oversight and in building fair, safe, and aligned AI systems.

The Building Blocks of Interpretability

Chris Olah
Arvind Satyanarayan
Ian Johnson
Shan Carter
Ludwig Schubert
Katherine Ye
Alexander Mordvintsev

Summary: Interpretability techniques are normally studied in isolation. We explore the powerful interfaces that arise when you combine them -- and the rich structure of this combinatorial space.

Thought as a Technology

Visualizing Representations: Deep Learning and Human Beings

Understanding neural networks through deep visualization

Using Artificial Intelligence to Augment Human Intelligence

Visualizing higher-layer features of a deep network

Feature Visualization

Deep inside convolutional networks: Visualising image classification models and saliency maps

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Inceptionism: Going deeper into neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Visualizing and understanding convolutional networks

Striving for simplicity: The all convolutional net

Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization

Interpretable Explanations of Black Boxes by Meaningful Perturbation

PatternNet and PatternLRP--Improving the interpretability of neural networks

The (Un)reliability of saliency methods

Axiomatic attribution for deep networks

Visualizing data using t-SNE

LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent neural networks

ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models

Do convolutional neural networks learn class hierarchy?

Going deeper with convolutions

Deconvolution and Checkerboard Artifacts

Learning deep features for discriminative localization

Information foraging

TCAV: Relative concept importance testing with Linear Concept Activation Vectors

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Understanding Black-box Predictions via Influence Functions

Interactive optimization for steering machine classification

Interacting with predictions: Visual inspection of black-box machine learning models

Modeltracker: Redesigning performance analysis tools for machine learning

Network dissection: Quantifying interpretability of deep visual representations

Intriguing properties of neural networks

Efficient estimation of word representations in vector space

Unsupervised representation learning with deep convolutional generative adversarial networks

Adversarial manipulation of deep representations

Automating the design of graphical presentations of relational information


 What are computers for?
-------------------------

 Historically, different answers to this question – that is,
 different visions of computing – have helped inspire and
 determine the computing systems humanity has ultimately
 built. Consider the early electronic computers. ENIAC, the
 world’s first general-purpose electronic computer, was
 commissioned to compute artillery firing tables for the United
 States Army. Other early computers were also used to solve
 numerical problems, such as simulating nuclear explosions,
 predicting the weather, and planning the motion of rockets. The
 machines operated in a batch mode, using crude input and output
 devices, and without any real-time interaction. It was a vision
 of computers as number-crunching machines, used to speed up
 calculations that would formerly have taken weeks, months, or more
 for a team of humans.

 In the 1950s a different vision of what computers are for began to
 develop. That vision was crystallized in 1962, when Douglas
 Engelbart proposed that computers could be used as a way
 of augmenting human
 intellect. In this view, computers weren’t primarily
 tools for solving number-crunching problems. Rather, they were
 real-time interactive systems, with rich inputs and outputs, that
 humans could work with to support and expand their own
 problem-solving process. This vision of intelligence augmentation
 (IA) deeply influenced many others, including researchers such as
 Alan Kay at Xerox PARC, entrepreneurs such as Steve Jobs at Apple,
 and led to many of the key ideas of modern computing systems. Its
 ideas have also deeply influenced digital art and music, and
 fields such as interaction design, data visualization,
 computational creativity, and human-computer interaction.

 Research on IA has often been in competition with research on
 artificial intelligence (AI): competition for funding, competition
 for the interest of talented researchers. Although there has
 always been overlap between the fields, IA has typically focused
 on building systems which put humans and machines to work
 together, while AI has focused on complete outsourcing of
 intellectual tasks to machines. In particular, problems in AI are
 often framed in terms of matching or surpassing human performance:
 beating humans at chess or Go; learning to recognize speech and
 images or translating language as well as humans; and so on.

 This essay describes a new field, emerging today out of a
 synthesis of AI and IA. For this field, we suggest the
 name *artificial intelligence augmentation* (AIA): the use
 of AI systems to help develop new methods for intelligence
 augmentation. This new field introduces important new fundamental
 questions, questions not associated to either parent field. We
 believe the principles and systems of AIA will be radically
 different to most existing systems.

 Our essay begins with a survey of recent technical work hinting at
 artificial intelligence augmentation, including work
 on *generative interfaces* – that is, interfaces
 which can be used to explore and visualize generative machine
 learning models. Such interfaces develop a kind of cartography of
 generative models, ways for humans to explore and make meaning
 from those models, and to incorporate what those models
 “know” into their creative work.

 Our essay is not just a survey of technical work. We believe now
 is a good time to identify some of the broad, fundamental
 questions at the foundation of this emerging field. To what
 extent are these new tools enabling creativity? Can they be used
 to generate ideas which are truly surprising and new, or are the
 ideas cliches, based on trivial recombinations of existing ideas?
 Can such systems be used to develop fundamental new interface
 primitives? How will those new primitives change and expand the
 way humans think?

 Using generative models to invent meaningful creative operations
------------------------------------------------------------------

 Let’s look at an example where a machine learning model makes a
 new type of interface possible. To understand the interface,
 imagine you’re a type designer, working on creating a new
 fontWe shall egregiously abuse the distinction between
 a font and a typeface. Apologies to any type designers who may be
 reading.. After sketching some initial designs, you
 wish to experiment with bold, italic, and condensed variations.
 Let’s examine a tool to generate and explore such variations, from
 any initial design. For reasons that will soon be explained the
 quality of results is quite crude; please bear with us.

 Of course, varying the bolding (i.e., the weight), italicization
 and width are just three ways you can vary a font. Imagine that
 instead of building specialized tools, users could build their own
 tool merely by choosing examples of existing fonts. For instance,
 suppose you wanted to vary the degree of serifing on a font. In
 the following, please select 5 to 10 sans-serif fonts from the top
 box, and drag them to the box on the left. Select 5 to 10 serif
 fonts and drag them to the box on the right. As you do this, a
 machine learning model running in your browser will automatically
 infer from these examples how to interpolate your starting font in
 either the serif or sans-serif direction:

 In fact, we used this same technique to build the earlier bolding
 italicization, and condensing tool. To do so, we used the
 following examples of bold and non-bold fonts, of italic and
 non-italic fonts, and of condensed and non-condensed fonts:

 To build these tools, we used what’s called a *generative
 model*; the particular model we use was trained
 by James Wexler. To
 understand generative models, consider that *a priori*
 describing a font appears to require a lot of data. For
 instance, if the font is 646464 by 646464 pixels, then we’d expect
 to need 64×64=4,09664 \times 64 = 4,09664×64=4,096 parameters to describe a single
 glyph. But we can use a generative model to find a much simpler
 description.

 We do this by building a neural network which takes a small number
 of input variables, called *latent variables*, and produces
 as output the entire glyph. For the particular model we use, we
 have 404040 latent space dimensions, and map that into the
 4,0964,0964,096-dimensional space describing all the pixels in the glyph.
 In other words, the idea is to map a low-dimensional space into a
 higher-dimensional space:

 The generative model we use is a type of neural network known as
 a variational autoencoder
 (VAE). For our purposes, the details of the generative
 model aren’t so important. The important thing is that by
 changing the latent variables used as input, it’s possible to get
 different fonts as output. So one choice of latent variables will
 give one font, while another choice will give a different font:

 You can think of the latent variables as a compact, high-level
 representation of the font. The neural network takes that
 high-level representation and converts it into the full pixel
 data. It’s remarkable that just 404040 numbers can capture the
 apparent complexity in a glyph, which originally required 4,0964,0964,096
 variables.

 The generative model we use is learnt from a training set of more
 than 505050 thousand
 fonts Bernhardsson
 scraped from the open web. During training, the weights and
 biases in the network are adjusted so that the network can output
 a close approximation to any desired font from the training set,
 provided a suitable choice of latent variables is made. In some
 sense, the model is learning a highly compressed representation of
 all the training fonts.

 In fact, the model doesn’t just reproduce the training fonts. It
 can also generalize, producing fonts not seen in training. By
 being forced to find a compact description of the training
 examples, the neural net learns an abstract, higher-level model of
 what a font is. That higher-level model makes it possible to
 generalize beyond the training examples already seen, to produce
 realistic-looking fonts.

 Ideally, a good generative model would be exposed to a relatively
 small number of training examples, and use that exposure to
 generalize to the space of all possible human-readable fonts.
 That is, for any conceivable font – whether existing or
 perhaps even imagined in the future – it would be possible
 to find latent variables corresponding exactly to that font. Of
 course, the model we’re using falls far short of this ideal
 – a particularly egregious failure is that many fonts
 generated by the model omit the tail on the capital
 “Q” (you can see this in the examples above). Still,
 it’s useful to keep in mind what an ideal generative model would
 do.

 Such generative models are similar in some ways to how scientific
 theories work. Scientific theories often greatly simplify the
 description of what appear to be complex phenomena, reducing large
 numbers of variables to just a few variables from which many
 aspects of system behavior can be deduced. Furthermore, good
 scientific theories sometimes enable us to generalize to discover
 new phenomena.

 As an example, consider ordinary material objects. Such objects
 have what physicists call a *phase* – they may be a
 liquid, a solid, a gas, or perhaps something more exotic, like a
 superconductor
 or [Bose-Einstein
 condensate](https://en.wikipedia.org/wiki/Bose%E2%80%93Einstein_condensate). *A priori*, such systems seem immensely
 complex, involving perhaps 102310^{23}1023 or so molecules. But the
 laws of thermodynamics and statistical mechanics enable us to find
 a simpler description, reducing that complexity to just a few
 variables (temperature, pressure, and so on), which encompass much
 of the behavior of the system. Furthermore, sometimes it’s
 possible to generalize, predicting unexpected new phases of
 matter. For example, in 1924, physicists used thermodynamics and
 statistical mechanics to predict a remarkable new phase of matter,
 Bose-Einstein condensation, in which a collection of atoms may all
 occupy identical quantum states, leading to surprising large-scale
 quantum interference effects. We’ll come back to this predictive
 ability in our later discussion of creativity and generative
 models.

 Returning to the nuts and bolts of generative models, how can we
 use such models to do example-based reasoning like that in the
 tool shown above? Let’s consider the case of the bolding tool. In
 that instance, we take the average of all the latent vectors for
 the user-specified bold fonts, and the average for all the
 user-specified non-bold fonts. We then compute the difference
 between these two average vectors:

 We’ll refer to this as the *bolding vector*. To make some
 given font bolder, we simply add a little of the bolding vector to
 the corresponding latent vector, with the amount of bolding vector
 added controlling the boldness of the resultIn
 practice, sometimes a slightly different procedure is used. In
 some generative models the latent vectors satisfy some constraints
 – for instance, they may all be of the same length. When
 that’s the case, as in our model, a more sophisticated
 “adding” operation must be used, to ensure the length
 remains the same. But conceptually, the picture of adding the
 bolding vector is the right way to think.:

 This technique was introduced
 by Larsen *et al*, and
 vectors like the bolding vector are sometimes called
 *attribute vectors*. The same idea is use to implement all
 the tools we’ve shown. That is, we use example fonts to creating
 a bolding vector, an italicizing vector, a condensing vector, and
 a user-defined serif vector. The interface thus provides a way of
 exploring the latent space in those four directions.

 The tools we’ve shown have many drawbacks. Consider the following
 example, where we start with an example glyph, in the middle, and
 either increase or decrease the bolding (on the right and left,
 respectively):

 Examining the glyphs on the left and right we see many unfortunate
 artifacts. Particularly for the rightmost glyph, the edges start to get
 rough, and the serifs begin to disappear. A better generative
 model would reduce those artifacts. That’s a good long-term
 research program, posing many intriguing problems. But even with
 the model we have, there are also some striking benefits to the
 use of the generative model.

 To understand these benefits, consider a naive approach to
 bolding, in which we simply add some extra pixels around a glyph’s
 edges, thickening it up. While this thickening perhaps matches a
 non-expert’s way of thinking about type design, an expert does
 something much more involved. In the following we show the
 results of this naive thickening procedure versus what is actually
 done, for Georgia and Helvetica:

 As you can see, the naive bolding procedure produces quite
 different results, in both cases. For example, in Georgia, the
 left stroke is only changed slightly by bolding, while the right
 stroke is greatly enlarged, but only on one side. In both
 fonts, bolding doesn’t change the height of the font, while the
 naive approach does.

 As these examples show, good bolding is *not* a trivial
 process of thickening up a font. Expert type designers have many
 heuristics for bolding, heuristics inferred from much previous
 experimentation, and careful study of historical
 examples. Capturing all those heuristics in a conventional program
 would involve immense work. The benefit of using the generative
 model is that it automatically learns many such heuristics.

 For example, a naive bolding tool would rapidly fill in the
 enclosed negative space in the enclosed upper region of the letter
 “A”. The font tool doesn’t do this. Instead, it goes
 to some trouble to preserve the enclosed negative space, moving
 the A’s bar down, and filling out the interior strokes more slowly
 than the exterior. This principle is evident in the examples
 shown above, especially Helvetica, and it can also be seen in the
 operation of the font tool:

 The heuristic of preserving enclosed negative space is not *a
 priori* obvious. However, it’s done in many professionally
 designed fonts. If you examine examples like those shown above
 it’s easy to see why: it improves legibility. During training,
 our generative model has automatically inferred this principle
 from the examples it’s seen. And our bolding interface then makes
 this available to the user.

 In fact, the model captures many other heuristics. For instance,
 in the above examples the heights of the fonts are (roughly)
 preserved, which is the norm in professional font design. Again,
 what’s going on isn’t just a thickening of the font, but rather
 the application of a more subtle heuristic inferred by the
 generative model. Such heuristics can be used to create fonts
 with properties which would otherwise be unlikely to occur to
 users. Thus, the tool expands ordinary people’s ability to
 explore the space of meaningful fonts.

 The font tool is an example of a kind of cognitive technology. In
 particular, the primitive operations it contains can be
 internalized as part of how a user thinks. In this it resembles a
 program such as *Photoshop* or a spreadsheet or 3D graphics
 programs. Each provides a novel set of interface primitives,
 primitives which can be internalized by the user as fundamental
 new elements in their thinking. This act of internalization of new
 primitives is fundamental to much work on intelligence
 augmentation.

 The ideas shown in the font tool can be extended to other domains.
 Using the same interface, we can use a generative model to
 manipulate images of human faces using qualities such as
 expression, gender, or hair color. Or to manipulate sentences
 using length, sarcasm, or tone. Or to manipulate molecules using
 chemical properties:

 #alternate-uses .root {
 max-width: 300px;
 margin: 0 auto;
 padding: 0 20px;
 }
 @media(min-width: 600px) {
 #alternate-uses .root {
 max-width: 760px;
 display: grid;
 grid-template-columns: 1fr 1fr 1fr;
 grid-gap: 50px;
 }
 }
 .note {
 margin-top: 12px;
 grid-column-end: span 2;
 font-size: 10px;
 line-height: 1.5em;
 text-align: left;
 color: rgba(0, 0, 0, 0.4);
 }

 Images from *Sampling Generative Networks* by White.

 Sentence from *Pride and Prejudice* by Jane Austen. Interpolated by the authors. Inspired by experiments done by the novelist Robin Sloan

 Images from *Automatic chemical design using a data-driven continuous representation of molecules* by Gómez-Bombarelli *et al*.

 Such generative interfaces provide a kind of cartography of
 generative models, ways for humans to explore and make meaning
 using those models.

 We saw earlier that the font model automatically infers relatively
 deep principles about font design, and makes them available to
 users. While it’s great that such deep principles can be
 inferred, sometimes such models infer other things that are wrong,
 or undesirable. For example, White
 points out the addition of a smile vector in some face
 models will make faces not just smile more, but also appear more
 feminine. Why? Because in the training data more women than men
 were smiling. So these models may not just learn deep facts about
 the world, they may also internalize prejudices or erroneous
 beliefs. Once such a bias is known, it is often possible to make
 corrections. But to find those biases requires careful auditing
 of the models, and it is not yet clear how we can ensure such
 audits are exhaustive.

 More broadly, we can ask why attribute vectors work, when they
 work, and when they fail? At the moment, the answers to these
 questions are poorly understood.

 For the attribute vector to work requires that taking any starting
 font, we can construct the corresponding bold version by adding
 the *same* vector in the latent space. However, *a
 priori* there is no reason using a single constant vector to
 displace will work. It may be that we should displace in many
 different ways. For instance, the heuristics used to bold serif
 and sans-serif fonts are quite different, and so it seems likely
 that very different displacements would be involved:

 Of course, we could do something more sophisticated than using a
 single constant attribute vector. Given pairs of example fonts
 (unbold, bold) we could train a machine learning algorithm to take
 as input the latent vector for the unbolded version and output the
 latent vector for the bolded version. With additional training
 data about font weights, the machine learning algorithm could
 learn to generate fonts of arbitrary weight. Attribute vectors
 are just an extremely simple approach to doing these kinds of
 operation.

 For these reasons, it seems unlikely that attribute vectors will
 last as an approach to manipulating high-level features. Over the
 next few years much better approaches will be developed. However,
 we can still expect interfaces offering operations broadly similar
 to those sketched above, allowing access to high-level and
 potentially user-defined concepts. That interface pattern doesn’t
 depend on the technical details of attribute vectors.

 Interactive Generative Adversarial Models
-------------------------------------------

 Let’s look at another example using machine learning models to
 augment human creativity. It’s the interactive generative
 adversarial networks, or iGANs, introduced
 by Zhu *et al* in 2016.

 One of the examples of Zhu *et al* is the use of iGANs in
 an interface to generate images of consumer products such as
 shoes. Conventionally, such an interface would require the
 programmer to write a program containing a great deal of knowledge
 about shoes: soles, laces, heels, and so on. Instead of doing
 this, Zhu *et al* train a generative model using 505050
 thousand images of shoes, downloaded from Zappos. They then use
 that generative model to build an interface that lets a user
 roughly sketch the shape of a shoe, the sole, the laces, and so
 on:

Excerpted from Zhu *et
 al*.

 The visual quality is low, in part because the generative model
 Zhu *et al* used is outdated by modern (2017) standards
 – with more modern models, the visual quality would be much
 higher.

 But the visual quality is not the point. Many interesting things
 are going on in this prototype. For instance, notice how the
 overall shape of the shoe changes considerably when the sole is
 filled in – it becomes narrower and sleeker. Many small
 details are filled in, like the black piping on the top of the
 white sole, and the red coloring filled in everywhere on the
 shoe’s upper. These and other facts are automatically deduced
 from the underlying generative model, in a way we’ll describe
 shortly.

 The same interface may be used to sketch landscapes. The only
 difference is that the underlying generative model has been
 trained on landscape images rather than images of shoes. In this
 case it becomes possible to sketch in just the colors associated
 to a landscape. For example, here’s a user sketching in some green
 grass, the outline of a mountain, some blue sky, and snow on the
 mountain:

Excerpted from Zhu *et
 al*.

 The generative models used in these interfaces are different than
 for our font model. Rather than using variational autoencoders,
 they’re based on generative
 adversarial networks (GANs). But the underlying idea is
 still to find a low-dimensional latent space which can be used to
 represent (say) all landscape images, and map that latent space to
 a corresponding image. Again, we can think of points in the
 latent space as a compact way of describing landscape images.

 Roughly speaking, the way the iGANs works is as follows. Whatever
 the current image is, it corresponds to some point in the latent
 space:

 Suppose, as happened in the earlier video, the user now sketches
 in a stroke outlining the mountain shape. We can think of the
 stroke as a constraint on the image, picking out a subspace of the
 latent space, consisting of all points in the latent space whose
 image matches that outline:

 The way the interface works is to find a point in the latent space
 which is near to the current image, so the image is not changed
 too much, but also coming close to satisfying the imposed
 constraints. This is done by optimizing an objective function
 which combines the distance to each of the imposed constraints, as
 well as the distance moved from the current point. If there’s
 just a single constraint, say, corresponding to the mountain
 stroke, this looks something like the following:

 We can think of this, then, as a way of applying constraints to
 the latent space to move the image around in meaningful ways.

 The iGANs have much in common with the font tool we showed
 earlier. Both make available operations that encode much subtle
 knowledge about the world, whether it be learning to understand
 what a mountain looks like, or inferring that enclosed negative
 space should be preserved when bolding a font. Both the iGANs and
 the font tool provide ways of understanding and navigating a
 high-dimensional space, keeping us on the natural space of fonts
 or shoes or landscapes. As Zhu *et al* remark:

>
>  [F]or most of us, even a simple image manipulation in Photoshop
>  presents insurmountable difficulties… any less-than-perfect
>  edit immediately makes the image look completely unrealistic. To
>  put another way, classic visual manipulation paradigm does not
>  prevent the user from “falling off” the manifold of
>  natural images.
>

 Like the font tool, the iGANs is a cognitive technology. Users
 can internalize the interface operations as new primitive elements
 in their thinking. In the case of shoes, for example, they can
 learn to think in terms of the difference they want to apply,
 adding a heel, or a higher top, or a special highlight. This is
 richer than the traditional way non-experts think about shoes
 (“Size 11, black” *etc*). To the extent that
 non-experts do think in more sophisticated ways –
 “make the top a little higher and sleeker” –
 they get little practice in thinking this way, or seeing the
 consequences of their choices. Having an interface like this
 enables easier exploration, the ability to develop idioms and the
 ability to plan, to swap ideas with friends, and so on.

 Two models of computation
---------------------------

 Let’s revisit the question we began the essay with, the question
 of what computers are for, and how this relates to intelligence
 augmentation.

 One common conception of computers is that they’re problem-solving
 machines: “computer, what is the result of firing this
 artillery shell in such-and-such a wind [and so on]?”;
 “computer, what will the maximum temperature in Tokyo be in
 5 days?”; “computer, what is the best move to take
 when the Go board is in this position?”; “computer,
 how should this image be classified?”; and so on.

 This is a conception common to both the early view of computers as
 number-crunchers, and also in much work on AI, both historically
 and today. It’s a model of a computer as a way of outsourcing
 cognition. In speculative depictions of possible future AI,
 this *cognitive outsourcing* model often shows up in the
 view of an AI as an oracle, able to solve some large class of
 problems with better-than-human performance.

 But a very different conception of what computers are for is
 possible, a conception much more congruent with work on
 intelligence augmentation.

 To understand this alternate view, consider our subjective
 experience of thought. For many people, that experience is verbal:
 they think using language, forming chains of words in their heads,
 similar to sentences in speech or written on a page. For other
 people, thinking is a more visual experience, incorporating
 representations such as graphs and maps. Still other people mix
 mathematics into their thinking, using algebraic expressions or
 diagrammatic techniques, such as Feynman diagrams and Penrose
 diagrams.

 In each case, we’re thinking using representations invented by
 other people: words, graphs, maps, algebra, mathematical diagrams,
 and so on. We internalize these cognitive technologies as we grow
 up, and come to use them as a kind of substrate for our thinking.

 For most of history, the range of available cognitive technologies
 has changed slowly and incrementally. A new word will be
 introduced, or a new mathematical symbol. More rarely, a radical
 new cognitive technology will be developed. For example, in 1637
 Descartes published his “Discourse on Method”,
 explaining how to represent geometric ideas using algebra, and
 vice versa:

 This enabled a radical change and expansion in how we think about
 both geometry and algebra.

 Historically, lasting cognitive technologies have been invented
 only rarely. But modern computers are a meta-medium enabling the
 rapid invention of many new cognitive technologies. Consider a
 relatively banal example, such
 as *Photoshop*. Adept *Photoshop* users routinely
 have formerly impossible thoughts such as: “let’s apply the
 clone stamp to the such-and-such layer.”. That’s an
 instance of a more general class of thought: “computer, [new
 type of action] this [new type of representation for a newly
 imagined class of object]”. When that happens, we’re using
 computers to expand the range of thoughts we can think.

 It’s this kind of *cognitive transformation* model which
 underlies much of the deepest work on intelligence augmentation.
 Rather than outsourcing cognition, it’s about changing the
 operations and representations we use to think; it’s about
 changing the substrate of thought itself. And so while cognitive
 outsourcing is important, this cognitive transformation view
 offers a much more profound model of intelligence augmentation.
 It’s a view in which computers are a means to change and expand
 human thought itself.

 Historically, cognitive technologies were developed by human
 inventors, ranging from the invention of writing in Sumeria and
 Mesoamerica, to the modern interfaces of designers such as Douglas
 Engelbart, Alan Kay, and others.

 Examples such as those described in this essay suggest that AI
 systems can enable the creation of new cognitive technologies.
 Things like the font tool aren’t just oracles to be consulted when
 you want a new font. Rather, they can be used to explore and
 discover, to provide new representations and operations, which can
 be internalized as part of the user’s own thinking. And while
 these examples are in their early stages, they suggest AI is not
 just about cognitive outsourcing. A different view of AI is
 possible, one where it helps us invent new cognitive technologies
 which transform the way we think.

 In this essay we’ve focused on a small number of examples, mostly
 involving exploration of the latent space. There are many other
 examples of artificial intelligence augmentation. To give some
 flavor, without being comprehensive:
 the sketch-rnn system, for neural
 network assisted drawing;
 the Wekinator, which enables
 users to rapidly build new musical instruments and artistic
 systems; TopoSketch, for developing
 animations by exploring latent spaces; machine learning models for
 designing overall typographic
 layout; and a generative model which enables
 interpolation between musical
 phrases. In each case, the systems use machine learning
 to enable new primitives which can be integrated into the user’s
 thinking. More broadly, artificial intelligence augmentation will
 draw on fields such as computational
 creativity and interactive machine
 learning.

 Finding powerful new primitives of thought
--------------------------------------------

 We’ve argued that machine learning systems can help create
 representations and operations which serve as new primitives in
 human thought. What properties should we look for in such new
 primitives? This is too large a question to be answered
 comprehensively in a short essay. But we will explore it briefly.

 Historically, important new media forms often seem strange when
 introduced. Many such stories have passed into popular culture:
 the near riot at the premiere of Stravinsky and Nijinksy’s
 “Rite of Spring”; the consternation caused by the
 early cubist paintings, leading
 *The New York Times* to
 comment: “What do they mean? Have those
 responsible for them taken leave of their senses? Is it art or
 madness? Who knows?”

 Another example comes from physics. In the 1940s, different
 formulations of the theory of quantum electrodynamics were
 developed independently by the physicists Julian Schwinger,
 Shin’ichirō Tomonaga, and Richard Feynman. In their work,
 Schwinger and Tomonaga used a conventional algebraic approach,
 along lines similar to the rest of physics. Feynman used a more
 radical approach, based on what are now known as Feynman diagrams,
 for depicting the interaction of light and matter:

![](images/feynmann-diagram.svg)
Image by [Joel
 Holdsworth](https://commons.wikimedia.org/w/index.php?curid=1764161)), licensed under a Creative Commons
 Attribution-Share Alike 3.0 Unported license

 Initially, the Schwinger-Tomonaga approach was easier for other
 physicists to understand. When Feynman and Schwinger presented
 their work at a 1948 workshop, Schwinger was immediately
 acclaimed. By contrast, Feynman left his audience mystified. As
 James Gleick put it in his biography of
 Feynman:

>
>  It struck Feynman that everyone had a favorite principle or
>  theorem and he was violating them all… Feynman knew he had
>  failed. At the time, he was in anguish. Later he said simply:
>  “I had too much stuff. My machines came from too far
>  away.”
>

 Of course, strangeness for strangeness’s sake alone is not
 useful. But these examples suggest that breakthroughs in
 representation often appear strange at first. Is there any
 underlying reason that is true?

 Part of the reason is because if some representation is truly new,
 then it will appear different than anything you’ve ever seen
 before. Feynman’s diagrams, Picasso’s paintings, Stravinsky’s
 music: all revealed genuinely new ways of making meaning. Good
 representations sharpen up such insights, eliding the familiar to
 show that which is new as vividly as possible. But because of
 that emphasis on unfamiliarity, the representation will seem
 strange: it shows relationships you’ve never seen before. In some
 sense, the task of the designer is to identify that core
 strangeness, and to amplify it as much as possible.

 Strange representations are often difficult to understand. At
 first, physicists preferred Schwinger-Tomonaga to Feynman. But as
 Feynman’s approach was slowly understood by physicists, they
 realized that although Schwinger-Tomonaga and Feynman were
 mathematically equivalent, Feynman was more powerful. As Gleick
 puts it:

>
>  Schwinger’s students at Harvard were put at a competitive
>  disadvantage, or so it seemed to their fellows elsewhere, who
>  suspected them of surreptitiously using the diagrams anyway. This
>  was sometimes true… Murray Gell-Mann later spent a semester
>  staying in Schwinger’s house and loved to say afterward that he
>  had searched everywhere for the Feynman diagrams. He had not
>  found any, but one room had been locked…
>

 These ideas are true not just of historical representations, but
 also of computer interfaces. However, our advocacy of strangeness
 in representation contradicts much conventional wisdom about
 interfaces, especially the widely-held belief that they should be
 “user friendly”, i.e., simple and immediately useable
 by novices. That most often means the interface is cliched, built
 from conventional elements combined in standard ways. But while
 using a cliched interface may be easy and fun, it’s an ease
 similar to reading a formulaic romance novel. It means the
 interface does not reveal anything truly surprising about its
 subject area. And so it will do little to deepen the user’s
 understanding, or to change the way they think. For mundane tasks
 that is fine, but for deeper tasks, and for the longer term, you
 want a better interface.

 Ideally, an interface will surface the deepest principles
 underlying a subject, revealing a new world to the user. When you
 learn such an interface, you internalize those principles, giving
 you more powerful ways of reasoning about that world. Those
 principles are the diffs in your understanding. They’re all you
 really want to see, everything else is at best support, at worst
 unimportant dross. The purpose of the best interfaces isn’t to be
 user-friendly in some shallow sense. It’s to be user-friendly in
 a much stronger sense, reifying deep
 principles about the world, making them the working
 conditions in which users live and create. At that point what once
 appeared strange can instead becomes comfortable and familiar,
 part of the pattern of thoughtA powerful instance of
 these ideas is when an interface reifies general-purpose
 principles. An example is an
 interface one of us developed
 based on the principle of conservation of energy. Such
 general-purpose principles generate multiple unexpected
 relationships between the entities of a subject, and so are a
 particularly rich source of insights when reified in an
 interface..

 What does this mean for the use of AI models for intelligence
 augmentation?

 Aspirationally, as we’ve seen, our machine learning models will
 help us build interfaces which reify deep principles in ways
 meaningful to the user. For that to happen, the models have to
 discover deep principles about the world, recognize those
 principles, and then surface them as vividly as possible in an
 interface, in a way comprehensible by the user.

 Of course, this is a tall order! The examples we’ve shown are just
 barely beginning to do this. It’s true that our models do
 sometimes discover relatively deep principles, like the
 preservation of enclosed negative space when bolding a font. But
 this is merely implicit in the model. And while we’ve built a tool
 which takes advantage of such principles, it’d be better if the
 model automatically inferred the important principles learned, and
 found ways of explicitly surfacing them through the interface.
 (Encouraging progress toward this has been made
 by InfoGANs, which use
 information-theoretic ideas to find structure in the latent
 space.) Ideally, such models would start to get at true
 explanations, not just in a static form, but in a dynamic form,
 manipulable by the user. But we’re a long way from that point.

 Do these interfaces inhibit creativity?
-----------------------------------------

 It’s tempting to be skeptical of the expressiveness of the
 interfaces we’ve described. If an interface constrains us to
 explore only the natural space of images, does that mean we’re
 merely doing the expected? Does it mean these interfaces can only
 be used to generate visual cliches? Does it prevent us from
 generating anything truly new, from doing truly creative work?

 To answer these questions, it’s helpful to identify two different
 modes of creativity. This two-mode model is over-simplified:
 creativity doesn’t fit so neatly into two distinct categories. Yet
 the model nonetheless clarifies the role of new interfaces in
 creative work.

 The first mode of creativity is the everyday creativity of a
 craftsperson engaged in their craft. Much of the work of a font
 designer, for example, consists of competent recombination of the
 best existing practices. Such work typically involves many
 creative choices to meet the intended design goals, but not
 developing key new underlying principles.

 For such work, the generative interfaces we’ve been discussing are
 promising. While they currently have many limitations, future
 research will identity and fix many deficiencies. This is
 happening rapidly with GANs: the original
 GANs had many limitations,
 but models soon appeared that were better adapted to
 images, improved the
 resolution, reduced artifactsSo much work has been
 done on improving resolution and reducing artifacts it seems
 unfair to single out any small set of papers, and to omit the many
 others., and so on. With enough iterations it’s
 plausible these generative interfaces will become powerful tools
 for craft work.

 The second mode of creativity aims toward developing new
 principles that fundamentally change the range of creative
 expression. One sees this in the work of artists such as Picasso
 or Monet, who violated existing principles of painting, developing
 new principles which enabled people to see in new ways.

 Is it possible to do such creative work, while using a generative
 interface? Don’t such interfaces constrain us to the space of
 natural images, or natural fonts, and thus actively prevent us
 from exploring the most interesting new directions in creative
 work?

 The situation is more complex than this.

 In part, this is a question about the power of our generative
 models. In some cases, the model can only generate recombinations
 of existing ideas. This is a limitation of an ideal GAN, since a
 perfectly trained GAN generator will reproduce the training
 distribution. Such a model can’t directly generate an image based
 on new fundamental principles, because such an image wouldn’t look
 anything like it’s seen in its training data.

 Artists such as [Mario
 Klingemann](http://quasimondo.com/) and [Mike
 Tyka](http://www.miketyka.com/) are now using GANs to create interesting
 artwork. They’re doing that using “imperfect” GAN
 models, which they seem to be able to use to explore interesting
 new principles; it’s perhaps the case that bad GANs may be more
 artistically interesting than ideal GANs. Furthermore, nothing
 says an interface must only help us explore the latent space.
 Perhaps operations can be added which deliberately take us out
 of the latent space, or to less probable (and so more
 surprising) parts of the space of natural images.

 Of course, GANs are not the only generative models. In a
 sufficiently powerful generative model, the generalizations
 discovered by the model may contain ideas going beyond what humans
 have discovered. In that case, exploration of the latent space may
 enable us to discover new fundamental principles. The model would
 have discovered stronger abstractions than human experts. Imagine
 a generative model trained on paintings up until just before the
 time of the cubists; might it be that by exploring that model it
 would be possible to discover cubism? It would be an analogue to
 something like the prediction of Bose-Einstein condensation, as
 discussed earlier in the essay. Such invention is beyond today’s
 generative models, but seems a worthwhile aspiration for future
 models.

 Our examples so far have all been based on generative models. But
 there are some illuminating examples which are not based on
 generative models. Consider the pix2pix system developed
 by Isola *et al*. This
 system is trained on pairs of images, e.g., pairs showing the
 edges of a cat, and the actual corresponding cat. Once trained,
 it can be shown a set of edges and asked to generate an image for
 an actual corresponding cat. It often does this quite well:

 .cat-grid .row {
 grid-template-columns: 1fr 1fr 0.5fr;
 align-items: center;
 }
 .cat-grid .row {
 border-bottom: 1px solid rgba(0, 0, 0, 0.1);
 }
 .cat-grid .row:last-child {
 border-bottom: none;
 }

#### Input

#### Output

![](images/cat-sample-input.jpg)
![](images/cat-sample-output.jpg)

[Live demo by Christopher Hesse](https://affinelayer.com/pixsrv/)

 When supplied with unusual constraints, pix2pix can produce
 striking images:

![](images/bread-cat-input.jpg)
![](images/bread-cat-output.jpg)
[Bread cat by Ivy Tsai](https://twitter.com/ivymyt/status/834174687282241537)

![](images/cat-beholder-input.jpg)
![](images/cat-beholder-output.jpg)
[Cat beholder by Marc Hesse](https://affinelayer.com/pixsrv/beholder.jpg)

![](images/spiral-cat-input.jpg)
![](images/spiral-cat-output.jpg)
Spiral cat

 This is perhaps not high creativity of a Picasso-esque level. But
 it is still surprising. It’s certainly unlike images most of us
 have ever seen before. How does pix2pix and its human user achieve
 this kind of result?

 Unlike our earlier examples, pix2pix is not a generative model.
 This means it does not have a latent space or a corresponding
 space of natural images. Instead, there is a neural network,
 called, confusingly, a generator – this is not meant in the
 same sense as our earlier generative models – that takes as
 input the constraint image, and produces as output the filled-in
 image.

 The generator is trained adversarially against a discriminator
 network, whose job is to distinguish between pairs of images
 generated from real data, and pairs of images generated by the
 generator.

 While this sounds similar to a conventional GAN, there is a
 crucial difference: there is no latent vector input to the
 generatorActually, Isola *et
 al* experimented with adding such a latent vector to
 the generator, but found it made little difference to the
 resulting images.. Rather, there is simply an input
 constraint. When a human inputs a constraint unlike anything seen
 in training, the network is forced to improvise, doing the best it
 can to interpret that constraint according to the rules it has
 previously learned. The creativity is the result of a forced
 merger of knowledge inferred from the training data, together with
 novel constraints provided by the user. As a result, even
 relatively simple ideas – like the bread- and beholder-cats
 – can result in striking new types of images, images not
 within what we would previously have considered the space of
 natural images.

Conclusion
----------

 It is conventional wisdom that AI will change how we interact with
 computers. Unfortunately, many in the AI community greatly
 underestimate the depth of interface design, often regarding it as
 a simple problem, mostly about making things pretty or
 easy-to-use. In this view, interface design is a problem to be
 handed off to others, while the hard work is to train some machine
 learning system.

 This view is incorrect. At its deepest, interface design means
 developing the fundamental primitives human beings think and
 create with. This is a problem whose intellectual genesis goes
 back to the inventors of the alphabet, of cartography, and of
 musical notation, as well as modern giants such as Descartes,
 Playfair, Feynman, Engelbart, and Kay. It is one of the hardest,
 most important and most fundamental problems humanity grapples
 with.

 As discussed earlier, in one common view of AI our computers will
 continue to get better at solving problems, but human beings will
 remain largely unchanged. In a second common view, human beings
 will be modified at the hardware level, perhaps directly through
 neural interfaces, or indirectly through whole brain emulation.

 We’ve described a third view, in which AIs actually change
 humanity, helping us invent new cognitive technologies, which
 expand the range of human thought. Perhaps one day those
 cognitive technologies will, in turn, speed up the development of
 AI, in a virtuous feedback cycle:

![](images/cycle.svg)

 It would not be a Singularity in machines. Rather, it would be a
 Singularity in humanity’s range of thought. Of course, this loop
 is at present extremely speculative. The systems we’ve described
 can help develop more powerful ways of thinking, but there’s at
 most an indirect sense in which those ways of thinking are being
 used in turn to develop new AI systems.

 Of course, over the long run it’s possible that machines will
 exceed humans on all or most cognitive tasks. Even if that’s the
 case, cognitive transformation will still be a valuable end, worth
 pursuing in its own right. There is pleasure and value involved
 in learning to play chess or Go well, even if machines do it
 better. And in activities such as story-telling the benefit often
 isn’t so much the artifact produced as the process of construction
 itself, and the relationships forged. There is intrinsic value in
 personal change and growth, apart from instrumental benefits.

 The interface-oriented work we’ve discussed is outside the
 narrative used to judge most existing work in artificial
 intelligence. It doesn’t involve beating some benchmark for a
 classification or regression problem. It doesn’t involve
 impressive feats like beating human champions at games such as
 Go. Rather, it involves a much more subjective and
 difficult-to-measure criterion: is it helping humans think and
 create in new ways?

 This creates difficulties for doing this kind of work,
 particularly in a research setting. Where should one publish?
 What community does one belong to? What standards should be
 applied to judge such work? What distinguishes good work from
 bad?

 We believe that over the next few years a community will emerge
 which answers these questions. It will run workshops and
 conferences. It will publish work in venues such as Distill. Its
 standards will draw from many different communities: from the
 artistic and design and musical communities; from the mathematical
 community’s taste in abstraction and good definition; as well as
 from the existing AI and IA communities, including work on
 computational creativity and human-computer interaction. The
 long-term test of success will be the development of tools which
 are widely used by creators. Are artists using these tools to
 develop remarkable new styles? Are scientists in other fields
 using them to develop understanding in ways not otherwise
 possible? These are great aspirations, and require an approach
 that builds on conventional AI work, but also incorporates very
 different norms.

Using Artificial Intelligence to Augment Human Intelligence

Summary: By creating user interfaces which let us work with the representations inside machine learning models, we can give people new tools for reasoning.

deeplearn.js font demo

Analyzing 50k fonts using deep neural networks

Sampling Generative Networks

Writing with the Machine

A Neural Representation of Sketch Drawings

Taking The Robots To Design School, Part 1

Hierarchical Variational Autoencoders for Music

Eccentric School of Painting Increased Its Vogue in the Current Art Exhibition — What Its Followers Attempt to Do

Thought as a Technology

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Image-to-Image Translation with Conditional Adversarial Networks