I'm happy a paper like this exists, because I think this kind of work is crucial in evaluating whether interpretability methods we're building are actually useful. I'm not surprised by the results, because this hasn't been done rigorously before, so researchers have never had any idea whether their method has produced good explanations or not.

The study is weakened by the low sample size, which makes many of the p-values not significant. My intuition says a few more of the methods would produce statistically significant positive results in one of the domains/prediction settings if the sample size was bigger, but it seems like some settings (forward prediction, and textual data) are very hard to improve, with none of the methods getting a better improvement in performance than 5.7% (which had a p-value of 0.197).

A really interesting point is the lack of strong correlation between user-preference and performance improvement. This could be explained by the fact that most of the methods are ineffective at performance improvement, but it seems plausible (to me) that it could hold even if some methods were effective: If the model behaviour being explained can't be explained cleanly, then methods which do explain the behaviour might produce messy and confusing (but true) explanations and hence get lower ratings from users than methods which give clean and clear (but false) explanations. I think this stems from the problem of a lack of definition of what exactly the goal is for these interpretation methods. Without a goal in mind, it's impossible to measure whether the method achieves this goal. I think working towards some form of quantifiable measurement is useful particularly for comparing methods as, if this study's evidence is anything to go by, asking humans to evaluate the model's output might not be the most useful evaluation.

Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

In this paper the authors perform user tests on 5 different model agnostic interpretability methods: LIME, Anchor, Decision Boundary, Prototype Model and a Composite model (LIME Anchor and Decision Boundary). The use cases they test are a tabular dataset predicting income, and a movie-review dataset predicting sentiment of the review from a single sentence.

Their experimental setup consists of 2 tests: **forward prediction** and **counterfactual prediction**. In forward prediction, the user is shown 16 examples of inputs and corresponding outputs and explanations, and then must predict the model’s output on new inputs (without the explanation, which often gives away the answer). In counterfactual prediction, after seeing 16 examples, the user is given an input-output-explanation triple, and then must predict how the output changes for a specific perturbation of the input.

Throughout the results they use a significance threshold of p < 0.05 (they don't use Bonferroni corrections). Their study has responses from 32 different students who'd taken at least 1 computer science course, with some screened out for outliers or low accuracy during training. There are approximately 200 individual predictions for each method/dataset-type combination, and each method/prediction-type combination.

Overall, their results show that **only LIME (Local Interpretable Model-agnostic Explanation) helps improve performance** with statistical significance on the tabular dataset across both prediction settings, and **only the Prototype model in counterfactual prediction across both datasets**. **No other result was statistically significant.** The improvement in accuracy for the statistically significant results is around 10% (from 70% to 80% in the Tabular dataset with LIME, and 63% to 73% for Prototype in counterfactual prediction).

They also showed that **user's ratings of the explanation method didn't correlate in a statistically significant way with the improvement the model gave to their predictions.**


Understanding deep learning through neuron deletion


I liked the paper, and I'm in favour of new work on interpreting reinforcement learning agents; I think it's under explored and useful, and relevant to AI safety. The methods seem in a similar vein to Feature Visualisation methods for classic vision, but focused solely on the resulting behaviour of the agent; it'd be interesting to see if such methods can give insight into the internals of RL agents. It's also a shame the demonstration of the results is wholly qualitative; the authors demonstrate some apparent flaws in the agents, but don't produce any results which show that the insights their method produces are useful. I think the insights are useful, but it's difficult to validate the claim, and I'm cautious of work which produces interesting and seemingly insightful methods but doesn't validate that the methods produce actually useful insight.

Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents

Christian Rupprecht, Cyril Ibrahim, Christopher J. Pal

This paper proposes a new visualization tool in order to understand the behaviour of agents trained using deep reinforcement learning. Specifically, they train a generative model which produces game states, and then optimise a distribution over state embeddings according to some target function (such as high reward for taking a specific action). By sampling from the resulting distribution, they create a diverse set of realistic states that score highly according to the target function. They propose a few target cost functions, which allow them to optimise for states in which the agent takes a particular action, states which are high reward (worst Q-value is large), states which are low reward (best Q-value is small), and critical states (large difference in Q value). They demonstrate results on Atari games as well as a simulated driving environment.


It's interesting to see methods able to identify which parts of the training data have an impact on the decisions of a model. I think the approach taken here (and in Influence Functions) of using the change in the test loss is OK, but it doesn't seem to be exactly what I think when I say "which datapoints had the most influence on this decision being made in this way?". It's also difficult to compare these methods without either a benchmark, a human experiment, or some way of demonstrating the method has produced novel insight which has been verified. The mislabelled data experiment partially fulfils this, but isn't what these methods are ultimately designed for, and is hence unsatisfactory.

Understanding Black-box Predictions via Influence Functions

Estimating Training Data Influence by Tracking Gradient Descent

Garima Pruthi, Frederick Liu, Mukund Sundararajan, Satyen Kale

This paper presents the TrackIn method for tracking the influence of training datapoints on the loss on a test datapoint. The purpose of the method is to discover influential training points for decisions made on the testing set. This is defined (loosely) for a training point **x** and test point **z** as the total change in loss on **z** caused by training on **x**. They present several approximations and methods for calculating this quantity efficiently, *allowing them to scale their method to ResNet 50 models trained on ImageNet*

The standard method of evaluation for these kinds of methods is finding mislabelled examples in the training dataset. Mislabelled examples are likely to have a strong positive influence on their own loss (strong as they're outliers, and positive as they'll reduce their own loss). Sorting the training dataset in decreasing order of this self-influence, we should hence expect to see more mislabelled examples at the beginning of the list. We can measure what proportion of mislabelled examples is present in each different initial segments of the list. The authors perform this experiment on CiFAR, first training a model to convergence, and then mislabelling 10% of the training set as the next highest predicted class, and then retraining a new model on which TrackIn is run. *When compared to the two previous methods from the literature (Influence Functions and Representer Points), TrackIn recovers more than 80% of the mislabelled data in the first 20% of the ranking, whereas the other methods recover less than 50% at the same point. For all segments TrackIn does significantly better.*

They demonstrate the method on a variety of domains, including NLP tasks and vision tasks. The influential examples found seem reasonable, but there's no quantification of these results.


The dataset perspective on visualization seems pretty useful as a quick diagnostic tool for practitioners, but less useful than feature visualization for a detailed understanding of a model. While I think that it is good to highlight invariances, I am not convinced that rotational invariance is actually desirable for visualizing intermediate layers of a neural network, as most nonlinearities are strongly affected by rotations.

Visualizing Neural Networks with the Grand Tour

Mingwei Li, Zhenge Zhao, Carlos Scheidegger

Visualizing a complete dataset instead of single input examples is helpful when we want to analyze the relationships between different input examples and how their classification changes during training, as we can do so by looking at a single video.

The authors use an example on MNIST in which the network learns to classify the numbers 1 and 7 in an almost discrete fashion during particular epochs to compare different methods for visualizing how the dataset is classified. They find that one problem with nonlinear dimensionality reduction like t-SNE and UMAPs is that changes to a subset of the dataset can strongly affect how unchanged data points are represented. Then they compare this to the Grand Tour, a classical technique that projects the data into two dimensions from varying points of view. As projections are linear in the input variables, it is rather easy to reason about how changes in the data affect this visualization and the times the classes 1 and 7 are learnt are indeed quite salient in their example. Another advantage of this method is that confusion between two specific classes can be identified more easily, as the corresponding data points will be projected onto the line connecting the clusters for these classes. A similar approach can be taken on a network's hidden layers to identify the layer in which different classes become clearly distinguishable. They find that they can identify adversarial examples generated by FGSM by looking at the second to last layer, where the adversarial examples form a cluster distinct from the real images.

As the Grand Tour involves varying rotations, it is basically unaffected by rotations of the data. The authors argue that this is a feature, as rotations are small changes to the data and should not have a large effect on the visualization.


Unsupervised Learning of Neural Networks to Explain Neural Networks

Quanshi Zhang, Yu Yang, Yuchen Liu, Ying Nian Wu, Song-Chun Zhu


Conceptual clarity about these concepts seems useful for more nuanced discussions and I like the emphasis on the importance of the observer's model for interpretability. However, it seems like concepts around interpretability that are not contingent on an agent's actual behaviour (or explicit planning) would be even more important. Many state-of-the-art RL agents do not perform explicit planning, and ideally we would like to know something about their behaviour before we deploy them in novel environments.

Explicability? Legibility? Predictability? Transparency? Privacy? Security? The Emerging Landscape of Interpretable Agent Behavior

Tathagata Chakraborti, Anagha Kulkarni, Sarath Sreedharan, David E. Smith, Subbarao Kambhampati2

This paper reviews and discusses definitions of concepts of interpretable behaviour. The first concept, **explicability** measures how close an agent's behaviour is to the observer's expectations. An agent that takes a turn while its goal is straight ahead does not behave explicably by this definition, even if it has good reasons for its behaviour, as long as these reasons are not captured in the observer's model. **Predictable** behaviour reduces the observer's uncertainty about the agent's future behaviour. For example, an agent that is tasked to wait in a room behaves more predictably if it shuts itself off temporarily than if it paced around the room. Lastly, **legibility** or **transparency** reduces observer's uncertainty about an agent's goal. This can be achieved by preferentially taking actions that do not help with other goals. For example, an agent tasked with collecting apples can increase its legibility by actively avoiding pears, even if it could collect them without any additional costs.

These definitions do not always assume correctness of the observer's model. In particular, an agent can explicably and predictably achieve the observer's task in a specific context while actually trying to do something else. Furthermore, these properties are dynamic. If the observer's model is imperfect and evolves from observing the agent, formerly inexplicable behaviour can become explicable as the agent's plans unfold.


Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations

Andrew Slavin Ross, Michael C. Hughes, Finale Doshi-Velez


I'd recommend you actually look at and play around with the visualization, it's very nice. The summary is short because the value of the work is in the visualization, not in the technical details.

Visualizing memorization in RNNs

This is a short Distill article that showcases a visualization tool that demonstrates how contextual information is used by various RNN units (LSTMs, GRUs, and nested LSTMs). The method is very simple: for each character in the context, they highlight the character in proportion to the gradient of the logits with respect to that character. Looking at this visualization allows us to see that GRUs are better at using long-term context, while LSTMs perform better for short-term contexts.


Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, Jeff Clune


Regularizing Black-box Models for Improved Interpretability

Gregory Plumb, Maruan Al-Shedivat, Eric Xing, Ameet Talwalkar


Tathagata Chakraborti, Anagha Kulkarni, Sarath Sreedharan, David E. Smith, Subbarao Kambhampati


Please Stop Explaining Black Box Models for High Stakes Decisions


Representer Point Selection for Explaining Deep Neural Networks

Chih-Kuan Yeh, Joon Sik Kim, Ian E.H. Yen, Pradeep Ravikumar


Towards Governing Agent's Efficacy: Action-Conditional β-VAE for Deep Transparent Reinforcement Learning

John Yang, Gyujeong Lee, Minsung Hyun, Simyung Chang, Nojun Kwak


Explaining Explanations in AI

Brent Mittelstadt, Chris Russell, Sandra Wachter


Training Machine Learning Models by Regularizing their Explanations


Towards Better Interpretability in Deep Q-Networks


Transparency and Explanation in Deep Reinforcement Learning Neural Networks

Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, Katia Sycara


I want people to read this paper, but I'm a bit biased here -- I've thought about this area a lot, and I expect that we'll need to have inductive biases of the form "something like an algorithm/program" to get good generalization from AI, and this paper is some evidence towards that. It's also the first paper I know of that makes RL-learned policies look like simple programs.

Programmatically Interpretable Reinforcement Learning

Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri

This work uses program synthesis in order to get interpretable reinforcement learning policies. Some of you can probably guess that I'm very excited by this paper :P As with most program synthesis techniques, they define a space of possible programs (policies), and then search through the space for the program that achieves the highest reward. Since they are using program synthesis, they can take advantage of standard tricks such as sketching. They also train a deep RL agent and use the agent to give feedback to the program synthesis algorithm, so that the algorithm produces the program whose outputs are closest to the outputs of the deep RL policy. They evaluate on TORCS (a racecar simulator) and find that the policy does almost as well as deep RL. However, it has a few major advantages over deep RL. Since it is a program, it is much more interpretable -- a human can actually look at the resulting program and understand it (and hence the title of the paper). It is also possible to use formal verification methods to prove properties about the program (whereas neural nets are often too large for these techniques to work). But perhaps most importantly, restricting your class of functions to the space of (small) programs is often a very useful inductive bias, and it is no different in this case -- the learned programs perform much better than deep RL when run on a new unseen track, showing good generalization.


AGI 2018: Artificial General Intelligence

This seems like a useful taxonomy that hits the kinds of interpretability research I know about, though the citation list is relatively low for a summary paper, and there are a few papers I expected to see that weren't present. On the other hand, I'm not actively a part of this field, so take it with a grain of salt.

Techniques for Interpretable Machine Learning

This paper summarizes work on interpretability, providing a classification of different ways of achieving interpretability. There are two main axes -- first, whether you are trying to gain insight into the entire model, or its classification of a particular example; and second, whether you try to create a new model that is inherently interpretable, or whether you are post-hoc explaining the decision made by an uninterpretable model. The whole paper is a summary of techniques, so I'm not going to summarize it even further.


Conference on Robot Learning (CoRL) 2018

Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, Rory Sayres


Measuring abstract reasoning in neural networks

David G. T. Barrett, Felix Hill, Adam Santoro,
Ari Morcos, and Timothy Lillicrap


I very much like the general idea that we really can look at counterfactuals for artificial agents, given that we can control their inputs and internal state. This is the same idea underlying <@cross-examination@>(@Writeup: Progress on AI Safety via Debate@), as well as various other kinds of interpretability research.

In addition, one nice aspect of causal models as your form of “explanation” is that you can modulate the size of the causal model based on how many nodes you add to the graph. The full causal model for e.g. GPT-3 would be way too complex to understand, but perhaps we can get a high-level understanding with a causal model with higher-level concepts. I’d be very interested to see research tackling these sorts of scaling challenges.

Paper: Causal Analysis of Agent Behavior for AI Safety

What mechanisms drive agent behaviour?

Grégoire Déletang, Jordi Grau-Moya, Miljan Martic, Tim Genewein, Tom McGrath, Vladimir Mikulik, Markus Kunesch, Shane Legg, Pedro A. Ortega

A common challenge when understanding the world is that it is very hard to infer causal structure from only observational data. Luckily, we aren’t limited to observational data in the case of AI systems: we can intervene on either the environment the agent is acting in, or the agent itself, and see what happens. In this paper, the authors present an “agent debugger” that helps with this, which has all the features you’d normally expect in a debugger: you can set breakpoints, step forward or backward in the execution trace, and set or monitor variables.

Let’s consider an example where an agent is trained to go to a high reward apple. However, during training the location of the apple is correlated with the floor type (grass or sand). Suppose we now get an agent that does well in the training environment. How can we tell if the agent looks for the apple and goes there, rather than looking at the floor type and going to the location where the apple was during training?

We can’t distinguish between these possibilities with just observational data. However, with the agent debugger, we can simulate what the agent would do in the case where the floor type and apple location are different from how they were in training, which can then answer our question.

We can go further: using the data collected from simulations using the agent debugger, we can also build a causal model that explains how the agent makes decisions. We do have to identify the features of interest (i.e. the nodes in the causal graph), but the probability tables can be computed automatically from the data from the agent debugger. The resulting causal model can then be thought of as an “explanation” for the behavior of the agent.


I am not sure whether the authors used an actual AI system or just handcrafted the input-prediction-explanation tuples, and how that might affect the correlation between explanations and the system's outputs, which can influence trust. Overall, the study reinforces my prior that trust induced by explanations is not a good predictor of an AI system's usefulness, but I am more sceptical that the differences between inductive and deductive explanations will be the same in different contexts.

Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems

Zana Buçinca*, Phoebe Lin*, Krzysztof Z. Gajos, Elena L. Glassman

As humans and AI systems have different strengths, it might make sense to combine them into human+AI teams for decision-making tasks. However, this does not always work well: if the human puts too little trust in a competent AI, the AI is of little use, and if they put too much trust in an incompetent AI, they might make worse decisions than had they been on their own.  A lot of explainability research has focused on instilling more trust in AI systems without asking how much trust would be appropriate, even though there is research showing that hiding model bias instead of truthfully revealing it can increase trust in an AI system.

The authors conduct two experiments using an AI system that predicts nutrition information from pictures of food. In the first experiment, participants were asked to predict the AI's decision based on the ground truth and one of two types of explanations. In the inductive condition, the explanation consisted of a series of images the AI had identified as similar. In the deductive condition, subjects were shown a list of main ingredients identified by the AI. Subjects put more trust in the inductive explanations but were equally good at predicting the system's output in both cases. In the second experiment, a new set of subjects was asked to predict nutritional values with the help of the AI's predictions. Overall, access to the AI strongly improved the subjects' accuracy from below 50% to around 70%, which was further boosted to a value slightly below the AI's accuracy of 75% when users also saw explanations. This time, subjects put more trust in the AI when given deductive explanations, but performed better when given inductive explanations, as they were more likely to go against the AI's wrong decisions in that case.

The authors hypothesize that the between-task difference in which explanations are trusted more is connected to the cognitive effort required by the tasks and for understanding the explanations, combined with human reluctance to exert mental effort. They suggest to pay more attention to the exact form of the human-AI interaction and recommend to view AI-based decision aids as sociotechnical systems that are to be evaluated by their usefulness for actual decision making, rather than trust.


AIES '18 Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society

Towards Robust Interpretability with Self-Explaining Neural Networks


Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning

Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, Lalana Kagal


Neural Stethoscopes: Unifying Analytic, Auxiliary and Adversarial Network Probing

Fabian B. Fuchs, Oliver Groth, Adam R. Kosiorek, Alex Bewley, Markus Wulfmeier, Andrea Vedaldi, Ingmar Posner


I like the idea of interpretability being aimed at trying to fill in mis- or under-specified optimisation objectives. I think this proposes that interpretability is more useful for outer alignment, which is interesting as I think that most people in the safety community think interpretability could help with inner alignment (for example, see <@An overview of 11 proposals for building safe advanced AI@>, in which transparency (which could be seen as interpretability) is used to solve inner alignment in 4 of the proposals).

Towards A Rigorous Science of Interpretable Machine Learning

This paper from 2017 discusses the field of interpretability research, and how it can be made more rigorous and well-defined. The authors first highlight the problem of defining interpretability in the first place - they don't have a resolution to this problem, but suggest that we can think of interpretability in terms of what it's used for. They claim that interpretability is used for confirming other important desiderata in ML systems, which stem from an incompleteness in the problem formalization. For example, if we want a system to be unbiased but aren't able to formally specify this in the reward function, or the reward we're optimising for is only a proxy of the true reward, then we could use interpretability to inspect our model and see whether it's reasoning how we want it to.

The authors next move on to discussing how we can evaluate interpretability methods, providing a taxonomy of different evaluation methods: Application-grounded is when the method is evaluated in the context it will actually be used in, by real humans (i.e. doctors getting explanations for AI diagnoses); Human-grounded is about conducting simpler human-subject experiments (who are perhaps not domain experts) using possibly simpler tasks than what the intended purpose of the method is; Functionally-grounded is where no humans are involved in the experiments, and instead some formal notion of interpretability is measured for the method to evaluate its quality. Each of these evaluation methods can be used in different circumstances, depending on the method and the context it will be used in.

Finally, the authors propose a data-driven approach to understanding the factors which are important in interpretability. They propose to try and create a dataset of applications of machine learning models to tasks, and then analyse this dataset to find important factors. They list some possible task- and method- related factors, and then conclude with recommendations to researchers doing interpretability.


This method has the nice property that it visualizes exactly what the neural net is doing -- it isn’t “rationalizing” an explanation, or eliding potentially important details. It is possible to gain interesting insights about the model: for example, the logit for digit 2 is always -2.39, implying that everything else is computed relative to -2.39. Looking at the images for digit 7, it seems like the model strongly believes that sevens must have the top few rows of pixels be blank, which I found a bit surprising. (I chose to look at the digit 7 somewhat arbitrarily.)

Of course, since the technique doesn’t throw away any information about the model, it becomes very complicated very quickly, and wouldn’t scale to larger models.

Sparsity and interpretability?

Stanislav Böhm, Robert Kirk, Tomáš Gavenčiak

If you want to visualize exactly what a neural network is doing, one approach is to visualize the entire computation graph of multiplies, additions, and nonlinearities. While this is extremely complex even on MNIST, we can make it much simpler by making the networks _sparse_, since any zero weights can be removed from the computation graph. Previous work has shown that we can remove well over 95% of weights from a model without degrading accuracy too much, so the authors do this to make the computation graph easier to understand.

They use this to visualize an MLP model for classifying MNIST digits, and for a DQN agent trained to play Cartpole. In the MNIST case, the computation graph can be drastically simplified by visualizing the first layer of the net as a list of 2D images, where the kth activation is given by the dot product of the 2D image with the input image. This deals with the vast majority of the weights in the neural net.


I like this general direction of thought, and hope that people continue to pursue it, especially since I think interpretability will be necessary for inner alignment. I think it would be easier to build on the ideas in this post if they were made more concrete.

How can Interpretability help Alignment?

Robert Kirk, Tomáš Gavenčiak, Flo Dorner

Interpretability seems to be useful for a wide variety of AI alignment proposals. Presumably, different proposals require different kinds of interpretability. This post analyzes this question to allow researchers to prioritize across different kinds of interpretability research.

At a high level, interpretability can either make our current experiments more informative to help us answer _research questions_ (e.g. “when I set up a <@debate@>(@AI safety via debate@) in this particular way, does honesty win?”), or it could be used as part of an alignment technique to train AI systems. The former only have to be done once (to answer the question), and so we can spend a lot of effort on them, while the latter must be efficient in order to be competitive with other AI algorithms.

The Authors then analyze how interpretability could apply to several alignment techniques, and come to several tentative conclusions. For example, they suggest that for recursive techniques like iterated amplification, we may want comparative interpretability, that can explain the changes between models (e.g. between distillation steps, in iterated amplification). They also suggest that by having interpretability techniques that can be used by other ML models, we can regularize a trained model to be aligned, without requiring a human in the loop.


I'm in favour of developing methods which allow us to rigorously compare interpretability methods. This benchmark is a step in the right direction, but I think it does have several flaws:

1. In images especially, there's high correlation between pixels, and greying out pixels isn't the same as "removing" the feature (the model trained on the altered images could learn something like "these pixels are greyed out (and hence important), so this must be a bird otherwise the pixels wouldn't be important").

2. The benchmark doesn't measure exactly what these methods are trying to capture. The methods are trying to answer "what parts of this image were important for making this classification?", which is at least slightly different from "what parts of this image, when removed, will prevent a new model from classifying accurately?".

I'd be interested in seeing the benchmark (or something conceptually similar) applied to domains other than images, where the correlation between features is lower: I imagine for some kinds of tabular data the methods would perform much better (although the methods have mostly been designed to work on images rather than tabular data).

A Benchmark for Interpretability Methods in Deep Neural Networks

Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, Been Kim

This paper presents an automatic benchmark for *feature importance* methods (otherwise known as saliency maps) called *RemOve And Retrain* (ROAR). The benchmark follows the following procedure:

1. Train an image classifier on a dataset (they use ResNet-50s on ImageNet, and get about 77% accuracy)

2. Measure the test-set accuracy at convergence

3. Using the feature importance method, find the most important features in the dataset, and remove them (by greying out the pixels)

4. Train another model on this new dataset, and measure the new test-set accuracy

5. **The difference between the accuracy in (4) and in (2) is the measure of how effective the feature importance method is at finding important features**

The idea behind retraining is that giving the original classifier images where many pixels have been greyed out will obviously result in lower accuracy, as they're out of the training distribution. Retraining solves this problem.

They benchmark a variety of feature importance methods (Gradient heatmap, Guided backprop, Integrated gradients, Classic SmoothGrad, SmoothGrad^2, VarGrad) on their benchmark, and compare to a random baseline, and a Sobel Edge detector (a hard-coded algorithm for finding edges in images). **Only SmoothGrad^2 and VarGrad (which are both methods which ensemble other feature importance methods) do better than random.** They can't explain why these methods perform better than other methods. They also note that even when removing 90% of the pixels in every image (i.e. the random baseline), the accuracy only drops from 77% to 63%, which shows how correlated pixels in images are.


This paper isn't revolutionary in its approach, but it's interesting to see work on interpreting RL agents, and the fact that the interpretability is built-in is interesting: it gives us a harder guarantee that this visualisation is actually showing us the parts of the input that the model thinks of as important, as they actually are important in its processing. It's promising to see that the in-built interpretability also didn't seem to penalise the performance much - it would be interesting to see this method applied to other, stronger kinds of models and see whether it still produces useful visualisations and how it affects their performance.

Towards Interpretable Reinforcement Learning Using Attention Augmented Agents

Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, Danilo Jimenez Rezende

In this paper the authors train a reinforcement learning agent with a soft attention module built into it. The attention module forms a bottleneck between the visual input and the network choosing the next action, which forces the model to learn to attend to only important parts of the scene. This means they can visualise which parts of the input the model thinks are important, as those are the parts of the input that the model is attending to. The queries to the attention model are determined by a top level recurrent network, without input from the current image, so act as a form of "top down" attention, where the top controller can be imagined to be querying the processed image for various locations and objects.

Having trained this agent (which still gets competitive performance with SOTA RL models on a fair few ATARI games), they qualitatively evaluate the attention visualisation on a variety of games. They find several common strategies in the attention schemes, such as the agents attending to specific points until an object crosses the point ("Tripwires"). The attention is computed over both regular pixels, as well as Fourier-based positional encoding. Thanks to this and other aspects of their architecture, the authors can check whether the queries are focused on pixel values (i.e. looking for a specific pattern of pixels anywhere) or on location features (i.e. asking what pixels are present at a specific location). For example, they find that the agent often queries the location where the score is displayed, presumably because it is useful for calculating the value function. They also compare their method with self-attention based models, and with other saliency methods.

The best way to get a feel for the visualisations is to go to the paper's website and watch the example videos.


It's striking to me how different this is from other work, e.g. <@Explicability? Legibility? Predictability? Transparency? Privacy? Security? The Emerging Landscape of Interpretable Agent Behavior@>. It seems like interpretability is a really vague, nebulous term that has so far (to my limited knowledge) not been made precise.

What is Interpretability?

Robert Kirk, Tomáš Gavenčiak, Stanislav Böhm

This post categorizes several interpretability methods based on their _goal_ and how they _enable humans_ to achieve the goal.


I like this type of investigation, but don't find their actual conclusions compelling. In particular, it doesn't matter whether "meaningless" adversarial attention weights can lead to the same classifications, as long as the ones actually learned by the system are interpretable. Also, the lack of correlation between attention weights and other methods could be explained either by attention weights being much worse than the other methods, or much better, or merely useful for different purposes.

Attention is not Explanation

This paper explores the usefulness of attention weights in interpreting neural networks' performance on NLP tasks. The authors present two findings: firstly, that attention weights are only weakly correlated with other metrics of word importance; and secondly, that there often exist adversarially-generated attention weights which are totally different from the learned weights, but which still lead to the same outputs. They conclude that these results undermine the explanatory relevance of attention weights.


I think the main point that explanation and manipulation can often look very similar is an important one. However, I'm not convinced that there aren't any ways of specifying the difference between them. Other factors which seem relevant include what mental steps the explainer/manipulator is going through, and how they would change if the statement weren't true or if the explainee were significantly smarter.

How much can value learning be disentangled?

Stuart argues that there is no clear line between manipulation and explanation, since even good explanations involve simplification, omissions and cherry-picking what to emphasise. He claims that the only difference is that explanations give us a better understanding of the situation - something which is very subtle to define or measure. Nevertheless, we can still limit the effects of manipulation by banning extremely manipulative practices, and by giving AIs values that are similar to our own, so that they don't need to manipulate us very much.


Personalized explanation in machine learning


Sanity Checks for Saliency Maps

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim


Stakeholders in Explainable AI

Alun Preece, Dan Harborne, Dave Braines, Richard Tomsett, Supriyo Chakraborty

There are at least four groups for whom "explainable" AI is relevant: developers (who want AI to be easier to work with), theorists (who want to understand fundamental properties of AI), ethicists (who want AI to behave well) and users (who want AI to be useful). This has complicated work on explanability/interpretability: the first two groups focus on understanding how a system functions internally (described in this paper as "verification"), while the latter two focus on understanding what the system does ("validation"). The authors propose an alternative framing of interpretability, based on known knowns, unknown knowns, etc.


I like the auditing game as a framework for constructing benchmarks for worst-case interpretability -- you can instantiate a particular benchmark by defining a specific adversary (or distribution of adversaries). Automating auditing against a human attacker seems like a good long-term goal, but it seems quite intractable given current capabilities.

Automating Auditing: An ambitious concrete technical research proposal

A core worry with inner alignment is that we cannot determine whether a system is deceptive or not just by inspecting its behavior, since it may simply be behaving well for now in order to wait until a more opportune moment to deceive us. In order for interpretability to help with such an issue, we need _worst-case_ interpretability that surfaces all the problems in a model. When we hear “worst-case”, we should be thinking of adversaries.

This post considers the _auditing game_, in which an attacker introduces a vulnerability in the model to violate some known specification, and the auditor must find and describe the vulnerability given only the modified model (i.e. it does not get to see the original model, or what the adversary did). The attacker aims to produce the largest vulnerability that they can get away with, and the auditor aims to describe the vulnerability as completely as possible. Note that both the attacker and the auditor can be humans (potentially assisted by AI tools). This game forms a good benchmark for worst-case interpretability work.

While the author is excited about direct progress on this game (i.e. better and better human auditors), he is particularly interested in fully _automating_ the auditors. For example, we could collect a dataset of possible attacks and the corresponding desired audit, and finetune a large language model on such a dataset.


Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning


2018 ICML Workshop on Human Interpretability in Machine Learning

Intelligible Artificial Intelligence


Counterfactual States for Atari Agents via Generative Deep Learning

Matthew L. Olson, Lawrence Neal, Fuxin Li, Weng-Keen Wong


Interpretable Reinforcement Learning with Ensemble Methods


The What, the Why, and the How of Artificial Explanations in Automated Decision-Making


IEEE International Conference on Robotics and Automation (ICRA) 2019

Learning Explanations from Language Data

David Harbecke, Robert Schwarzenberg, Christoph Alt


XAI workshop on the IJCAI conference 2018

I wish that they had let users choose the questions in their user study, rather than just evaluating questions that had been generated by their method where they wrote the alternative policy using template policies they had written. I'd be pretty excited and think it was a good step forward in this area if end users (i.e. not ML researchers) could ask novel contrastive questions (perhaps in some restricted class of questions).

Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences

Jasper van der Waa, Jurriaan van Diggelen, Karel van den Bosch, Mark Neerincx

This paper aims to provide contrastive explanations for the behavior of an RL agent, meaning that they contrast why the RL agent used one policy instead of another policy. They do this by computing the expected outcomes under the alternate policy, and then describing the difference between the two. (An outcome is a human-interpretable event -- they assume that they are given a function that maps states to outcomes.)


This looks like that: deep learning for interpretable image recognition

Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, Cynthia Rudin


Interpretable Discovery in Large Image Data Sets


Open the Black Box Data-Driven Explanation of Black Box Decision Systems

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Luca Pappalardo, Salvatore Ruggieri, Franco Turini


How Can Neural Network Similarity Help Us Understand Training and Generalization?

Maithra Raghu, Google Brain Team and Ari S. Morcos, DeepMind


You do need to be familiar with IDA. I found it useful to learn about the particular assumption made (corrigibility for low complexity queries). In addition, this post makes the proof of safety substantially more concrete.

Universality and security amplification

When you are learning from a source that you can't fully trust (such as humans), one major issue is that the errors that the source makes may be correlated, so you can't just correct for them by averaging over many samples. Paul's overall approach to AI alignment requires the overseer to be corrigible, but there may be attacks that cause the overseer to behave incorrigibly. This post explains how we might solve this, even though the errors can be correlated. Specifically, we assume that the overseer is corrigible for queries that are sufficiently low complexity (for some definition of complexity, he gives an example definition). Then, by automatically breaking down an input task into low complexity subqueries (eg. using meta-execution), we can make sure the overseer always remains corrigible. The post has a lot more detail on each of these points, of course.


I enjoyed this a lot: the problems found were crisp and the solutions had good arguments that they actually solved the identified problem. Reading through the actual examples and arguments made me more optimistic about debate in general, mostly from a felt sense that the actual concrete results were getting closer to matching the theoretical ideal, and that there actually could be reasonable solutions to "messy" problems like ambiguity.

The full post has formal explanations and actual examples, which I highly recommend.

Writeup: Progress on AI Safety via Debate

Beth Barnes, Paul Christiano, Long Ouyang, Geoffrey Irving

This post reports on work done on creating a <@debate@>(@AI safety via debate@) setup that works well with human players. In the game, one player is honest (i.e. arguing for the correct answer) and one is malicious (i.e. arguing for some worse answer), and they play a debate in some format, after which a judge must decide which player won the debate. They are using Thinking Physics questions for these debates, because they involve questions with clear answers that are confusing to most people (the judges) but easy for some experts (the players).

Early freeform text debates did not work very well, even with smart, motivated judges. The malicious player could deflect on questions they didn't want to answer, e.g. by claiming that the question was ambiguous and redirecting attention by asking new questions. In addition, when the malicious player got to go first and give an incorrect "framework" for finding the answer, and then made individually true claims to "fill in" the framework, it was hard for the honest player to rebut it. So, they moved to a framework without such asymmetries: both players gave a claim (simultaneously), both gave constructive arguments, and both rebutted the other's arguments. In addition, part of the appeal of debate is that the agents can "zoom in" on the particular disagreement they have, and the judge need only analyze the smallest disagreement in order to declare an overall winner.

This suggests the following setup: players simultaneously provide an answer supported with subclaims. Then, after looking at the other player's answer and subclaims, they can provide objections (perhaps on particular subclaims, or the overall structure of the argument). Players can concede or challenge objections left by the other player; the debate then recurses on one of the challenged objections (currently, the malicious player chooses which claim to recurse on, in order to be conservative). There is a depth limit to the recursion, once it is reached (or if there are no challenged objections) the judge is given a transcript of the debate and must declare a winner.

A new problem arises: the malicious player can exploit ambiguity to their advantage. For example, suppose the malicious player is arguing that there are 5 primes between 0 and 10, and makes the subclaims "there are 3 primes between 0 and 5" and "there are 2 primes between 5 and 10". If challenged on the first, they will say "the three primes are 2, 3, and 5", if challenged on the second, they will say "the two primes are 5 and 7". While in this simple case the honest player could register a top-level objection claiming that the word "between" is ambiguous, and ask for clarification, in general it would be too hard to eliminate all ambiguity.

To mitigate this, the team added a method of _cross examination_, in which each player is allowed to ask an earlier version of the other agent a question, and quote its answer as part of the debate transcript. (The post talks about how this can be approximated with humans, but I won't go into the details.) In the complexity theory analogy, this boosts debate from PSPACE to NEXP. Intuitively, cross examination allows players to force each other to precommit to specific claims and arguments, because if they don't stay consistent in their claims and arguments, then their opponent can demonstrate the inconsistency. Now, if the malicious player tries to make an argument that depends on interpreting an ambiguous concept in two different ways, then the honest player can cross examine and ask which of the two interpretations they mean. If they are inconsistent, that can be demonstrated to the judge; if they consistently answer one way, then the honest player can challenge the part of the argument that depends on the other interpretation.

They then identify several open concerns with debate, of which they highlight the long computation problem. This is a problem when you no longer assume that the debaters have optimal play: in this case, the malicious player could create a complicated argument that neither debater understands well, that supports the malicious case but that the honest player doesn't know how to refute.


I wouldn't read too much into the low accuracies of the depth 2 trees: it seems quite plausible that this is specific to the movie review setting, and in settings with clearer answers you could do better. Like with the previous post, I found the actual examples quite illuminating: it's always interesting to see what happens when theory collides with the real world.

Evaluating Arguments One Step at a Time

This blog post reports on the results of Ought's experiments with <@factored evaluation@>(@Delegating open-ended cognitive work@) by humans on questions about movie reviews. There are two human players, one honest and one malicious. Half the time, the honest player chooses a true top-level claim, decomposes it into supporting subclaims up to some depth, and provides supporting quotes for each claim and subclaim. The other half of the time, the malicious player does the same, except they choose a _false_ top-level claim. In either case, the opposite player then chooses quotes that aim to _refute_ the claims and subclaims. For each claim and subclaim, a new judge evaluates whether that claim is "valid" or "invalid". If _any_ claim is marked invalid, then the top-level claim is judged to be false, otherwise it is judged true.

They found that depth 1 and depth 2 trees outperformed depth 0 trees (for which judgments were at about chance). However, they did not perform amazingly well -- with posthoc hyperparameter tuning, depth 1 and 2 trees got accuracies of 65% and 69% respectively. False positives (accepting false claims) were primarily caused due to malicious claims that couldn't easily be rebutted with the limited budget (but could be rebutted with a larger budget), and individual steps being complex enough that the judge made mistakes. False negatives (rejecting true claims) tend to happen when players choose hard-to-argue true claims, especially ones in which no specific quote clearly supports the claim, even though the full paragraph supports the claim through its tone and style.

There are several approaches that could theoretically solve these problems, such as increasing the size of claim trees, and improving quality control for judgments (e.g. by aggregating multiple judgments together).


Introduction to ascription universality

Published a year ago, [this](https://ai-alignment.com/towards-formalizing-universality-409ab893a456) [sequence](https://ai-alignment.com/informed-oversight-18fcb5d3d1e1) [of](https://ai-alignment.com/universality-and-model-based-rl-b08701394ddd) [five](https://ai-alignment.com/universality-and-consequentialism-within-hch-c0bee00365bd) [posts](https://ai-alignment.com/training-robust-corrigibility-ce0e0a3b9b4d) introduced the idea of _ascription universality_. I didn't really get it on a first reading, and only recently read it in enough detail that I think I understand the main ideas. This entire newsletter will focus on ascription universality; treat all of it as a "Highlight".

The key idea of these posts is that of _universality_: when we can say that some agent "knows everything that any other agent could know". Of course, there will always be some agent with arbitrarily complex beliefs, but we could hope to have agents that know everything that is known by any agent of some complexity class (e.g. agents that run in polynomial time). However, there are still simple programs that can have complex _true_ beliefs: we could make agents that believe that P != NP and that P = NP, and one of those agents will fall into this class. However, these agents are somewhat degenerate -- they don't have a good reason to _trust_ that their belief is true, and so we don't want to worry about them. Since we want to talk about _justified_ belief, let's consider a property based on which of the two agents _we_ would trust.

So, for now let's adopt an informal definition: a program A[C] is universal with respect to some class of programs C if we would trust any beliefs reported by A[C], no matter what beliefs we hear reported by programs in C. (I'll talk about the specific notion of "ascription universality" later.) Notably, we need to trust A[C] even over our own beliefs, that is, if A[C] believes something, we discard our position and adopt A[C]'s belief.

Note that this is still a very strong criterion. Consider the question of whether AI will lead to x-risk: if one agent thinks about fixed-point theorems, while another studies neural nets, and a third tries to predict the response of democratic institutions, the agent A[C] must compete with _everything_ that these agents (justifiably) conclude. For now, let's set aside the question of how to build a universal agent, and instead understand what it enables us to do.


This application of universality is the one I believe the most; even intuitively it seems like the entire informed oversight problem is that the overseer _doesn't_ know everything that the agent knows, and universality straightforwardly fixes that. So I see this as the motivating example for caring about universality at all.

Informed oversight (revisited)

Introduced in [this older post](https://ai-alignment.com/the-informed-oversight-problem-1b51b4f66b35), we consider a setting in which an overseer is training an agent, and the overseer is smarter than the agent. Intuitively, this should work out well, because if the agent ever tries to do something bad, the overseer should notice this (since it is smarter), and penalize it. However, the problem is that the agent might _in the course of computing an answer_ come up with considerations that the overseer can't easily find. For example, an agent tasked with creating novel literature might end up plagiarizing from one book; the overseer may not be able to search through all the books to detect the plagiarism. However, if the overseer is universal, then it should also know that the agent's output is plagiarized from the book (because the agent knows that), and it can penalize it appropriately.

We can even construct a semi-formal argument (see the post for the math). Let us assume that there is some true utility function U, and when the agent takes some action a, the overseer provides it with reward r = E[U(a)], where the expectation is relative to the overseer's beliefs. Let us assume away problems of distribution shift: in particular, we'll assume that when choosing between actions a1 and a2, the agent always chooses the action with higher reward. Now suppose that the agent chooses action a1. We can conclude that a1 has higher reward than a2. This means that the observer believes that a1 has higher true utility than a2. But by universality, we always trust the observer over the agent and ourselves, and so we must also believe that a1 has higher true utility than a2, which means that we believe the agent is choosing the right action.


I was skeptical of the <@previous iteration of this post@>(@Techniques for optimizing worst-case performance@), which proposed adversarial training, verification, and interpretability, because it seemed like none of them would allow us to get a _guarantee_ on performance. I'm significantly more optimistic after reading this post, mainly because a couple of previous confusions have been resolved:

1. The point of verification is not that we can prove a theorem saying "this agent is beneficial"; the point is that by making _relaxations_ (pseudo-inputs), a technique commonly used in formal verification, we can reduce the burden on the other methods being used (such as adversarial training).
2. Similarly, the point of interpretability is not to help _us_ understand what the agent is doing or will do, it's to help the _overseer_ (or adversary in adversarial training) understand that. Unlike us, the overseer / adversary can scale up along with the agent itself.

I still think that it would be hard to get a guarantee with adversarial training, given that adversarial training has to eliminate _all_ vulnerabilities. On the other hand, it only has to find all of the settings where the agent is _maliciously optimizing against us_, which you might hope is a more natural category that is easier to identify without looking too much at particular inputs. This seems like an empirical question on which we'll hopefully get data, though even if it works in all cases that we see, that doesn't rule out the possibility that we failed to notice some issue that will only be triggered in the future (as in the RSA-2048 example).

Worst-case guarantees (Revisited)

Let's start considering distribution shift again: we imagine that the agent has already been trained, and there is no overseer any more. When the agent goes off distribution, it may not choose the action that would have gotten the highest reward, and in fact it could be chosen maliciously. As a silly motivating example, perhaps the agent learned the policy "behave well until I see that RSA-2048 (a product of two very large primes) has been factored"; such a policy will behave perfectly during training but will probably commit a treacherous turn eventually.

In this setting, we would like to have a worst-case guarantee, that the model will never behave unacceptably. However, we need to enforce this guarantee at training time, _before_ we know what situations will arise at test time. Universality of the overseer isn't enough to guarantee this, because universality only means that the overseer knows what the agent _currently_ knows; the agent can't really be said to "know" that it will deceive us in the future.

Nonetheless, it seems like the solution is very related to universality, in that we would like to "peer inside" the agent to figure out how it works, to make sure there aren't situations in which it would behave unacceptably. We could use adversarial training for this purpose: train an adversary to find inputs on which the model behaves unacceptably, and then train the model on those inputs to not do that. This will require interpretability techniques, so that the adversary actually stands a chance of finding the relevant inputs. In order to handle cases like RSA-2048 above, the adversary should also be able to provide "pseudo-inputs" like "the world in which the agent sees that RSA-2048 is factored" (otherwise the adversary would have to factor RSA-2048 itself to demonstrate an input on which the agent would fail, which is far too difficult), while not being allowed to provide pseudo-inputs like "the world in which the agent kills all humans" (which no agent can behave acceptably on). Ultimately, we need to make sure the adversary will always be able to find a bad pseudo-input if one exists, without making it too difficult for the agent to behave acceptably on all pseudo-inputs. See also <@relaxed adversarial training@>(@Relaxed adversarial training for inner alignment@).


Universality and model-based RL

So far, we've been talking about the model-free setting, where the overseer provides the incentives. What about model-based RL? Here, we might want to learn separate distributions over models and utility functions using iterated amplification or HCH, and then plan using any off-the-shelf algorithm, such as MCTS. The first new problem that arises is that our distribution over models may be malicious (see [benign induction](https://agentfoundations.org/item?id=1263)). Using universality, we can solve this problem in two different ways. First, we could use HCH to filter out malicious models. By universality, since a malicious model knows what its goals are, so does HCH, and thus HCH will detect it. Second, we could ask HCH to give us a good prior over models. By universality, this prior over models must be better than the prior used by the malicious models, and so the malicious models' predictions will be worse than that of HCH as a whole, and so the malicious models will not be able to affect the computation much.

The second new problem in the model-based setting is that it isn't clear how our utility function can extract all the relevant information from the model. For example, if the model is a detailed simulation of physics, how do you find the "humans" and evaluate how "happy" they are? Universality is at least necessary for this: if there are facts the model "knows", those facts may be relevant to the utility calculation, and so the utility function also needs to know them. We might hope that universality is sufficient: in the previous case, we might think that the physics simulation "knows" whether humans are happy. But this depends on details of what "know" means that haven't yet been fleshed out, and it depends on what "kind" of universality we can achieve in practice.


I don't understand this post that well, so I'll refrain from opining on it. I think I would benefit from a more concrete example of how malicious optimization could be implemented "on top of" HCH, even when the humans are restricted to a small set of inputs on which they are "secure".

Universality and consequentialism within HCH

In all the settings discussed so far, we hope to use <@HCH@>(@Humans Consulting HCH@). What if HCH itself contains malicious optimization? Specifically, recall that HCH involves an arbitrarily large tree of humans breaking questions down into subquestions. Memetic selection on 10^100 humans should be _much_ stronger than the selection done by evolution to produce us, and so could be quite dangerous. We would like to use HCH to check whether its own computations are malicious.

So, we'll define a pair of mutually recursive functions, Filter and Filtered-HCH. Filter takes as input a transcript from HCH, and using filtered-HCH it finds the best argument suggesting that the transcript is problematic. It then uses filtered-HCH to ask whether, in light of this argument, the transcript should be treated as problematic. This check is repeated thrice for robustness. (Note that the original HCH treats the transcript as trusted computations, while the checker treats it as an untrusted input.) Filtered-HCH has the same type signature as HCH (i.e. it produces answers to questions). Just like HCH, it computes an answer to the question, with the help of recursive calls to filtered-HCH to get answers to subquestions. Instead of returning the answer directly, it calls Filter to check that the computation done was problematic, and returns the special symbol ⟂ instead of the answer if so.

The basic hope is that if there's some malicious subcomputation implemented "on top of" the base agents in HCH, and it tries to break HCH, then it "knows" why that attempt could work, and so if filtered-HCH is universal with respect to computations that can be instantiated by (smaller versions of) HCH, then filtered-HCH will also "know" that the computation could break HCH, and will censor it as a result. Consider the threat model from <@Reliability amplification@>, where the chance of any particular HCH computation with budget n becoming malicious is ε, independent of the input or any past HCH calls. Then, intuitively, filtered-HCH (which needs a higher budget, say kn) has a kε chance of becoming malicious, but since the check is repeated three times and a majority vote is taken, the chance of a malicious computation getting through is k^2ε^2, which will be much smaller than the original ε if k^2 ε << 1. (This is significantly better than the scheme in <@Reliability amplification@>, because the question "is there a problem?" is a binary yes/no question, and so majority vote works a lot better.) The harder case is when specific inputs cause very high probabilities of failure (which I believe is the threat model from <@Security amplification@>). I particularly don't understand this section, but I believe the hope is that since the malicious computation is implemented "on top of" the (aligned) humans, only a small "part" of the computation can be malicious, and this can be detected by a _small_ instance of Filtered-HCH on which the malicious computation cannot be implemented.


I was initially a bit confused by the decomposition into the idealized and the practical variant, but I think it makes more sense now: the assumption of _perfect_ optimization gives you a lot of mileage. For example, A[C] might choose a particular computation c in C, and then use its beliefs; if that _choice of computation_ were already "perfectly optimized" for convincing us, then we would always be at least as convinced of A[C]'s answer as the answer of any other c in C. Debate under optimal play is pretty similar: under optimal play, the agent only tells us about the most important considerations, and only brings up other considerations if its opponent asks about them; you would hope that this is like selecting the best c in C and reporting its reasons for its beliefs.

What's happening here is that the perfect optimization allows us to select the best c in C, or the best argument, etc. "for free" (i.e. without using any computation), giving us huge savings; everything else just has to present the results properly. Consider again the question of whether AI will lead to x-risk: while there are different agents thinking about fixed-point theorems, neural nets, and democratic institutions, perfect optimization lets us "pick" the agent whose arguments are most important. Then, we should only need about as much computation as the agent itself.

However, "presenting the results properly" is non-trivial: A[C] needs to report the beliefs _in plain English_, even if the computation c in C has its beliefs "encoded" (as in the case of a detailed physics simulation that can be said to "know" that humans are happy, but doesn't say that in plain English). It's not clear to me whether this will work out.

You might very reasonably be worried that perfect optimization of "the most convincing argument" will get you something that is convincing for the wrong reasons, and is not actually correlated with the truth. The definition of universality doesn't preclude this: it simply says that we trust A[C]'s beliefs over our own beliefs and any other agent's beliefs; this does hold for convincing but invalid arguments. I believe (but am not sure) that the hope is to deal with this by restricting inputs to the human to a small secure space, and by the techniques discussed above in [Universality and consequentialism within HCH](https://ai-alignment.com/universality-and-consequentialism-within-hch-c0bee00365bd).

Towards formalizing universality

We've now seen a bunch of applications of universality. So what is _ascription_ universality? It's a specific formalization of the concept of universality. Recall our informal definition so far: a program A[C] is universal with respect to some class of programs C if we would trust any beliefs reported by A[C], no matter what beliefs we hear reported by programs in C. But how can a program "report beliefs"? For HCH, to find out its belief for some question, we simply ask HCH the question and take its answer as its "belief". However, we can't do this for arbitrary computations: in particular, if our agent is deliberately hiding a belief from us (and so wouldn't report it if asked), we shouldn't simply say that it doesn't have that belief. This is where _ascriptions_ come in: an ascription is any way of ascribing beliefs to the agent. Simply asking the agent and trusting its answer is one possible ascription. Another class of ascriptions would look at the agent's code and memory, and make inferences about the agent's beliefs from that. We want to say that HCH will dominate _any_ "reasonable" ascription of beliefs to the agent, where an ascription procedure is "reasonable" if it isn't "doing the hard work"; for example, if an agent always says "Yes", it is unreasonable to ascribe to it the belief that Fermat's Last Theorem is true (even though when given the question "Is Fermat's Last Theorem true" it answers yes).

Putting it all together, A[C] is _ascription universal_ with respect to some class of programs C if, for every "reasonable" ascription procedure and program c in C, A[C]'s beliefs (which we find by asking it) _epistemically dominate_ the beliefs ascribed to c. A set of beliefs B1 _epistemically dominates_ another set of beliefs B2 _from our perspective_ if, given arbitrary information about B1 and B2, for every question our best guess at the answer is what we expect B1 would answer.

So far we've been talking about A[C] dominating an entire class of programs C. We might call this _idealized_ ascription universality, and hope that it applies to systems with perfect optimization, such as <@debate@>(@AI safety via debate@) under optimal play. In practice, we'll never get perfect optimization, and our overseer isn't going to be able to think of _everything_ that any agent could be thinking about. Instead, we could aim for _practical_ ascription universality, in which A[c] dominates a particular computation c, and it gets to look at the computation c as a hint. This would allow an overseer to train an agent, while epistemically dominating that agent. Effectively we're asking two questions: first, would our procedure be universal if the optimization was performed perfectly? Second, is weight sharing / interpretability strong enough to solve the optimization problems needed for universality?

The blog post also has some informal thoughts on why we should expect that a careful Bayesian should be able to epistemically dominate computations found via machine learning (which is why we might hope that HCH could be ascription universal).


Of all of these, I'm most worried about the second and third problems. I definitely have a weak intuition that there are many important tasks that we care about that can't easily be decomposed, but I'm optimistic that we can find out with experiments. For the point about having to train a by-default unaligned neural net to imitate aligned agents, I'm somewhat optimistic about informed oversight with strong interpretability techniques, but I become a lot less optimistic if we think that won't be enough and need to use other techniques like verification, which seem unlikely to scale that far. In any case, I'd recommend reading this post for a good explanation of common critiques of IDA.

Challenges to Christiano’s capability amplification proposal

A list of challenges faced by iterated distillation and amplification. First, a collection of aligned agents interacting does not necessarily lead to aligned behavior. (Paul's response: That's not the reason for optimism, it's more that there is no optimization pressure to be unaligned.) Second, it's unclear that even with high bandwidth oversight, that a collection of agents could reach arbitrary levels of capability. For example, how could agents with an understanding of arithmetic invent Hessian-free optimization? (Paul's response: This is an empirical disagreement, hopefully it can be resolved with experiments.) Third, while it is true that exact imitation of a human would avoid the issues of RL, it is harder to create exact imitation than to create superintelligence, and as soon as you have any imperfection in your imitation of a human, you very quickly get back the problems of RL. (Paul’s response: He's not aiming for exact imitation, he wants to deal with this problem by having a strong overseer aka informed oversight, and by having techniques that optimize worst-case performance.) Fourth, since Paul wants to use big unaligned neural nets to imitate humans, we have to worry about the possibility of adversarial behavior. He has suggested using large ensembles of agents and detecting and pruning the ones that are adversarial. However, this would require millions of samples per unaligned agent, which is prohibitively expensive. (Paul's response: He's no longer optimistic about ensembles and instead prefers the techniques in [this post](https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99), but he could see ways of reducing the sample complexity further.)


I haven't played the particular debate game that OpenAI released, and so it was interesting to see what sort of strategies emerged. It was initially quite unintuitive to me how debate picks out a particular path in an argument tree, and I think if reading about particular concrete examples (as in this post) would have helped.

Thoughts on AI Safety via Debate

Vaniver has played several debate games on the [website](https://debate-game.openai.com/) and wrote up some of his experiences. He ended up more optimistic about debate, but still worries that the success of the technique relies on the toy examples being toy.


It's great to see another approach that directly tackles the problem of defining a training signal that if optimized well would lead to an aligned AI. There are a lot of empirical questions that would influence whether or not debate actually works in practice, and I'm excited to see what experiments find.

AI safety via debate

At a high level, a major issue with building superintelligent AI is that humans would not be able to provide strong oversight for the AI. Amplification solves this by using the AI as a tool that can help the human (in particular, if the human can break a task down into subtasks, the AI can solve the subtasks). Debate also provides the AI as a tool for human overseer, but in a different way -- now, in order to train the AI, we have the AI debate against itself in order to convince a human of the answer to some target question. Given some question whose answer is too hard to directly judge, the human can look at the arguments and counterarguments to figure out whether or not the answer is actually correct.

The paper describes debate in a lot more depth and has an initial experiment involving MNIST. I can't possibly do it justice here -- I encourage you to simply read the full paper. You probably have an intuition right now of why this wouldn't work, such as "but humans believe what they want to hear, not what is true". The paper spends 5 (!) pages listing ten such problems and analyzing them, so go read it.


This is my new favorite post/paper for explaining how iterated amplification works, since it very succinctly and clearly makes the case for iterated amplification as a strategy for generating a good training signal. I'd recommend reading the [paper](https://arxiv.org/abs/1810.08575) in full, as it makes other important points that I haven't included in the summary.

Note that it does not explain a lot of Paul's thinking. It explains one particular training method that allows you to train an AI system with a more intelligent and informed overseer.

Learning Complex Goals with Iterated Amplification

This blog post and the accompanying [paper](https://arxiv.org/abs/1810.08575) introduces iterated amplification, focusing on how it can be used to define a training signal for tasks that humans cannot perform or evaluate, such as designing a transit system. The key insight is that humans are capable of decomposing even very difficult tasks into slightly simpler tasks. So, in theory, we could provide ground truth labels for an arbitrarily difficult task by a huge tree of humans, each decomposing their own subquestion and handing off new subquestions to other humans, until questions are easy enough that a human can directly answer them.

We can turn this into an efficient algorithm by having the human decompose the question only once, and using the current AI system to answer the generated subquestions. If the AI isn't able to answer the subquestions, then the human will get nonsense answers. However, as long as there are questions that the human + AI system can answer but the AI alone cannot answer, the AI can learn from the answers to those questions. To reduce the reliance on human data, another model is trained to predict the decomposition that the human performs. In addition, some tasks could refer to a large context (eg. evaluating safety for a specific rocket design), so they model the human as being able to access small pieces of the context at a time.

They evaluate on simple algorithmic tasks like distance between nodes in a graph, where they can program an automated human decomposition for faster experiments, and there is a ground truth solution. They compare against supervised learning, which trains a model on the ground truth answers to questions (which iterated amplification does not have access to), and find that they can match the performance of supervised learning with only slightly more training steps.


This is an excellent post that improved my understanding of Paul Christiano's agenda, which is not something I usually say about posts not written by Paul himself. I definitely have not captured all of the important ideas in my summary, so you should read it.

Understanding Iterated Distillation and Amplification: Claims and Oversight

The post introduces a distinction between flavors of iterated distillation and amplification -- whether the overseer is low bandwidth or high bandwidth. Let's think of IDA as building a deliberation tree out of some basic overseer. In the high bandwidth case, we can think of the overseer as a human who can think about a problem for 15 minutes, without access to the problem's context. However, there could be "attacks" on such overseers. In order to solve this problem, we can instead use low-bandwidth overseers, who only look at a sentence or two of text, and verify through testing that there are no attacks on such overseers. However, it seems much less clear that such an overseer would be able to reach high levels of capability.


This does seem like a challenging problem to address, and as the authors mention, it also affects iterated amplification. (Intuitively, if during iterated amplification the decomposition chosen happens to be one that ends up being obfuscated, then iterated amplification will get to the wrong answer.) I’m not really sure whether I expect this to be a problem in practice -- it feels like it could be, but it also feels like we should be able to address it using whatever techniques we use for robustness. But I generally feel very confused about this interaction and want to see more work on it.

Debate update: Obfuscated arguments problem

We’ve <@previously seen@>(@Writeup: Progress on AI Safety via Debate@) work on addressing potential problems with debate, including (but not limited to):

1. Evasiveness: By introducing structure to the debate, explicitly stating which claim is under consideration, we can prevent dishonest debaters from simply avoiding precision.
2. Misleading implications: To prevent the dishonest debater from “framing the debate” with misleading claims, debaters may also choose to argue about the meta-question “given the questions and answers provided in this round, which answer is better?”.
3. Truth is ambiguous: Rather than judging whether answers are _true_, which can be ambiguous and depend on definitions, we instead judge which answer is _better_.
4. Ambiguity: The dishonest debater can use an ambiguous concept, and then later choose which definition to work with depending on what the honest debater says. This can be solved with <@cross-examination@>(@Writeup: Progress on AI Safety via Debate@).

This post presents an open problem: the problem of _obfuscated arguments_. This happens when the dishonest debater presents a long, complex argument for an incorrect answer, where neither debater knows which of the series of steps is wrong. In this case, any given step is quite likely to be correct, and the honest debater can only say “I don’t know where the flaw is, but one of these arguments is incorrect”. Unfortunately, honest arguments are also often complex and long, to which a dishonest debater could also say the same thing. It’s not clear how you can distinguish between these two cases.

While this problem was known to be a potential theoretical issue with debate, the post provides several examples of this dynamic arising in practice in debates about physics problems, suggesting that this will be a problem we have to contend with.


Paul's research agenda FAQ

Exactly what it sounds like. I'm not going to summarize it because it's long and covers a lot of stuff, but I do recommend it.


It is unclear whether most tasks can actually be decomposed as required for iterated distillation and amplification, so I'm excited to see experiments that can answer that question! The questions that Ought is trying seem quite hard, so it should be a good test of breaking down reasoning. There's a lot of detail in the presentation that I haven't covered, I encourage you to read it.

Factored Cognition (old)

This is a presentation that Andreas has given a few times on Factored Cognition, a project by [Ought](https://ought.org/) that is empirically testing one approach to amplification on humans. It is inspired by [HCH](https://ai-alignment.com/strong-hch-bedb0dc08d4e) and [meta-execution](https://ai-alignment.com/meta-execution-27ba9b34d377). These approaches require us to break down complex tasks into small, bite-sized pieces that can be solved separately by copies of an agent. So far Ought has built a web app in which there are workspaces, nodes, pointers etc. that can allow humans to do local reasoning to answer a big global question.


This is a good point; it shows that you need your overseer to be ascription universal with respect to the entire training process, not just the final agent.

Nuances with ascription universality

Consider a formalization of honesty / transparency related to ascription universality: a system is honest, if relative to our expectations, its explicitly stated beliefs give us just as much information as any other way of ascribing beliefs to the system. A system that searches for ways to commit a treacherous turn would not be honest, since it would not state that it was performing this search. However, what if this search was "distilled" into the weights of a neural net (akin to a lookup table)? In that case, since we wouldn't ascribe beliefs to the lookup table, the system could be deemed honest by this criterion. So, what we actually need is for the entire training process to be honest. (A similar critique applies directly to ascription universality.)


This is distinct progress since the last update, though I think the experiments are still exploratory enough that it's hard to have any big takeaways. The difficulty of generating good strategies suggests that it's particularly important that we train our human overseers (as suggested in <@AI Safety Needs Social Scientists@>) to provide the right kind of feedback, for example if we would like them to reward only <@corrigible reasoning@>(@Corrigibility@). I'm particularly excited for the next update, where we could see experiments powerful enough to come to more solid conclusions.

Ought Progress Update October 2019

While this update provides details about Ought as a whole, I will focus only on the research they've done. As a reminder, Ought's goal is to understand how we can delegate hard questions to machine and human experts. They initially focused on <@Factored Cognition@>, where each question was recursively decomposed into subquestions that would help _find_ the answer. They now call this "Factored Generation", and use "Factored Cognition" as the umbrella term for any approach that helps answer hard questions using some kind of decomposition.

While Ought has run some ML experiments aimed at automating decompositions, they have primarily worked on human experiments for <@Factored Evaluation@>(@Delegating open-ended cognitive work@), where the decompositions are used to help _evaluate_ answers. They are focusing on evaluating answers instead of finding answers for a few reasons. First, there may already exist experts whose answers we can use: this can be directly plugged into Factored Evaluation, but not Factored Generation. Second, evaluation is often easier than generation. Third, factored evaluation is more amenable to time-saving tricks like only focusing on a single branch of the tree (as in <@AI safety via debate@>). Finally, when training ML systems, if a model is unable to exactly imitate the experts, we need something that can distinguish between better and worse behavior, which is possible with Factored Evaluation but not Factored Generation. (See also [Against Mimicry](https://ai-alignment.com/against-mimicry-6002a472fc42).)

The experiments involve a judge trying to decide which of two answers to a question is best, where one answer is provided by an honest expert and one by a malicious expert. (Subquestions may also get different answers from the two experts.) The goal is to find a setup where the judge reliably picks the honest answer, regardless of how powerful the malicious expert is: if this could be established, it would be strong evidence that the setup incentivizes honest answers. Of course, this requires us to quantify over all malicious strategies, but humans may not find the best malicious strategies. To mitigate this, the experiments provide the malicious expert with lots of advantages that the honest one doesn't have, to simulate very powerful malicious experts.

They have already learned from their experiments. Initially, they hoped participants would develop good meta-level strategies for determining the truth. (Meta-level here means that the strategies would generalize to other domains, e.g. a heuristic of always splitting an answer into separate claims and asking for the evidence for each claim separately.) They found that these strategies _don't_ emerge organically, and so are planning to spend concentrated staff time on finding good strategies. They also found that malicious experts sometimes won due to avoidable mistakes, and are hoping to eliminate this by ensembling work from multiple people for increased robustness.


Figuring out an interesting and useful project to work on is one of the major challenges of any research project, and it may require a distinct skill set from the project's implementation. As a result, I appreciate the authors enabling other researchers to jump straight into solving the problems. Given how detailed the motivation, instructions, and related work are, this document strikes me as an excellent way for someone to begin her first research project on IDA or AI safety more broadly. Additionally, while there are [many](https://arxiv.org/abs/1810.08575) [public](https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616) [explanations](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd) of IDA, I found this to be one of the most clear and complete descriptions I have read.

Machine Learning Projects on IDA

Owain Evans, William Saunders, Andreas Stuhlmüller

This document describes three suggested projects building on Iterated Distillation and Amplification (IDA), a method for training ML systems while preserving alignment. The first project is to apply IDA to solving mathematical problems. The second is to apply IDA to neural program interpretation, the problem of replicating the internal behavior of other programs as well as their outputs. The third is to experiment with adaptive computation where computational power is directed to where it is most useful. For each project, they also include motivation, directions, and related work.


List of resolved confusions about IDA

This is a useful post clarifying some of the terms around IDA. I'm not summarizing it because each point is already quite short.


Amusingly, I liked this post primarily because comparing this setting to the typical setting for iterated amplification was useful for seeing the design choices and intuitions that motivated iterated amplification.

Aligning a toy model of optimization

Current ML capabilities are centered around **local search**: we get a gradient (or an approximation to one, as with evolutionary algorithms), and take a step in that direction to find a new model. Iterated amplification takes advantage of this fact: rather than a sequence of gradient steps on a fixed reward, we can do a sequence of amplification steps and distillation gradient steps.

However, we can consider an even simpler model of ML capabilities: **function maximization**. Given a function from n-bit strings to real numbers, we model ML as allowing us to find the input n-bit string with the maximum output value, **in only O(n) time** (rather than the O(2^n) time that brute force search would take). If this were all we knew about ML capabilities, could we still design an aligned, competitive version of it? While this is not the actual problem we face, **due to its simplicity it is more amenable to theoretical analysis**, and so is worth thinking about.

We could make an unaligned AI that maximizes some explicit reward using only 2 calls to Opt: first, use Opt to find a good world model M that can predict the dynamics and reward, and then use Opt to find a policy that does well when interacting with M. This is unaligned for all the usual reasons: most obviously, it will try to seize control of the reward channel.

An aligned version does need to use Opt, since **that's the only way of turning a naively-exponential search into a linear one**; without using Opt the resulting system won't be competitive. We can't just generalize iterated amplification to this case, since iterated amplification relies on a _sequence_ of applications of ML capabilities: this would lead to an aligned AI that uses Opt many times, which will not be competitive since the unaligned AI only requires 2 calls to Opt.

One possible approach is to design an AI with good incentives (in the same way that iterated amplification aims to approximate <@HCH@>(@Humans Consulting HCH@)) that "knows everything that the unaligned AI knows". However, it would also be useful to produce a proof of impossibility: this would tell us something about what a solution must look like in more complex settings.


I think one of the key questions underlying Debate is how efficiently natural language can summarise reasoning about properties of the world. This question is subject to some disagreement (at one extreme, Facebook's [roadmap towards machine intelligence](https://research.fb.com/publications/a-roadmap-towards-machine-intelligence/) describes a training environment which is "entirely linguistically defined") and probably deserves more public discussion in the context of safety.

**Rohin's note:** If you've read the previous posts on debate, the novel parts of this podcast are on the relation between iterated amplification and debate (which has been discussed before, but not in as much depth), and the reasons for optimism and pessimism about debate.

AI Alignment Podcast: AI Alignment through Debate

We want AI safety solutions to scale to very intelligent agents; debate is one scalability technique. It's formulated as a two player zero-sum perfect information game in which agents make arguments in natural language, to be evaluated by a human judge. Whether or not such debates are truth-conducive is an empirical question which we can try to evaluate experimentally; doing so will require both technical and social science expertise (as discussed in a <@previous post@>(@AI Safety Needs Social Scientists@)).


Rissanen Data Analysis: Examining Dataset Characteristics via Description Length

We are often interested in estimating how useful a particular capability might be for a model. For example, for <@Factored Cognition@> we're interested in how useful the "decomposition" ability is, that is, how useful it is to decompose the original question into subquestions (as in <@this paper@>(@Unsupervised Question Decomposition for Question Answering@)). This paper proposes a simple methodology: give the model oracle access to the capability in question, and see how much it improves its predictions. This is measured in an online learning setup (rather than in one fell swoop at the end of training), in order to evaluate how useful the capability is in both low and high data regimes.

(The paper frames this as asking how much better you can compress the labels when you have access to the capability, relative to not having the capability. This can be seen as an upper bound on the minimum description length, which in turn is one way of operationalizing Occam's razor. I find the prediction view more intuitive, and as far as I can tell the two views are equivalent in the context of this paper.)

They then use this framework to investigate a bunch of empirical questions:
1. For question answering models trained from scratch, both ML decompositions and human decompositions are helpful, though ML still has a long way to go to catch up to human decompositions.
2. One way to evaluate gender bias in a dataset is to ask, "how useful is the "capability" of seeing the male-gendered words", relative to the same question for female-gendered words. This confirms the general male-gendered bias, even in a dataset that has more female-gendered words.
3. Some papers have claimed that neural nets are effectively "bag-of-words" models, i.e. they don't pay attention to the ordering of words in a sentence. They evaluate how useful the capability of "getting the correct order" is, and find that it does lead to significantly better results.


AXRP Episode 6 - Debate and Imitative Generalization

This podcast covers a bunch of topics, such as <@debate@>(@AI safety via debate@), <@cross examination@>(@Writeup: Progress on AI Safety via Debate@), <@HCH@>(@Humans Consulting HCH@), <@iterated amplification@>(@Supervising strong learners by amplifying weak experts@), and <@imitative generalization@>(@Imitative Generalisation (AKA 'Learning the Prior')@) (aka [learning the prior](https://www.alignmentforum.org/posts/SL9mKhgdmDKXmxwE4/learning-the-prior) ([AN #109](https://mailchi.mp/ee62c1c9e331/an-109teaching-neural-nets-to-generalize-the-way-humans-would))), along with themes about <@universality@>(@Towards formalizing universality@). Recommended for getting a broad overview of this particular area of AI alignment.


Universality Unwrapped

This post explains the ideas behind universality and ascription universality, in a more accessible way than the [original posts](https://ai-alignment.com/towards-formalizing-universality-409ab893a456) and with more detail than [my summary](https://mailchi.mp/6078fe4f9928/an-81-universality-as-a-potential-solution-to-conceptual-difficulties-in-intent-alignment).


How should AI debate be judged?

<@Debate@>(@AI safety via debate@) requires a human judge to decide which of two AI debaters should win the debate. How should the judge make this decision? The discussion on this page delves into this question in some depth.


Probably? If you are familiar with Paul's approach, you may not find anything new here.

An unaligned benchmark

We could get a very powerful AI system using model-based RL with MCTS (i.e. the technique underlying AlphaGo), where the reward function is learned from human feedback, but such a system would likely be incorrigible. In particular, it would be optimizing for making humans think that good outcomes are happening, which may or may not happen by actually having good outcomes. In addition, as the AI system becomes more intelligent, the environment distribution will change as the AI exerts more control over it, leading to a distributional shift that could also lead to bad outcomes. If we instead try to emulate this sort of system using iterated distillation and amplification, the complicated plans made by the AI ultimately cash out to (a very large amount of) human judgment and deliberation, which should not fall prey to these problems.


How does iterated amplification exceed human abilities?