This seems like a good way to generate good concrete empirical projects to work on. It does differ from the original post in placing less of an emphasis on “fuzzy” tasks, where aligned metrics are hard to come by, though it isn’t incompatible with it (in a “fuzzy” task, you probably still want as aligned a metric as you can get in order to measure progress).

Some criteria for sandwiching projects

This post outlines the pieces needed in order to execute a “sandwiching” project on <@aligning narrowly superhuman models@>(@The case for aligning narrowly superhuman models@), with the example of answering questions about a text when humans have limited access to that text. (Imagine answering questions about a paper, where the model can read the full paper but human labelers can only read the abstract.) The required pieces are:

1. **Aligned metric:** There needs to be some way of telling whether the project succeeded, i.e. the technique made the narrowly superhuman model more aligned. In the Q&A case, we get the aligned metric by seeing how humans answer when they can read the entire text.
2. **A narrowly superhuman model:** The model must have the capability to outperform the labelers on the task. In the Q&A case, we get this by artificially restricting the input that the labelers get (relative to what the model gets). In other cases we could use labelers who lack the relevant domain expertise that the model instead knows.
3. **Headroom on the aligned metric:** Baseline methods (such as training from labeler feedback) should not perform very well, so that there is room for a better technique to improve performance. It would be especially nice if making the model larger led to no improvement in the aligned metric; this would mean that we are working in a situation that is primarily an alignment failure.
4. **A natural plan of attack:** We have some approach for doing better than the baseline. For the Q&A example, we could train one model that selects the most relevant piece of text (by training on labelers’ ratings of relevance) and another model that answers the question given that relevant piece.


Adaptive Mechanism Design: Learning to Promote Cooperation

Mechanism design for AI

One cause of outcomes worse than extinction could be escalating conflicts between very capable AI systems (that could eg. threaten to simulate suffering beings). It is worth studying how we could have AI systems implement mechanism design in order to guide such systems into more cooperative behavior.


High Impact Careers in Formal Verification: Artificial Intelligence

This post considers the applicability of formal verification techniques to AI alignment. Now in order to “verify” a property, you need a specification of that property against which to verify. The author considers three possibilities:

1. **Formally specifiable safety:** we can write down a specification for safe AI, _and_ we’ll be able to find a computational description or implementation
2. **Informally specifiable safety:** we can write down a specification for safe AI mathematically or philosophically, but we will not be able to produce a computational version
3. **Nonspecifiable safety:** we will never write down a specification for safe AI.

Formal verification techniques are applicable only to the first case. Unfortunately, it seems that no one expects the first case to hold in practice: even CHAI, with its mission of building provably beneficial AI systems, is talking about proofs in the informal specification case (which still includes math), on the basis of comments like [these](https://www.alignmentforum.org/posts/nd692YfFGfZDh9Mwz/an-69-stuart-russell-s-new-book-on-why-we-need-to-replace?commentId=4LhBaSuYPyFvTnDrQ) in Human Compatible. In addition, it currently seems particularly hard for experts in formal verification to impact actual practice, and there doesn’t seem to be much reason to expect that to change. As a result, the author is relatively pessimistic about formal verification as a route to reducing existential risk from failures of AI alignment.


I like it -- I think I will send this to newer researchers as a precursor to the full paper.

A Summary of Concrete Problems in AI Safety

A nice summary of [Concrete Problems in AI Safety](https://blog.openai.com/concrete-ai-safety-problems/) that's a lot quicker to read than the original paper.


I would be really excited to see good work on this agenda, it would be a big step forward on how good our design process for neural nets is.

Mechanistic Transparency for Machine Learning

One useful thread of alignment research would be to figure out how to take a neural net, and distill parts or all of it into pseudocode or actual code that describes how the neural net actually works. This could then be read and analyzed by developers to make sure the neural net is doing the right thing. Key quote: "I'm excited about this agenda because I see it as giving the developers of AI systems tools to detect and correct properties of their AI systems that they see as undesirable, without having to deploy the system in a test environment that they must laboriously ensure is adequately sandboxed."


Alignment of Language Agents

Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, Geoffrey Irving

This paper analyzes the various problems we consider in AI alignment from the perspective of language agents. Problems covered include <@specification gaming@>(@Specification gaming examples in AI@), <@whom and what to align to@>(@Artificial Intelligence, Values and Alignment@), <@intent alignment@>(@Clarifying "AI Alignment"@), <@removing tampering incentives@>(@Avoiding Tampering Incentives in Deep RL via Decoupled Approval@), and <@inner alignment@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@). These can be categorized as different kinds of misspecification, namely misspecification in the _training data_, the _training process_, and the _behavior under distributional shift_.

While the conceptual problems are similar to the ones already considered for embodied RL agents, the ways they manifest are different. In particular, the authors highlight the possibility that language agents will _deceive us_, _manipulate us_, or _produce harmful content_. The authors review some existing definitions of deception and manipulation that are purely behavioral (that is, the definitions do not require an _intent_ to deceive or manipulate). A signaller **deceives** a receiver if the signaller transmits (or suggestively doesn’t transmit) a signal that causes the receiver to believe some false claim that benefits the signaller. **Manipulation** is similar, except rather than causing the receiver to believe a false claim, it causes the receiver to take some action that benefits the signaller, that in some sense the receiver “shouldn’t” have taken. We could cash out “the receiver ‘shouldn’t’ have taken the action” just as "the action is harmful to the receiver", but from a safety / security mindset, the authors prefer a broader definition that aims to identify bad _means_ of influencing the receiver, instead of only focusing on whether the _ends_ were bad.

Some other miscellaneous points:
- Since the “action space” is just language, it seems like it should be easier (though still requires work) to prevent language agents from causing physical harm.
- It will hopefully be easier to train language agents to be explainable, since they have native fluency in natural language with which they can explain their behavior.


If you have strongly suffering-focused ethics (unlike me), this seems mostly right. The post claims that suffering-focused AI safety should be more tractable than AI alignment, because it focuses on a subset of risks and only tries to minimize them. However, it's not necessarily the case that focusing on a simpler problem makes it easier to solve. It feels easier to me to figure out how to align an AI system to humans, or how to enable human control of an AI system, than to figure out all the ways in which vast suffering could happen, and solve each one individually. You can make an analogy to mathematical proofs and algorithms -- often, you want to try to prove a _stronger_ statement than the one you are looking at, because when you use induction or recursion, you can rely on a stronger inductive hypothesis.

An introduction to worst-case AI safety

Argues that people with suffering-focused ethics should focus on "worst-case AI safety", which aims to find technical solutions to risks of AIs creating vast amounts of suffering (which would be much worse than extinction).


This assumes an ontology where there exists a utility function that an AI is optimizing, and changes to the AI seem especially likely to change the utility function in a random direction. In such a scenario, yes, you probably should be worried. However, in practice, I expect that powerful AI systems will not look like they are explicitly maximizing some utility function. If you change some component of the system for the worse, you are likely to degrade its performance, but not likely to drastically change its behavior to cause human extinction. For example, even in RL (which is the closest thing to expected utility maximization), you can have serious bugs and still do relatively well on the objective. A public example of this is in OpenAI Five (https://blog.openai.com/openai-five/), but I also hear this expressed when talking to RL researchers (and see this myself). While you still want to be very careful with self-modification, it seems generally fine not to have a formal proof before making the change, and evaluating the change after it has taken place. (This would fail dramatically if the change drastically changed behavior, but if it only degrades performance, I expect the AI would still be competent enough to notice and undo the change.) It may be the case that adversarial subprocesses could take advantage of these sorts of bugs, but I expect that we need adversarial-subprocess-specific research to address this, not zero-shot reasoning.

Another take on agent foundations: formalizing zero-shot reasoning

There are strong incentives to build a recursively self-improving AI, and in order to do this without value drift, the AI needs to be able to reason effectively about the nature of changes it makes to itself. In such scenarios, it is insufficient to "reason with extreme caution", where you think really hard about the proposed change, and implement it if you can't find reasons not to do it. Instead, you need to do something like "zero-shot reasoning", where you prove under some reasonable assumptions that the proposed change is good. This sort of reasoning must be very powerful, enabling the AI to eg. build a spacecraft that lands on Mars, after observing Earth for one day. This motivates many of the problems in MIRI's agenda, such as Vingean reflection (self-trust), logical uncertainty (how to handle being a bounded reasoner), counterfactuals, etc., which all help to formalize zero-shot reasoning.


I think I agree with this post? Certainly for a superintelligence that is vastly smarter than humans, I buy this argument (and in general am not optimistic about solving alignment). However, humans seem to be fairly good at keeping each other in check, without a deep understanding of what makes humans tick, even though humans often do optimize against each other. Perhaps we can maintain this situation inductively as our AI systems get more powerful, without requiring a deep understanding of what's going on? Overall I'm pretty confused on this point.

Optimization Amplifies

One model of the difference between mathematicians and scientists is that a scientist is good at distinguishing between 0.01%, 50% and 99.99%, whereas a mathematician is good at distinguishing between 99.99% and 100%. Certainly it seems like if we can get 99.99% confidence that an AI system is aligned, we should count that as a huge win, and not hope for more (since the remaining 0.01% is extremely hard to get), so why do we need mathematicians? Scott argues that optimization is particularly special, in that the point of very strong optimization is to hit a very narrow target, which severely affects extreme probabilities, moving them from 0.01% to near-100%. For example, if you draw a million samples from a normal distribution and optimize for the largest one, it is almost certain to be 4 standard deviations above the mean (which is incredibly unlikely for a randomly chosen sample). In this sort of setting, the deep understanding of a problem that you get from a mathematician is still important. Note that Scott is _not_ saying that we don't need scientists, nor that we should aim for 100% certainty that an AI is aligned.


One possible conclusion is that safety research should be more diversified across different paradigms and artefacts, in order to properly maximize expected safety. However, this isn’t obvious: it seems likely that if the dominant paradigm has 50% of the research, it will also have, say, 80% of future real-world deployments, and so it could make sense to have 80% of the safety research focused on it. Rather than try to predict which paradigm will become dominant (a very difficult task), it may be more efficient to simply observe which paradigm becomes dominant and then redirect resources at that time (even though that process takes 5 years to happen).

AI Paradigms and AI Safety: Mapping Artefacts and Techniques to Safety Issues

Jose Hernandez-Orallo, Fernando Martinez-Plumed, Shahar Avin, Jess Whittlestone, Seán Ó hÉigeartaigh

What should prioritization within the field of AI safety look like? Ideally, we would proactively look for potential issues that could arise with many potential AI technologies, making sure to cover the full space of possibilities rather than focusing on a single area. What does prioritization look like in practice? This paper investigates, and finds that it is pretty different from this ideal.

In particular, they define a set of 14 categories of AI _techniques_ (examples include neural nets, planning and scheduling, and combinatorial optimization), and a set of 10 kinds of AI _artefacts_ (examples include agents, providers, dialoguers, and swarms). They then analyze trends in the amount of attention paid to each technique or artefact, both for AI safety and AI in general. Note that they construe AI safety very broadly by including anything that addresses potential real-world problems with AI systems.

While there are a lot of interesting trends, the main conclusion is that there is an approximately 5-year delay between the emergence of an AI paradigm and safety research into that paradigm. In addition, safety research tends to neglect non-dominant paradigms.


This is eminently sensible to me and I agree with the takeaways. See also [Methodology of Unbounded Analysis](https://arbital.com/p/unbounded_analysis/).

Infinite Data/Compute Arguments in Alignment

This reference post makes a short argument for why we might consider hypotheticals in which we have infinite data and compute. The core idea is that this allows us to focus on _hard subproblems_. Compute and data capacity have been growing substantially, and so it makes sense to treat them as “cheap”; the hard subproblems are then the ones that remain when we assume unlimited compute and data.

In particular, in this case we can get perfect predictive power, using Bayesian updates on low-level physics models, or Solomonoff induction. Indeed, most of ML tends to be about figuring out how to turn the problem of interest into a prediction or optimization problem, after which we use off-the-shelf algorithms. So the hard subproblems are the ones that arise even when you can use Bayesian updates on low-level physics models.


I was expecting a list of questions that were specific to AI somehow, but the actual list is very broad and feels to me like a list of subquestions of "how do you design a good society in the face of technological development". For example, it includes unemployment, system maintenance, the potential of blackmail, side-channel attacks, etc.

AI Services as a Research Paradigm

The <@CAIS report@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@) suggests that future technological development will be driven by _systems of AI services_, rather than a single monolithic AGI agent. However, there has not been much followup research since the publication of the report. This document posits that this is because the concepts of tasks and services introduced in the report are not amenable to formalization, and so it is hard to do research with them. So, it provides a simple abstract model of a system of services, in which each service takes some inputs, produces some outputs, and is trained with some loss function, and these services are interconnected with each other. (Some of the services could be provided by humans instead of AI systems.) It then lists several research problems that could be tackled under this generic view.


I fall more on the side of preferring indirect approaches, though by that I mean that we should delegate to future humans, as opposed to defining some particular value-finding mechanism into an AI system that eventually produces a definition of values.

The Value Definition Problem

This post considers the Value Definition Problem: what should we make our AI system <@try to do@>(@Clarifying "AI Alignment"@) to have the best chance of a positive outcome? It argues that an answer to the problem should be judged based on how much easier it makes alignment, how competent the AI system has to be to optimize it, and how good the outcome would be if it was optimized. Solutions also differ on how "direct" they are -- on one end, explicitly writing down a utility function would be very direct, while on the other, something like [Coherent Extrapolated Volition](https://intelligence.org/files/CEV.pdf) would be very indirect: it delegates the task of figuring out what is good to the AI system itself.


Drexler on AI Risk

This is another analysis of [Comprehensive AI Services](https://www.fhi.ox.ac.uk/reframing/). You can read [my summary of CAIS](https://www.alignmentforum.org/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services) ([AN #40](https://mailchi.mp/b649f32b07da/alignment-newsletter-40)) to get my views.


I don't know how hard it would be to create integrative biological simulations, but it does strike me as very useful if we did have them. If we had a complete mechanistic understanding of how intelligence happens in biological brains (in the sense that we can simulate them), the obvious next step would be to understand _how_ the mechanistic procedures lead to intelligence (in the same way that we currently try to understand why neural nets work). If we succeed at this, I would expect to get several insights into intelligence that would translate into significant progress in AI. However, I know very little about biological neurons and brains so take this with many grains of salt.

On the value learning side, it would be a good test of inverse reinforcement learning to see how well it could work on simple organisms, though it's not obvious what the ground truth is. I do want to note that this is specific to inverse reinforcement learning -- other techniques depend on uniquely human characteristics, like the ability to answer questions posed by the AI system.

Integrative Biological Simulation, Neuropsychology, and AI Safety

Gopal P. Sarma, Adam Safron, Nick J. Hay

This paper argues that we can make progress on AI capabilities and AI safety through integrative biological simulation, that is, a composite simulation of all of the processes involved in neurons that allow us to simulate brains. In the near future, such simulations would be limited to simple organisms like Drosophila, but even these organisms exhibit behavior that we find hard to replicate today using our AI techniques, especially at the sample efficiency that the organisms show. On the safety side, even such small brains share many architectural features with human brains, and so we might hope that we could discover neuroscience-based methods for value learning that generalize well to humans. Another possibility would be to create test suites (as in [AI Safety Gridworlds](https://arxiv.org/abs/1711.09883)) for simulated organisms.


I agree with the general point that figuring out a human utility function and then optimizing it is unlikely to work, but for different reasons (see the first chapter of the [Value Learning sequence](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc)). I also agree that humans are complex and you can’t get away with modeling them as Boltzmann rational and optimizing some fixed utility function. I wouldn’t try to make the model more accurate (eg. a model of a bunch of interacting subagents, each with their own utility function), I would try to make the model less precise (eg. a single giant neural net), because that reduces the chance of model misspecification. However, given the [impossibility result](https://arxiv.org/abs/1712.05812) saying that you must make assumptions to make this work, we probably have to give up on having some nice formally specified meaning of “values”. I think this is probably fine -- for example, iterated amplification doesn’t have any explicit formal value function.

Multi-agent minds and AI alignment

This post argues against the model of humans as optimizing some particular utility function, instead favoring a model based on predictive processing. This leads to several issues with the way standard value learning approaches like inverse reinforcement learning work. There are a few suggested areas for future research. First, we could understand how hierarchical models of the world work (presumably for better value learning). Second, we could try to invert game theory to learn objectives in multiagent settings. Third, we could learn preferences in multiagent settings, which might allow us to better infer norms that humans follow. Fourth, we could see what happens if we take a system of agents, infer a utility function, and then optimize it -- perhaps one of the agents' utility functions dominates? Finally, we can see what happens when we take a system of agents and give it more computation, to see how different parts scale. On the non-technical side, we can try to figure out how to get humans to be more self-aligned (i.e. there aren't "different parts pulling in different directions").


Evaluating Existing Approaches to AGI Alignment


Phenomenological AI Safety Research Institute

Robustness to fundamental uncertainty in AGI alignment


ICML Uncertainty and Robustness Workshop Accepted Papers

The Uncertainty and Robustness Workshop accepted papers are available. Topics include out-of-distribution detection, generalization to stochastic corruptions, label corruption robustness, and so on.


Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings

Aviral Kumar, Sunita Sarawagi and Ujjwal Jain


Section 4.3 of [this](https://openreview.net/pdf?id=HyxCxhRcY7) paper makes similar observations and ameliorates the issue. [This](https://arxiv.org/abs/1809.04729v1) paper also demonstrates the fragility of density estimators on out-of-distribution data.

Do Deep Generative Models Know What They Don't Know?

Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan


Previous calibration work in machine learning conferences would often to focus on calibrating regression models, but this work has renewed interest in calibrating classifiers. For that reason I view this paper highly. That said, this paper’s evaluation measure, the “Expected Calibration Error” is not a proper scoring rule, so optimizing this does not necessarily lead to calibration. In their approximation of the ECE, they use equally-wide bins when there is reason to use adaptively sized bins. Consequently I think [Nguyen and O’Connor](https://arxiv.org/pdf/1508.05154.pdf) Sections 2 and 3 provide a better calibration explanation, better calibration measure, and better estimation procedure. They also suggest using a convex optimization library to find the softmax temperature, but at least libraries such as CVXPY require far more time and memory than a simple softmax temperature grid search. Finally, an understandable limitation of this work is that it assumes test-time inputs are in-distribution, but when inputs are out-of-distribution this method hardly improves calibration.

On Calibration of Modern Neural Networks

Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger

Models should not be unduly confident, especially when said confidence is used for decision making or downstream tasks. This work provides a simple method to make models more calibrated so that the confidence estimates are closer to the true correctness likelihood. (For example, if a calibrated model predicts “toucan” with 60% confidence, then 60% of the time the input was actually a toucan.) Before presenting their method, they observe that batch normalization can make models less calibrated, while unusually large weight decay regularization can increase calibration. However, their proposed approach to increase calibration does not impact accuracy or require substantive model changes. They simply adjust the temperature of the softmax to make the model’s “confidence” (here the maximum softmax probability) more calibrated. Specifically, after training they tune the softmax temperature to minimize the cross entropy (negative average log-likelihood) on validation data. They then measure model calibration with a measure which is related to the Brier score, but with absolute values rather than squares.


I like this paper: the intuition makes sense, the math is straightforward, and the empirical results are strong, and have continued to be strong when looking at later work that builds on it.

Representation Learning with Contrastive Predictive Coding

Aaron van den Oord, Yazhe Li, Oriol Vinyals

This paper from 2018 proposed Contrastive Predictive Coding (CPC): a method of unsupervised learning that has been quite successful. At its core it is quite simple: it simply combines the ideas of predictive coding and contrastive losses, both of which have been significantly studied in the past.

The simplest form of unsupervised learning would be data compression via generative models (as in e.g. VAEs), in which, to model the data **p(x)**, you attempt to encode **x** into a latent (hidden) state **z** in such a way that you can then recover the original data point **x** from **z**. Intuitively, we want **z** to have high mutual information with **x**.

For sequential data in a partially observed setting, you need to deal with the full sequence. Consider natural language: in this setting, each x would be a single word. Consider the sentence "I sat on the chair". If the **z** corresponding to the word "the" only has to reconstruct the word "the", it's not going to "remember" that the past context involved sitting, and so that **z** would be terrible at predicting that the next word will be chair. To fix this, we can use predictive coding, where we instead require that we can _predict_ future words using **z**. This now incentivizes **z_t** to have high mutual information with **x_{t+k}**.

There is still a problem: reconstructing the entire input **x** would require a lot of irrelevant information, such as e.g. the background color of the environment in RL, even if that never changes. How can we get rid of these irrelevant features? Contrastive losses allow us to do this: intuitively, since the irrelevant features are the ones that are common across all the **x**s (and so are fully captured by **p(x)** ), if we train the neural net to _distinguish_ between various **x**s, we can incentivize only the relevant features. In particular, given a latent state **z_t**, we take the true **x_{t+k}**, and throw in a bunch of other **x**s sampled from **p(x)** (known as _negative samples_), and train the network to correctly classify **x_{t+k}**. The authors show that the optimum of this loss function is indeed for the neural net to compute **p(x | z) / p(x)**, which implies that it is maximizing a lower bound on the mutual information between X and Z.

This gives us a pretty simple overall algorithm. Take a sequence **x_1 ... x_T**, compute **z_t** using a recurrent model on **x_1 ... x_t**, put **x_{t+k}** and some negative samples into a set, and train a classifier to correctly predict which of the samples is the true **x_{t+k}**. In practice, we do batches of these at the same time, and for every data point in the batch we use all of the other data points as our negative examples. The features you learn are then the ones that help _distinguish_ between **x_{t+k}** and the negative samples, and you'll ignore any features that are common across all the samples. This means that the results depend quite a lot on how you choose your samples (this effectively determines what **p(x)** you are using).

The authors evaluate their algorithm on several domains and show that it achieves or surpasses state of the art on them.


On Variational Bounds of Mutual Information

Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, George Tucker

This paper is a pretty dense and technical explanation of various ways in which we can estimate and/or optimize the mutual information between two variables. I specifically want to highlight that it provides a proof that the Contrastive Predictive Coding objective (summarized above) is a lower bound on the mutual information between the input and the representation, and compares it to other lower bounds on mutual information.


I really like this paper; I think it shines a light in an interesting new direction, and I look forward to seeing future work that builds on this in theoretical, mechanistic, and applied manners. On the other hand, I felt they did gloss over how exactly they do curriculum learning, and their reinforcement learning experiment was a little unclear to me.

I think the implications of this work are enormous. In a future where we might be limited by the maturity of available simulation platforms or inundated by deluges of data with little marginal information, this approach can circumvent such problems for the selection and (pre)training of suitable student networks.

Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, Jeff Clune

The Generative Teaching Networks (GTN) paper breaks new ground by training generators that produce synthetic data that can enable learner neural networks to learn faster than when training on real data. The process is as follows: The generator produces synthetic training data by transforming some sampled noise vector and label; a newly-initialized learner is trained on this synthetic data and evaluated on real data; the error signal from this evaluation is backpropagated to the generator via meta-gradients, to enable it to produce synthetic samples that will train the learner networks better. They also demonstrate that their curriculum learning variant, where the input vectors and their order are learned along with generator parameters, is especially powerful at teaching learners with few samples and few steps of gradient descent.

They apply their system to neural architecture search, and show an empirical correlation between performance of a learner on synthetic data and its eventual performance when trained on real data. In this manner, they make the argument that data from a trained GTN can be used to cheaply assess the likelihood of a given network succeeding to learn on the real task, and hence GTN data can tremendously speed up architecture search.


I really liked this paper; it was cleanly written, conceptually straightforward, and did a generally useful scientific service of checking whether an advance in one area might change our beliefs about a previous result. I particularly enjoyed looking at the "reconstructed" images they got by running their encoder and then generator: more so than anything I recall seeing from a VAE pixel-based reconstructor, this model seems to be treating images as valid reconstructions of one another if they're of the same class (i.e. two pizzas) even if the colors and low level detail are different. This makes reasonable sense if you think that those two pizzas are probably nearby in latent space, and so each is a plausible reconstruction of each other's latent space encoding, but it's still cool to see concretely borne out.

Large Scale Adversarial Representation Learning

The BigGAN paper, published last September, used a much larger model (and a handful of optimization tricks to facilitate training it) to achieve a huge leap forward in the quality of generated images. However, it was unclear from the earlier paper whether this improvement in generation quality would also be tied to an increase in the model's usefulness as a source of unsupervised semantic representations of images.  This paper set out to answer that question by taking an existing technique for learning representations with GANs - called BiGAN - and combining it with the BigGan architecture, which hadn't been available when BiGAN was originally published. BiGAN, short for Bidirectional GAN, works by learning both a latent space to image transformation, and also an image to latent space encoder, and then enforcing that pairs of (latent, image) from these two distributions be indistinguishable from one another. They evaluated the quality of learned representations by measuring the performance of a linear model trained using the  encoder's learned latent vectors as input, and did find it to be the case that a BiGAN trained with a BigGAN architecture performs better than one trained with a smaller architecture.


This is a clean, clear summary, but one without any real technical depth or detail; this would be a good writeup to hand someone without any machine learning background who wanted to get an intuitive grasp for unsupervised learning as a field.

Unsupervised learning: the curious pupil

A high-level but well-written explanation of why many believe unsupervised learning will be key to achieving general intelligence, touching on the approaches of GANs and autoregressive models as examples.


This strikes me as a really valuable injection of empirical realism, of the kind that tends to be good for research fields to have periodically, even if it can be a bit painful or frustrating. I appreciate in particular the effort and clarity that this paper puts into articulating the implicit assumptions of how disentanglement can be used or evaluated, and trying to test those assumptions under more real-world settings, such as the one where you don't have any ground truth factors of variation, since the real world doesn't tend to just hand out the Correct factorized model of itself.

Evaluating the Unsupervised Learning of Disentangled Representations

This blog post and paper describe a Google-scale comparative study of different representation learning methods designed to learn "disentangled" representations, where the axes of the representation are aligned with the true underlying factors generating the data. The paper's claims are a sobering result for the field, both theoretically and empirically. Theoretically, they show that in an unsupervised context, it's not possible to find a disentangled representation without embedding some form of inductive bias into your model. Empirically, they present evidence suggesting that variation between random seeds for a given hyperparameter setting (in particular, regularization strength) matters as much or more than variation between that hyperparameter's values. Finally, they run experiments that call into question whether disentangled representations actually support transfer learning, or can be identified as in fact being disentangled without using a metric that relies on having ground truth factors of variation to begin with, making it difficult to evaluate on the many realistic contexts where these aren't available.


This is a cool technique; I like the combination of two approaches (meta-learning and unsupervised learning) aimed at making deep learning applicable to many more real-world datasets. I can imagine promising follow-ups - e.g. randomly scaling embedding dimensions to get different clusters seems a bit hacky to me, so I wonder if there's a better approach (maybe learning many different embeddings?). It's interesting to note that their test-time performance is sometimes better than their training performance, presumably because some of the unsupervised training clusterings are "nonsensical", so there is room to improve here.

Unsupervised Learning via Meta-Learning

This paper trains a meta-learner on tasks which were generated using unsupervised learning. This is done by first learning an (unsupervised) embedding for a dataset, then clustering in that embedding space using k-means. Clustering is done many times with random scaling on each dimension; each meta-learning task is then based on one set of clusters. The resulting meta-learner is then evaluated on the actual task for that dataset, performing better than approaches based just on embeddings, and sometimes getting fairly close to the supervised-learning equivalent.


While the Infomin principle seems powerful and their results look impressive, I am not really convinced that the principle actually played an important role in finding the image augmentations they ended up using, as there is little description of how that happened and the augmentations rather look like the result of combining previously used approaches and doing some hyperparameter optimization.

What makes for good views for contrastive learning

Understanding View Selection for Contrastive Learning

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, Phillip Isola

<@Contrastive multiview learning@>(@Representation Learning with Contrastive Predictive Coding@) is a self-supervised approach to pretraining classifiers in which different views of data points are created and an encoder is trained to minimize the distance between encodings of views corresponding to data points with the same label while maximizing the distance between encodings of views with different labels.

The efficacy of this approach depends on the choice of views as well as the downstream task the neural network is going to be trained for. To find the most promising views, the authors propose the Infomin principle: all views should keep task-relevant information while the mutual information between views is minimized. The principle is supported by various observations: Firstly, earlier approaches to contrastive learning in the image domain that use data augmentation to preserve object identity while creating diverse views can be seen as an implicit application of the Infomin principle. Secondly, varying the mutual information between views (for example by changing the distance between two cropped views of the same image) creates an inverted U-curve for downstream performance corresponding to poor performance if there is too much or too little mutual information between the views. Lastly, the authors also find an inverted U-curve in performance for different colour spaces when using channels as views and the Lab colour space which was built to mimic human colour perception is close to the optimum, meaning that human colour perception might be near-optimal for self-supervised representation learning.

The authors then use the Infomin principle to select image augmentations for contrastive pretraining and improve the state of the art in linear readout on ImageNet from 69.3% to 73% for Top-1 accuracy and from 89% to 91.1% for Top-5 accuracy.


It continues to baffle me why model distillation is so helpful -- you’d think that if you train a student model to mimic a teacher model, it would do at most as well as the teacher, but in fact it seems to do better. It's remarkable that just "training a bigger model and then distilling it down" leads to an increase of _16 percentage points_ (when we just have 1% of the labels). Another thing to add to the list of weird empirical facts about deep learning that we don’t understand.

Big Self-Supervised Models are Strong Semi-Supervised Learners

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton

Previously, <@SimCLR@>(@A Simple Framework for Contrastive Learning of Visual Representations@) showed that you can get good results on semi-supervised learning on ImageNet, by first using self-supervised learning with a contrastive loss to learn good representations for images, and then finetuning a classifier on top of the representations with very few labels. This paper reports a significantly improved score, using three main improvements:

1. Making all of the models larger (in particular, deeper).
2. Incorporating momentum contrast, as done <@previously@>(@Improved Baselines with Momentum Contrastive Learning@).
3. Using model distillation to train a student network to mimic the original finetuned classifier.

On linear classification on top of learned features with a ResNet-50 architecture, they get a top-1 accuracy of 71.7%, so lower than the previous paper. Their main contribution is to show what can be done with larger models. According to top-1 accuracy on ImageNet, the resulting system gets 74.9% with 1% of labels, and 80.1% with 10% of labels. In comparison, standard supervised learning with a ResNet-50 (which is about 33x smaller) achieves 76.6% with all labels, and just 57.9% with 1% of labels and 68.4% with 10% of labels. When they distill down their biggest model into a ResNet-50, it gets 73.9% with 1% of labels and 77.5% with 10% of labels.


As I (Flo) <@previously@>(@The Quiet Semi-Supervised Revolution@) argued, forcing representations to respect invariances in the task seems likely to yield further improvements when labelled data is sparse. However, my intuition is that invariances will be a lot harder to identify for RL or advanced NLP tasks like coming up with consistent novel arguments, so I am unsure how much these improvements will directly matter for our trajectory to transformative AI.

Advancing Self-Supervised and Semi-Supervised Learning with SimCLR

A Simple Framework for Contrastive Learning of Visual Representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton

Contrastive learning is a major recent development, in which we train a neural net to learn representations by giving it the task of maximizing "agreement" between similar images, while minimizing it across dissimilar images. It has been used to achieve excellent results with semi-supervised learning on ImageNet.

The authors performed a large empirical study of contrastive learning. Their framework consists of three components. First, the _data augmentation method_ specifies how to get examples of "similar images": we simply take an (unlabeled) training image, and apply data augmentations to it to create two images that both represent the same underlying image. They consider random crops, color distortion, and Gaussian blur. Second is the _neural network architecture_, which is split into the first several layers f() which compute the representation from the input, and the last few layers g() which compute the similarity from the representation. Finally, the _contrastive loss function_ defines the problem of maximizing agreement between similar images, while minimizing agreement between dissimilar images. They primarily use the same InfoNCE loss used in <@CPC@>(@Representation Learning with Contrastive Predictive Coding@).

They then show many empirical results, including:
1. Having a simple linear layer in g() is not as good as introducing one hidden layer, or in other words, the representations in the penultimate layer are more useful than those in the final layer.
2. Larger batch sizes, longer training, and larger networks matter even more for unsupervised contrastive learning than they do for supervised learning.


Momentum Contrast for Unsupervised Visual Representation Learning

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick

In most deep learning settings, the batch size primarily controls the variance of the gradient, with higher batch sizes decreasing variance. However, with typical contrastive learning, batch size also determines the _task_: typically, the task is to maximize agreement between two examples in the batch, and minimize agreement with all the other examples in the batch. Put another way, given one input, you have to correctly classify which of the remaining examples in the minibatch is a differently transformed version of that input. So, the batch size determines the number of negative examples.

So, besides decreasing variance, large batch sizes also increase the difficulty of the task to be solved. However, such large batch sizes are hard to fit into memory and are computationally expensive. This paper proposes _momentum contrast_ (MoCo), in which we get large numbers of negative examples for contrastive learning, while allowing for small batch sizes.

Think of contrastive learning as a dictionary lookup task -- given one transformed image (the query), you want to find the same image transformed in a different way out of a large list of images (the keys). The key idea of this paper is to have the minibatch contain queries, while using all of the previous N minibatches as the keys (for some N > 1), allowing for many negative examples with a relatively small minibatch.

Of course, this wouldn't help us if we had to encode the keys again each time we trained on a new minibatch. So, instead of storing the images directly as keys, we store their _encoded representations_ in the dictionary, ensuring that we don't have to rerun the encoder every iteration on all of the keys. This is where the computational savings come from.

However, the encoder is being updated over time, which means that different keys are being encoded differently, and there isn't a consistent kind of representation against which similarity can be computed. To solve this, the authors use a momentum-based version of the encoder to encode keys, which ensures that the key encodings change slowly and smoothly, while allowing the query encoder to change rapidly. This means that the query representation and the key representations will be different, but the layers on top of the representations can learn to deal with that. What's important is that _within_ the key representations, the representations are approximately consistent.


Improved Baselines with Momentum Contrastive Learning

Xinlei Chen, Haoqi Fan, Ross Girshick, Kaiming He

This paper applies the insights from the SimCLR paper to the MoCo framework: it adds an extra hidden layer on top of the representations while training on the contrastive loss, and adds the blur data augmentation. This results in a new SOTA on self-supervised representation learning for images.


2018 EMNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

Learning deep representations by mutual information estimation and maximization

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio


Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies

Alessandro Achille, Tom Eccles, Loic Matthey, Christopher P. Burgess, Nick Watters, Alexander Lerchner, Irina Higgins


I’m not sure why we should expect this method to work, but empirically it does. Presumably I’d understand the motivation better if I read through all the related work it’s building on.

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin

There has been a lot of work in self-supervised representation learning for image classification (previously summarized in [AN #92](https://mailchi.mp/d7e950bc8dbd/an-92learning-good-representations-with-contrastive-predictive-coding) and [AN #99](https://mailchi.mp/4f7ffc5cbe53/an-99-doubling-times-for-the-efficiency-of-ai-algorithms)). This paper sets a new SOTA of 75.3% top-1 ImageNet accuracy, when allowed to first do self-supervised representation learning on ImageNet, and then to train a linear classifier on top of the learned features using all of ImageNet.

Previous methods use a contrastive loss across the learned representations (possibly after being processed by a few MLP layers), which can be thought of as using the learned representation to predict the representation of augmented versions of the same input. In contrast, this paper uses the representation to predict “codes” of augmented versions, where the codes are computed using clustering.


Robots that Learn to Use Improvised Tools

Annie Xie,  Frederik Ebert, Sergey Levine, Chelsea Finn


Toward an AI Physicist for Unsupervised Learning


Unsupervised Learning of Syntactic Structure with Invertible Neural Projections

Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick


Conclusion to the sequence on value learning

This post summarizes the value learning sequence, putting emphasis on particular parts. I recommend reading it in full -- the sequence did have an overarching story, which was likely hard to keep track of over the three months that it was being published.


Future directions for narrow value learning

This post summarizes some future directions for narrow value learning that I'm particularly interested in from a long-term perspective.


The human side of interaction

The lens of [human-AI interaction](https://www.alignmentforum.org/posts/4783ufKpx8xvLMPc6/human-ai-interaction) ([AN #41](https://mailchi.mp/8c3f02cabccd/alignment-newsletter-41)) also suggests that we should focus on what the _human_ should do in AI alignment.

Any feedback that the AI system gets must be interpreted using some assumption. For example, when a human provides an AI system a reward function, it shouldn't be interpreted as a description of optimal behavior in every possible situation (which is what we currently do implicitly). [Inverse Reward Design](https://arxiv.org/abs/1711.02827) (IRD) suggests an alternative, more realistic assumption: the reward function is likely to the extent that it leads to high true utility in the training environment. Similarly, in inverse reinforcement learning (IRL) human demonstrations are often interpreted under the assumption of Boltzmann rationality.

Analogously, we may also want to train humans to give feedback to AI systems in the manner that they are expecting. With IRD, the reward designer should make sure to test the reward function extensively in the training environment. If we want our AI system to help us with long-term goals, we may want the overseers to be much more cautious and uncertain in their feedback (depending on how such feedback is interpreted). Techniques that learn to reason like humans, such as iterated amplification and debate, would by default learn to interpret feedback the way humans do. Nevertheless it will probably be useful to train humans to provide useful feedback: for example, in debate, we want humans to judge which side provided more true and useful information.


Following human norms

One approach to preventing catastrophe is to constrain the AI system to never take catastrophic actions, and not focus as much on what to do (which will be solved by progress in AI more generally). In this setting, we hope that our AI systems accelerate our rate of progress, but we remain in control and use AI systems as tools that allow us make better decisions and better technologies. Impact measures / side effect penalties aim to _define_ what not to do. What if we instead _learn_ what not to do? This could look like inferring and following human norms, along the lines of [ad hoc teamwork](http://www.cs.utexas.edu/users/ai-lab/?AdHocTeam).

This is different from narrow value learning for a few reasons. First, narrow value learning also learns what _to_ do. Second, it seems likely that norm inference only gives good results in the context of groups of agents, while narrow value learning could be applied in singe agent settings.

The main advantages of learning norms is that this is something that humans do quite well, so it may be significantly easier than learning "values". In addition, this approach is very similar to our ways of preventing humans from doing catastrophic things: there is a shared, external system of norms that everyone is expected to follow. However, norm following is a weaker standard than [ambitious value learning](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/5eX8ko7GCxwR5N9mN) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)), and there are more problems as a result. Most notably, powerful AI systems will lead to rapidly evolving technologies, that cause big changes in the environment that might require new norms; norm-following AI systems may not be able to create or adapt to these new norms.


Reward uncertainty

Given that we need human feedback for the AI system to stay "on track" as the environment changes, we might design a system that keeps an estimate of the reward, chooses actions that optimize that reward, but also updates the reward over time based on feedback. This has a few issues: it typically assumes that the human Alice knows the true reward function, it makes a possibly-incorrect assumption about the meaning of Alice's feedback, and the AI system still looks like a long-term goal-directed agent where the goal is the current reward estimate.

This post takes the above AI system and considers what happens if you have a distribution over reward functions instead of a point estimate, and during action selection you take into account future updates to the distribution. (This is the setup of [Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137).) While we still assume that Alice knows the true reward function, and we still require an assumption about the meaning of Alice's feedback, the resulting system looks less like a goal-directed agent.

In particular, the system no longer has an incentive to disable the system that learns values from feedback: while previously it changed the AI system's goal (a negative effect from the goal's perspective), now it provides more information about the goal (a positive effect). In addition, the system has more of an incentive to let itself be shut down. If a human is about to shut it down, it should update strongly that whatever it was doing was very bad, causing a drastic update on reward functions. It may still prevent us from shutting it down, but it will at least stop doing the bad thing. Eventually, after gathering enough information, it would converge on the true reward and do the right thing. Of course, this is assuming that the space of rewards is well-specified, which will probably not be true in practice.


Ambitious vs. narrow value learning

While previously I defined narrow value learning as obtaining good behavior in a particular domain, this post defines it as learning the narrow subgoals and instrumental values that the human is pursuing. I believe that these are pointing at the same underlying concept. There are a few reasons to be optimistic about using only narrow value learning to build very powerful AI systems. First, it should be relatively easy to infer many instrumental goals that humans have, such as "acquiring more resources under my control", "better understanding the world and what I want", "remaining in control of deployed AI systems", etc. which an AI system could then pursue with all of its ingenuity. Second, we could infer enough of human preferences to keep humans in a safe environment where they can deliberate in order to figure out what they want to do with the future. Third, humans could use these narrow AI systems as tools in order to implement sophisticated plans, allowing them to perform tasks that we would currently consider to be beyond human ability.


Human-AI Interaction

One of the lessons of control theory is that you can achieve significantly stronger guarantees if you are able to make use of _feedback_. Self-driving cars without any sensors would be basically impossible to build. [Ambitious value learning](https://www.alignmentforum.org/posts/5eX8ko7GCxwR5N9mN/what-is-ambitious-value-learning) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)) aims to find the utility function that will determine the optimal behavior for the rest of time -- without any feedback. However, human preferences and values will evolve over time as we are exposed to new technologies, cultural norms, and governance structures. This is analogous to the environmental disturbances that control theory assumes, and just as with control theory it seems likely that we will need to have feedback, in the form of some data about human preferences, to accommodate these changes.

This suggests we might consider an AI design where the AI system constantly elicits information from humans about their preferences using narrow value learning techniques, and acts based on its current understanding. The obvious way of doing this would be to have an estimate of the reward that is updated over time, and actions are chosen based on the current estimate. However, this still has several problems. Most notably, if the AI system chooses actions that are best according to the current reward estimate, it still has convergent instrumental subgoals, and in particular actions that disable the narrow value learning system to lock in the current reward estimate would be rated very highly. Another problem is that this model assumes that human preferences and values change "by magic" and any such change is good -- but in reality, we likely want to make sure this process is "good", and in particular does not end up being determined by the AI manipulating our preferences.


What is narrow value learning?

This post introduces the concept of narrow value learning (as opposed to [ambitious value learning](https://www.alignmentforum.org/posts/5eX8ko7GCxwR5N9mN/what-is-ambitious-value-learning) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)), where an AI system is trained to produce good behavior within a particular domain, without expecting generalization to novel circumstances. Most current work in the ML community on preference learning or reward learning falls under narrow value learning.


AI safety without goal-directed behavior

The main thrust of the second chapter of the sequence is that it is not _required_ for a superintelligent AI system to be goal-directed. While there are certainly economic arguments suggesting that we will build goal-directed AI, these do not have the force of a theorem. Given the strong arguments we've developed that goal-directed AI would likely be dangerous, it seems worth exploring other options. Some possibilities are AI systems that infer and follow norms, corrigible AI, and bounded and episodic AI services.

These other possibilities can be cast in a utility-maximization framework. However, if you do that then you are once again tempted to say that you are screwed if you get the utility function slightly wrong. Instead, I would want to build these systems in such a way that the desirable properties are inherent to the way that they reason, so that it isn't even a coherent question to ask "what if we get it slightly wrong".


Will humans build goal-directed agents?

The [previous post](https://www.alignmentforum.org/posts/NxF5G6CJiof6cemTw/coherence-arguments-do-not-imply-goal-directed-behavior) argued that coherence arguments do _not_ mean that a superintelligent AI must have goal-directed behavior. In this post, I consider other arguments suggesting that we'll build goal-directed AI systems.

- Since humans are goal-directed, they will build goal-directed AI to help them achieve their goals. *Reaction:* Somewhat agree, but this only shows that the human + AI system should be goal-directed, not the AI itself.
- Goal-directed AI can exceed human performance. *Reaction:* Mostly agree, but there could be alternatives that still exceed human performance.
- Current RL agents are goal-directed. *Reaction:* While the math says this, in practice this doesn't seem true, since RL agents learn from experience rather than planning over the long term.
- Existing intelligent agents are goal-directed. *Reaction:* Seems like a good reason to not build AI using evolution.
- Goal-directed agents are more interpretable and so more desirable. *Reaction:* Disagree, it seems like we're arguing that we should build goal-directed AI so that we can more easily predict that it will cause catastrophe.


I broadly agree with Rohin's points in this and the previous post, and am glad that he's making these arguments explicit. However, while goal-directedness is a tricky property to reason about, I think it's still useful to consider it a property of an agent rather than a property of our model of that agent. It's true that when we have a detailed explanation of how an agent works, we're able to think of cases in which its goal-directedness breaks down (e.g. adversarial examples). However, when these examples are very rare, they don't make much practical difference (e.g. knowing that AlphaGo has a blind spot in certain endgames might not be very helpful in beating it, because you can't get to those endgames).

Coherence arguments do not imply goal-directed behavior

In this post, Rohin argues against the claim that "simply knowing that an agent is intelligent lets us infer that it is goal-directed". He points out that all behaviour can be rationalized as expected utility maximisation over world-histories - but this may not meet our criteria for goal-directed behaviour, and slightly misspecifying such a utility function may well be perfectly safe. What's more interesting - and dangerous - is expected utility maximisation over world-states - but he claims that we shouldn't assume that advanced AI will have this sort of utility function, unless we have additional information (e.g. that it has a utility function simple enough to be explicitly represented). There are plenty of intelligent agents which aren't goal-directed - e.g. ones which are very good at inference but only take trivial actions.


Intuitions about goal-directed behavior

Rohin discusses the "misspecified goal argument for AI risk": that even a small misspecification in goals can lead to adversarial behaviour in advanced AI. He argues that whether behaviour is goal-directed depends on whether it generalises to new situations in ways that are predictable given that goal. He also raises the possibility that thinking of an agent as goal-directed becomes less useful the more we understand about how it works. If true, this would weaken the misspecified goal argument.


Future directions for ambitious value learning

This post is a summary of many different research directions related to ambitious value learning that are currently being pursued.


Latent Variables and Model Mis-Specification

The key thesis of this post is that when you use a probabilistic model with latent variables (also known as hidden variables, or the variables whose vaues you don't know), the values inferred for those latent variables may not have the intended meaning if the model is mis-specified. For example, in inverse reinforcement learning we use a probabilistic model that predicts the _observed_ human behavior from the _latent_ utility function, and we hope to recover the latent utility function and optimize it.

A mis-specified model is one in which there is no setting of the parameters such that the resulting probability distribution matches the _true_ distribution from which the data is sampled. For such a model, even in the limit of infinite data, you are not going to recover the true distribution. (This distinguishes it from _overfitting_, which is not a problem with infinite data.) In this case, instead of the latent variables taking on the values that we want (eg. in IRL, the true utility function), they could be repurposed to explain parts of the distribution that can't be adequately modeled (eg. in IRL, if you don't account for humans learning, you might repurpose the utility function parameters to say that humans like to change up their behavior a lot). If you then use the inferred latent variable values, you're going to be in for a bad time.

So, under mis-specification, the notion of the "true" value of latent variables is no longer meaningful, and the distribution over latent variables that you learn need not match reality. One potential solution would be counterfactual reasoning, which informally means that your model must be able to make good predictions on many different distributions.


Model Mis-specification and Inverse Reinforcement Learning

While the previous post focused on mis-specification in general, this one looks at inverse reinforcement learning (IRL) in particular. In IRL, the latent variable is the utility function, which predicts the observed variable, behavior. They identify three main categories where mis-specification could harm IRL. First, IRL could misunderstand the actions available to the human. For example, if I accidentally hit someone else due to a reflex, but IRL doesn't realize it's a reflex and thinks I could have chosen not to do that, it would infer I don't like the other person. In addition, inferring actions is hard, since in many cases we would have to infer actions from video frames, which is a challenging ML problem. Second, IRL could misunderstand what information and biases are available to the human. If I go to a cafe when it is closed, but IRL thinks that I know it's closed, it's might incorrectly infer a preference for taking a walk. Similarly, if it doesn't know about the planning bias, it might infer that humans don't care about deadlines. Third, IRL may not realize that humans are making long-term plans, especially if the data they are trained on is short and episodic (a form of mis-specification that seems quite likely). If you see a student studying all the time, you might infer that they like studying, instead of that they want a good grade. Indeed, this inference probably gets you 99% accuracy, since the student does in fact spend a lot of time studying. The general issue is that large changes in the model of the human might only lead to small changes in predictive accuracy, and this gets worse with longer-term plans.


Humans can be assigned any values whatsoever…

This post formalizes the thinking in the previous post. Since we need to model human irrationality in order to surpass human performance, we can formalize the human's planning algorithm p, which takes as input a reward or utility function R, and produces a policy pi = p(R). Within this formalism, we would like to infer p and R for a human simultaneously, and then optimize R alone. However, the only constraint we have is that p(R) = pi, and there are many pairs of p and R that work besides the "reasonable" p and R that we are trying to infer. For example, p could be expected utility maximization and R could place reward 1 on the (history, action) pairs in the policy and reward 0 on any pair not in the policy. And for every pair, we can define a new pair (-p, -R) which negates the reward, with (-p)(R) defined to be p(-R), that is the planner negates the reward (returning it to its original form) before using it. We could also have R = 0 and p be the constant function that always outputs the policy pi. All of these pairs reproduce the human policy pi, but if you throw away the planner p and optimize the reward R alone, you will get very different results. You might think that you could avoid this impossibility result by using a simplicity prior, but at least a Kolmogorov simplicity prior barely helps.


Preface to the Sequence on Value Learning

This is a preface, read it if you're going to read the full posts, but not if you're only going to read these summaries.


The easy goal inference problem is still hard

One concrete way of thinking about ambitious value learning is to think about the case where we have the full human policy, that is, we know how a particular human responds to all possible inputs (life experiences, memories, etc). In this case, it is _still_ hard to infer a utility function from the policy. If we infer a utility function assuming that humans are optimal, then an AI system that maximizes this utility function will recover human behavior, but will not surpass it. In order to surpass human performance, we need to accurately model the _mistakes_ a human makes, and correct for them when inferring a utility function. It's not clear how to get this -- the usual approach in machine learning is to choose more accurate models, but in this case even the most accurate model only gets us to human imitation.


What is ambitious value learning?

The specification problem is the problem of _defining_ the behavior we want out of an AI system. If we use the common model of a superintelligent AI maximizing some explicit utility function, this reduces to the problem of defining a utility function whose optimum is achieved by behavior that we want. We know that our utility function is too complex to write down (if it even exists), but perhaps we can learn it from data about human behavior? This is the idea behind _ambitious_ value learning -- to learn a utility function from human behavior that can be safely maximized. Note that since we are targeting the _specification_ problem, we only want to define the behavior, so we can assume infinite compute, infinite data, perfect maximization, etc.


Conference on Uncertainty in Artificial Intelligence, 2018

Lots of AI alignment researchers talk about provable guarantees from our AI system, that are quite broad and comprehensive, even if not a proof of "the AI is aligned and will not cause catastrophe". Both this and the previous paper seem like an advancement in our ability to prove things about neural nets, and so could help with that goal. My probably-controversial opinion is that in the long term the harder problem is actually figuring out what you want to prove, and writing down a formal specification of it in a form that is amenable to formal verification that will generalize to the real world, if you want to go down that path. To be clear, I'm excited about this research, both because it can be used both to solve problems that affect current AI systems (eg. to verify that a neural net on a plane will never crash under a mostly-realistic model of the world) and because it can be used as a tool for developing very capable, safer AI systems in the future -- I just don't expect it to be the main ingredient that gives us confidence that our AI systems are aligned with us.

On the methods themselves, it looks like the Raghunathan paper can achieve much tighter bounds if you use their training procedure, which can optimize the neural net weights in tandem with the certificate of robustness -- they compute a bound of 35% on MNIST with perturbations of up to size 26 (where the maximum is 256). However, there are many restrictions on the applicability of the method. The Dvijotham paper lifts many of these restrictions (multilayer neural nets instead of just one hidden layer, any training procedure allowed) but gets much looser bounds as a result -- the bounds are quite tight at perturbations of size 1 or 2, but by perturbations of size 7-10 the bounds are trivial (i.e. a bound of 100%). The training procedure that Raghunathan et al use is crucial -- without it, their algorithm finds non-trivial bounds on only a single small neural net, for perturbations of size at most 1.

A Dual Approach to Scalable Verification of Deep Networks

Krishnamurthy (Dj)Dvijotham, Robert Stanforth, Sven Gowal, Timothy Mann, Pushmeet Kohli

This paper tackles the same problem as the previous one, but can work on general feedforward and recurrent neural nets, though they show the math specifically for nets with fully connected layers and componentwise activations. They start by defining an optimization problem, where the property to be verified is encoded as the optimization objective, and the mechanics of the neural net are encoded as equality constraints. If the optimal value is negative, then the property has been verified. The key idea to solving this problem is to break down the hard problem of understanding a sequence of linear layers followed by nonlinearities into multiple independent problems each involving a single layer and a nonlinearity. They do this by computing bounds on the values coming out of each layer (both before and after activations), and allowing the constraints to be satisfied with some slack, with the slack variables going into the objective with Lagrange multipliers. This dual problem satisfies weak duality -- the solution to the dual problem for any setting of the Lagrange multipliers constitutes an upper bound on the solution to the original problem. If that upper bound is negative, then we have verified the property. They show how to solve the dual problem -- this is easy now that the slack variables allow us to decouple the layers from each other. They can then compute a tighter upper bound by optimizing over the Lagrange multipliers (which is a convex optimization problem, and can be done using standard techniques). In experiments, they show that the computed bounds on MNIST are reasonably good for very small perturbations, even on networks with 2-3 layers.


Certified Defenses against Adversarial Examples

Aditi Raghunathan, Jacob Steinhardt, Percy Liang

Even when defenses are developed to make neural nets robust against adversarial examples, they are usually broken soon after by stronger attacks. Perhaps we could prove once and for all that the neural net is robust to adversarial examples? The abstract summarizes the approach well: "[W]e study this problem for neural networks with one hidden layer. We first propose a method based on a semidefinite relaxation that outputs a certificate that for a given network and test input, no attack can force the error to exceed a certain value. Second, as this certificate is differentiable, we jointly optimize it with the network parameters, providing an adaptive regularizer that encourages robustness against all attacks. On MNIST, our approach produces a network and a certificate that no attack that perturbs each pixel by at most \epsilon = 0.1 can cause more than 35% test error."

To compute the certificate, they consider the optimal attack A. Given a particular input x, the optimal attack A is the one that changes f(A(x)) to a different class, where f is the ML model, and A(x) is restricted to not change x too much. They leverage the structure of f (linear models and neural nets with one hidden layer) and the restrictions on A to compute a bound on f(A(x)) in terms of x. So, for each data point in the training set, the bound either says “guaranteed that it can’t be adversarially attacked” or “might be possible to adversarially attack it”. Averaging this over the training set or test set gives you an estimate of an upper bound on the optimal adversarial attack success rate.


I really like this as an example of combining the performance of neural networks with the robustness of symbolic approaches. I especially like the fact that the shield is specialized to the current policy and updated over time: I think ML scales so well partly because it only deals with a tiny portion of the input space and can completely ignore the vast majority of possible inputs, and so if you want to add anything on top of ML you need to ensure you preserve this property to ensure scalability. Previous approaches required a shield that is correct across all possible states, failing to preserve this property; in contrast, this approach only requires a shield that is correct for the sequence of learned policies (on whichever states they visit).

I should note that a large portion of why I like this paper is that it feels like it elegantly fits in _both_ the formal verification _and_ the ML fields. (I used to work in programming languages, of which formal verification is a subfield.) On the formal verification side, the guarantees are clean and simple, and the techniques used are canonical. On the ML side, I mentioned above why I like the fact that the shield is policy-specific and updated over time.

As I’ve said before, I think the real challenge in formal verification for AI alignment is how to handle fuzzy specifications. I think this paper shows a path forward: since the safety is established by an inductive invariant that can change over time, we could potentially use human feedback to establish these inductive invariants and update them over time, without requiring a human to fully specify at the outset exactly what is safe and what isn’t. You could think of it as an expanding whitelist of states which the policy is allowed to visit.

Neurosymbolic Reinforcement Learning with Formally Verified Exploration

Greg Anderson, Abhinav Verma, Isil Dillig, Swarat Chaudhuri

A typical approach to formally verified safe exploration in RL is to compute a _shield_, which identifies a safe set of states and actions. After this shield is computed, it is “wrapped” around the environment to ensure that if a potentially unsafe action is about to be taken, it is replaced with a safe one. Then, a policy learning algorithm is applied as normal to learn a good policy.

The key insight of this paper is to compute shields for specific _policies_, rather than creating a one-time shield that must apply to the entire state space. Since any given policy will only visit a small fraction of the state space, the shields are easier to compute and can be more permissive.

They assume access to a _worst-case dynamics model_, which given a state and action outputs a _set_ of states that could be visited. Given a policy π, an _inductive safety invariant_ is a set of safe states that includes all possible initial states and is closed under worst-case transitions: if you start at a state in the set, for any action that π suggests and for any state from the worst-case transition dynamics, that new state will still be in the set. Our algorithm will ensure that any policy we execute will have a corresponding inductive safety invariant.

Formal verification techniques allow us to find inductive safety invariants for restricted classes of policies. This paper uses the space of deterministic, piecewise linear policies as its set of symbolic policies. But how do we apply this to neural nets? The key idea is to start with a safe symbolic policy, convert it to a neurosymbolic policy, take a neural net gradient step, convert back to a safe symbolic policy, and repeat until done. Let’s go over each of these steps.

First, let’s suppose we have a symbolic policy g with inductive safety invariant ø. Then for any neural net f, we construct the policy h = “f(s) if no matter what we stay within ø, otherwise g(s)”. It is easy to see that ø is also an inductive safety invariant for h. Which f should we use to create h? The authors train a neural net to imitate g, and use that as their f. (Note that imitating g only requires executing g in the environment, and we know that g is safe.)

Now that we have our neurosymbolic policy h, we need to take gradient steps on it. We collect data in the environment using h, but then for the gradient we ignore the symbolic part, and take a gradient step as though the data were collected using f. (It seems they used an on-policy algorithm for this, introducing bias; I am not sure why they didn’t simply use an off-policy algorithm.) This produces a new neurosymbolic policy h’ that is still safe (since g and ø are unchanged, and that’s what guarantees safety).

Finally, we need to convert h’ back into a symbolic policy g’. This is done by a version of imitation learning that works in the symbolic policy space, where a new inductive safety invariant for g’ is found using formal verification techniques.

To start off the whole process, we need an initial symbolic policy, which must be constructed by hand. The authors show using experiments in simple continuous control environments that this method can learn high-reward policies without ever having a safety violation.


Certified Adversarial Robustness via Randomized Smoothing

Jeremy M Cohen, Elan Rosenfeld, J. Zico Kolter


Evaluating Robustness of Neural Networks with Mixed Integer Programming

I've only read the abstract so far, but this paper claims to find the _exact_ adversarial accuracy of an MNIST classifier within an L infinity norm ball of radius 0.1, which would be a big step forward in the state of the art for verification.


Since this paper was published at PLDI, it is both longer and goes into a lot more of the details of how to actually perform each of these steps, as well as showing it with a running example on the inverted pendulum (where safety is defined as not going beyond a certain angle). I’m not going to summarize them here but anyone interested in these technical details should check out this paper before the highlighted one (which is constrained by ML page limits and can’t explain the techniques very well).

Just as a reminder that learning programs does not automatically confer interpretability, I present to you the symbolic policy learned by their method for the inverted pendulum:

&&IMAGE HERE&&

An Inductive Synthesis Framework for Verifiable Reinforcement Learning

He Zhu, Zikang Xiong, Stephen Magill, Suresh Jagannathan

This older paper has a pretty similar idea to the one in the highlighted paper. In order to compute a safety shield for a neural network RL agent, we first transform the neural network into a simpler more symbolic policy, prove safety of the symbolic policy, and then use the generated inductive safety invariant as a shield. This paper also uses deterministic piecewise linear policies as its space of symbolic policies. It only proves safety of the final learned RL policy, and so only guarantees safety at deployment, not at training time. (In other words, it does not guarantee safe exploration, and instead assumes that you are training in simulation so that safety is not a concern.)


Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming

Sumanth Dathathri*, Krishnamurthy Dvijotham*, Alexey Kurakin*, Aditi Raghunathan*, Jonathan Uesato*, Rudy Bunel, Shreya Shankar, Jacob Steinhardt, Ian Goodfellow, Percy Liang, Pushmeet Kohli

In parallel with extending verification to sequential settings, as well as learning what specifications to verify, we also need to make verification significantly cheaper in order for it to be feasible to apply it to large neural networks. So far, we have only been able to achieve one of two very desirable properties at a time:

1. The method can scale up to large, independently trained networks. (This has been achieved by methods using linear (LP) relaxations like <@this one@>(@A Dual Approach to Scalable Verification of Deep Networks@).)
2. The method produces tight bounds and thus avoids producing vacuous results. (Achieved by using relaxations based on semidefinite programming (SDP) instead of linear ones.)

This paper shows how you can massage the SDP version such that the resulting algorithm becomes scalable, changing the runtime and memory requirements from O(n^6) and O(n^4) to O(n) per iteration. The resulting algorithm can be applied to larger neural nets than previous SDP approaches and gives much tighter bounds than LP approaches. For example, on an adversarially trained CNN for MNIST (which SDP algorithms haven’t previously been applied to), they can verify 87.8% adversarial accuracy, while LP methods can only verify 0.4%.


It's interesting to see other groups also aiming to have what are essentially robustness guarantees, but motivated instead from the perspective of responsibility and liability. The actual method seems reasonable for the impoverished systems we have today, where we must specify everything that we want the system to do.

Ethical Mission Definition and Execution for Maritime Robots Under Human Supervision

Don Brutzman, Curtis L. Blais, Duane T. Davis, Robert B. McGhee

While underwater robots can perform missions that humans cannot, they cannot be held liable for their actions. Our society requires that someone be responsible for (and can be held liable for) the actions of any such robot, leading to a form of the specification problem: how do we program robots such that it is reasonable to hold their operators accountable for their actions?

This paper divides mission execution into three main parts: the execution level (hardware control), the tactical level (low-level behaviors), and the strategic level (what the robot should do). It proposes that, at the strategic level, we use formal methods to specify what the robot should do. The language should be expressive enough to be useful, while still keeping it sufficiently limited to allow exhaustive testing. They propose using state machines augmented with constraints. The constraints can be used to specify things like "the robot must stay at least 10m away from obstacles". The state machine decides which behaviors to execute, and each such behavior can have three results: success, failure, or exception (in the case that a constraint would have been violated had the behavior continued operating).


Training verified learners with learned verifiers

Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O'Donoghue, Jonathan Uesato, Pushmeet Kohli


Verification and Transparency

This post points out that verification and transparency have similar goals. Transparency produces an artefact that allows the user to answer questions about the system under investigation (e.g. "why did the neural net predict that this was a tennis ball?"). Verification on the other hand allows the user to pose a question, and then automatically answers that question (e.g. "is there an adversarial example for this image?").


Optimization + Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness

Greg Anderson, Shankara Pailoor, Isil Dillig, Swarat Chaudhuri


Algorithms for Verifying Deep Neural Networks

Changliu Liu, Tomer Arnon, Christopher Lazarus, Clark Barrett, Mykel J. Kochenderfer

This is a survey paper about verification of properties of deep neural nets.


On a Formal Model of Safe and Scalable Self-driving Cars

Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua


On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models

Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, Pushmeet Kohli


Verification for Machine Learning, Autonomy, and Neural Networks Survey

Weiming Xiang, Patrick Musau, Ayana A. Wild, Diego Manzanas Lopez, Nathaniel Hamilton, Xiaodong Yang, Joel Rosenfeld, Taylor T. Johnson


As with the previous papers on verification, I'm excited in the improvement in our capability to prove things about neural nets. I do think that the more important problem is how to even state properties that we care about in a way that we could begin to prove them. For example, [last week](https://mailchi.mp/d7b5059d64ed/alignment-newsletter-24) we saw the unrestricted adversarial examples challenge, where humans are the judge of what a legal example is -- how can we formalize that for a verification approach?

On this paper specifically, I wish they had included the number of timeouts that their method has -- it's hard to interpret the provable accuracy numbers without that. Based on the numbers in the paper, I'm guessing this method is still much more computationally expensive than other methods. If so, I'm not sure what benefit it gives over them -- presumably it's that we can compute the exact adversarial accuracy, but if we don't have enough compute, such that other methods can prove better lower bounds anyway, then it doesn't seem worth it.

Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability

Kai Y. Xiao, Vincent Tjeng, Nur Muhammad Shafiullah, Aleksander Madry

The idea behind verification is to consider all possible inputs at the same time, and show that no matter what the input is, a particular property is satisfied. In ML, this is typically applied to adversarial examples, where inputs are constrained to be within the L-infinity norm ball of dataset examples. Prior papers on verification (covered in [AN #19](https://mailchi.mp/4b19d2caa5a9/alignment-newsletter-19)) solve a computationally easier relaxation of the verification problem, that gives a lower bound on the performance of the classifier. This paper aims to use exact verification, since it can compute the exact adversarial performance of the classifier on the test set, and to figure out how to improve its performance.

One easy place to start is to encourage weights to be zero, since these can be pruned from the problem fed in to the constraint solver. (Or more likely, they feed it in anyway, but the constraint solver immediately gets rid of them -- constraint solvers are pretty smart.) This can be done using L1 regularization and pruning small weights. This already gives two orders of magnitude of speedup, making it able to verify that there is no adversarial attack with ϵ = 0.1 on a particular MNIST digit in 11 seconds on average.

Next, they note that verification with linear constraints and functions is easy -- the challenging aspect is the Relu units that force the verifier to branch into two cases. (Since relu(x) = max(x, 0), it is the identity function when x is positive, and the zero function otherwise.) So why not try to ensure that the Relu units are also linear? Obviously we can't just make all the Relu units linear -- the whole point of them is to introduce nonlinearity to make the neural net more expressive. But as a start, we can look at the behavior of the Relu units on the examples we have, and if they are almost always active (inputs are positive) or almost always inactive (inputs are negative), then we replace them with the corresponding linear function (identity and zero, respectively), which is easier to verify. This gets another ~2x speedup.

But what if we could also change the training procedure? Maybe we could augment the loss so that the Relu units are either decisively active or decisively inactive on any dataset example. They propose that _during training_ we consider the L-infinity norm ball around each example, use that to create intervals that each pixel must be in, and then make a forward pass through the neural net using interval arithmetic (which is fast but inexact). Then, we add a term to the loss that incentivizes the interval for the input to each Relu to exclude zero (so that the Relu is either always active or always inactive). They call this the Relu Stability loss, or RS loss.

This leads to a further 4-13x speedup with similar test set accuracy. They then also test on MNIST with ϵ = 0.2, 0.3 and CIFAR with ϵ = 2/255, 8/255. It leads to speedup in all cases, with similar test set accuracy on MNIST but reduced accuracy on CIFAR. The provable accuracy goes up, but this is probably because when there's no RS loss, more images time out in verification, not because the network becomes better at classification. Other verification methods do get better provable accuracies on CIFAR, even though in principle they could fail to detect that a safe example is safe. This could be because their method times out frequently, or because their method degrades the neural net classifier -- it's hard to tell since they don't report number of timeouts.


It is pretty nice to have a program representation. This paper doesn't delve into specifics (besides a motivating example worked out by hand), but I'm excited to see an actual instantiation of this framework in the future!

Towards Mixed Optimization for Reinforcement Learning with Program Synthesis

Surya Bhupatiraju, Kumar Krishna Agrawal, Rishabh Singh

This paper proposes a framework in which policies are represented in two different ways -- as neural nets (the usual way) and as programs. To go from neural nets to programs, you use _program synthesis_ (as done by [VIPER](http://arxiv.org/abs/1805.08328) and [PIRL](https://arxiv.org/abs/1804.02477), both summarized in previous newsletters). To go from programs to neural nets, you use _distillation_ (basically use the program to train the neural net with supervised training). Given these transformations, you can then work with the policy in either space. For example, you could optimize the policy in both spaces, using standard gradient descent in neural-net-space, and _program repair_ in program-space. Having a program representation can be helpful in other ways too, as it makes the policy more interpretable, and more amenable to formal verification of safety properties.


While the approach is straightforward and will certainly increase robustness in many cases, it seems worth mentioning two serious issues. First, they assume that the initial DQN training learns the perfect Q function. Second, the provided certificates are about individual actions, not policy performance: the Q-values approximated in DQN assume optimal performance starting from the next action, which is not a given here. I am a bit concerned that these limitations were not really discussed, while the paper claims that “the resulting policy comes with a certificate of solution quality”.

Certified Adversarial Robustness for Deep Reinforcement Learning

Michael Everett*, Bjorn Lutjens*, Jonathan P. How

<@Certified adversarial robustness@>(@Certified Defenses against Adversarial Examples@) provides guarantees about the effects of small perturbations on a neural network’s outputs. This paper uses that approach to make reinforcement learning more robust by training a DQN and acting by choosing the action with the best worst-case Q-value under adversarial perturbations (called the robust-optimal action) estimated from the certificate bounds, instead of the action with the highest Q-value.

The approach is evaluated on Cartpole and a navigation task that requires avoiding collisions, with an adversary perturbing observations in both cases. For small perturbations, this technique actually increases performance, but as perturbations get large the agent’s conservatism can lead to a large degradation in performance.


MixTrain: Scalable Training of Formally Robust Neural Networks

Shiqi Wang, Yizheng Chen, Ahmed Abdou, Suman Jana


Formal Methods for the Informal Engineer: Workshop Recommendations

Gopal Sarma, James Koppel, Gregory Malecha, Patrick Schultz, Eric Drexler, Ramana Kumar, Cody Roux, Philip Zucker

This is the writeup from the <@Formal Methods for the Informal Engineer@> workshop. The main thrust is a call for increased application of formal methods in order to increase confidence in critical AI/ML systems, especially in the life sciences. They provide five high-level recommendations for this purpose.


Verifiably Safe Exploration for End-to-End Reinforcement Learning

Nathan Hunt, Nathan Fulton, Sara Magliacane, Nghia Hoang, Subhro Das, Armando Solar-Lezama

As we saw in the highlight, applications of formal verification to reinforcement learning and safe exploration often rely on _shielding_, in which any proposed unsafe actions are replaced by randomly chosen safe actions. Typically, this requires having an MDP model in a high-level, symbolic state space, such as by defining the MDP over the Atari simulator state, rather than learning from pixels.

This paper demonstrates that we can relax this requirement and learn policies on low-level observations, while still getting the safety guarantees of the shielding approach. The approach is simple: we define (manually) an abstract model of the environment, with a symbolic state space and dynamics model, and use this to create a shield as usual. Then, to learn the policy (which gets pixels as input), we use an object detector to transform the pixels into a symbolic state, and then use the shield if necessary to select which action to take. The authors show that as long as the error of the object detection step is low, the overall policy learning will remain safe.


Many people believe that ultimately we will need to prove theorems about the safety of our AIs. I don't understand yet what kind of theorems they have in mind, so I don't really want to speculate on how this relates to it. It does seem like the robustness guarantee is the most relevant one, since in general we won't have access to a perfect model of the dynamics.

Verifiable Reinforcement Learning via Policy Extraction

Osbert Bastani, Yewen Pu, Armando Solar-Lezama

Since it is hard to verify properties of neural nets, we can instead first train a decision tree policy to mimic the policy learned by deep RL, and then verify properties about that. The authors generalize [DAGGER](https://www.cs.cmu.edu/~sross1/publications/Ross-AIStats11-NoRegret.pdf) to take advantage of the Q-function and extract decision tree policies. They then prove a correctness guarantee for a toy version of Pong (where the dynamics are known), a robustness guarantee for Pong (with symbolic states, not pixels) (which can be done without known dynamics), and stability of cartpole.


A Game-Based Approximate Verification of Deep Neural Networks with Provable Guarantees

Min Wu, Matthew Wicker, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska