Human-AI Interaction

One of the lessons of control theory is that you can achieve significantly stronger guarantees if you are able to make use of _feedback_. Self-driving cars without any sensors would be basically impossible to build. [Ambitious value learning](https://www.alignmentforum.org/posts/5eX8ko7GCxwR5N9mN/what-is-ambitious-value-learning) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)) aims to find the utility function that will determine the optimal behavior for the rest of time -- without any feedback. However, human preferences and values will evolve over time as we are exposed to new technologies, cultural norms, and governance structures. This is analogous to the environmental disturbances that control theory assumes, and just as with control theory it seems likely that we will need to have feedback, in the form of some data about human preferences, to accommodate these changes.

This suggests we might consider an AI design where the AI system constantly elicits information from humans about their preferences using narrow value learning techniques, and acts based on its current understanding. The obvious way of doing this would be to have an estimate of the reward that is updated over time, and actions are chosen based on the current estimate. However, this still has several problems. Most notably, if the AI system chooses actions that are best according to the current reward estimate, it still has convergent instrumental subgoals, and in particular actions that disable the narrow value learning system to lock in the current reward estimate would be rated very highly. Another problem is that this model assumes that human preferences and values change "by magic" and any such change is good -- but in reality, we likely want to make sure this process is "good", and in particular does not end up being determined by the AI manipulating our preferences.


What is narrow value learning?

This post introduces the concept of narrow value learning (as opposed to [ambitious value learning](https://www.alignmentforum.org/posts/5eX8ko7GCxwR5N9mN/what-is-ambitious-value-learning) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)), where an AI system is trained to produce good behavior within a particular domain, without expecting generalization to novel circumstances. Most current work in the ML community on preference learning or reward learning falls under narrow value learning.


AI safety without goal-directed behavior

The main thrust of the second chapter of the sequence is that it is not _required_ for a superintelligent AI system to be goal-directed. While there are certainly economic arguments suggesting that we will build goal-directed AI, these do not have the force of a theorem. Given the strong arguments we've developed that goal-directed AI would likely be dangerous, it seems worth exploring other options. Some possibilities are AI systems that infer and follow norms, corrigible AI, and bounded and episodic AI services.

These other possibilities can be cast in a utility-maximization framework. However, if you do that then you are once again tempted to say that you are screwed if you get the utility function slightly wrong. Instead, I would want to build these systems in such a way that the desirable properties are inherent to the way that they reason, so that it isn't even a coherent question to ask "what if we get it slightly wrong".


Will humans build goal-directed agents?

The [previous post](https://www.alignmentforum.org/posts/NxF5G6CJiof6cemTw/coherence-arguments-do-not-imply-goal-directed-behavior) argued that coherence arguments do _not_ mean that a superintelligent AI must have goal-directed behavior. In this post, I consider other arguments suggesting that we'll build goal-directed AI systems.

- Since humans are goal-directed, they will build goal-directed AI to help them achieve their goals. *Reaction:* Somewhat agree, but this only shows that the human + AI system should be goal-directed, not the AI itself.
- Goal-directed AI can exceed human performance. *Reaction:* Mostly agree, but there could be alternatives that still exceed human performance.
- Current RL agents are goal-directed. *Reaction:* While the math says this, in practice this doesn't seem true, since RL agents learn from experience rather than planning over the long term.
- Existing intelligent agents are goal-directed. *Reaction:* Seems like a good reason to not build AI using evolution.
- Goal-directed agents are more interpretable and so more desirable. *Reaction:* Disagree, it seems like we're arguing that we should build goal-directed AI so that we can more easily predict that it will cause catastrophe.


I broadly agree with Rohin's points in this and the previous post, and am glad that he's making these arguments explicit. However, while goal-directedness is a tricky property to reason about, I think it's still useful to consider it a property of an agent rather than a property of our model of that agent. It's true that when we have a detailed explanation of how an agent works, we're able to think of cases in which its goal-directedness breaks down (e.g. adversarial examples). However, when these examples are very rare, they don't make much practical difference (e.g. knowing that AlphaGo has a blind spot in certain endgames might not be very helpful in beating it, because you can't get to those endgames).

Coherence arguments do not imply goal-directed behavior

In this post, Rohin argues against the claim that "simply knowing that an agent is intelligent lets us infer that it is goal-directed". He points out that all behaviour can be rationalized as expected utility maximisation over world-histories - but this may not meet our criteria for goal-directed behaviour, and slightly misspecifying such a utility function may well be perfectly safe. What's more interesting - and dangerous - is expected utility maximisation over world-states - but he claims that we shouldn't assume that advanced AI will have this sort of utility function, unless we have additional information (e.g. that it has a utility function simple enough to be explicitly represented). There are plenty of intelligent agents which aren't goal-directed - e.g. ones which are very good at inference but only take trivial actions.


Intuitions about goal-directed behavior

Rohin discusses the "misspecified goal argument for AI risk": that even a small misspecification in goals can lead to adversarial behaviour in advanced AI. He argues that whether behaviour is goal-directed depends on whether it generalises to new situations in ways that are predictable given that goal. He also raises the possibility that thinking of an agent as goal-directed becomes less useful the more we understand about how it works. If true, this would weaken the misspecified goal argument.


Future directions for ambitious value learning

This post is a summary of many different research directions related to ambitious value learning that are currently being pursued.


Latent Variables and Model Mis-Specification

The key thesis of this post is that when you use a probabilistic model with latent variables (also known as hidden variables, or the variables whose vaues you don't know), the values inferred for those latent variables may not have the intended meaning if the model is mis-specified. For example, in inverse reinforcement learning we use a probabilistic model that predicts the _observed_ human behavior from the _latent_ utility function, and we hope to recover the latent utility function and optimize it.

A mis-specified model is one in which there is no setting of the parameters such that the resulting probability distribution matches the _true_ distribution from which the data is sampled. For such a model, even in the limit of infinite data, you are not going to recover the true distribution. (This distinguishes it from _overfitting_, which is not a problem with infinite data.) In this case, instead of the latent variables taking on the values that we want (eg. in IRL, the true utility function), they could be repurposed to explain parts of the distribution that can't be adequately modeled (eg. in IRL, if you don't account for humans learning, you might repurpose the utility function parameters to say that humans like to change up their behavior a lot). If you then use the inferred latent variable values, you're going to be in for a bad time.

So, under mis-specification, the notion of the "true" value of latent variables is no longer meaningful, and the distribution over latent variables that you learn need not match reality. One potential solution would be counterfactual reasoning, which informally means that your model must be able to make good predictions on many different distributions.


Model Mis-specification and Inverse Reinforcement Learning

While the previous post focused on mis-specification in general, this one looks at inverse reinforcement learning (IRL) in particular. In IRL, the latent variable is the utility function, which predicts the observed variable, behavior. They identify three main categories where mis-specification could harm IRL. First, IRL could misunderstand the actions available to the human. For example, if I accidentally hit someone else due to a reflex, but IRL doesn't realize it's a reflex and thinks I could have chosen not to do that, it would infer I don't like the other person. In addition, inferring actions is hard, since in many cases we would have to infer actions from video frames, which is a challenging ML problem. Second, IRL could misunderstand what information and biases are available to the human. If I go to a cafe when it is closed, but IRL thinks that I know it's closed, it's might incorrectly infer a preference for taking a walk. Similarly, if it doesn't know about the planning bias, it might infer that humans don't care about deadlines. Third, IRL may not realize that humans are making long-term plans, especially if the data they are trained on is short and episodic (a form of mis-specification that seems quite likely). If you see a student studying all the time, you might infer that they like studying, instead of that they want a good grade. Indeed, this inference probably gets you 99% accuracy, since the student does in fact spend a lot of time studying. The general issue is that large changes in the model of the human might only lead to small changes in predictive accuracy, and this gets worse with longer-term plans.


Humans can be assigned any values whatsoever…

This post formalizes the thinking in the previous post. Since we need to model human irrationality in order to surpass human performance, we can formalize the human's planning algorithm p, which takes as input a reward or utility function R, and produces a policy pi = p(R). Within this formalism, we would like to infer p and R for a human simultaneously, and then optimize R alone. However, the only constraint we have is that p(R) = pi, and there are many pairs of p and R that work besides the "reasonable" p and R that we are trying to infer. For example, p could be expected utility maximization and R could place reward 1 on the (history, action) pairs in the policy and reward 0 on any pair not in the policy. And for every pair, we can define a new pair (-p, -R) which negates the reward, with (-p)(R) defined to be p(-R), that is the planner negates the reward (returning it to its original form) before using it. We could also have R = 0 and p be the constant function that always outputs the policy pi. All of these pairs reproduce the human policy pi, but if you throw away the planner p and optimize the reward R alone, you will get very different results. You might think that you could avoid this impossibility result by using a simplicity prior, but at least a Kolmogorov simplicity prior barely helps.


The easy goal inference problem is still hard

One concrete way of thinking about ambitious value learning is to think about the case where we have the full human policy, that is, we know how a particular human responds to all possible inputs (life experiences, memories, etc). In this case, it is _still_ hard to infer a utility function from the policy. If we infer a utility function assuming that humans are optimal, then an AI system that maximizes this utility function will recover human behavior, but will not surpass it. In order to surpass human performance, we need to accurately model the _mistakes_ a human makes, and correct for them when inferring a utility function. It's not clear how to get this -- the usual approach in machine learning is to choose more accurate models, but in this case even the most accurate model only gets us to human imitation.


What is ambitious value learning?

The specification problem is the problem of _defining_ the behavior we want out of an AI system. If we use the common model of a superintelligent AI maximizing some explicit utility function, this reduces to the problem of defining a utility function whose optimum is achieved by behavior that we want. We know that our utility function is too complex to write down (if it even exists), but perhaps we can learn it from data about human behavior? This is the idea behind _ambitious_ value learning -- to learn a utility function from human behavior that can be safely maximized. Note that since we are targeting the _specification_ problem, we only want to define the behavior, so we can assume infinite compute, infinite data, perfect maximization, etc.


Conference on Uncertainty in Artificial Intelligence, 2018

Lots of AI alignment researchers talk about provable guarantees from our AI system, that are quite broad and comprehensive, even if not a proof of "the AI is aligned and will not cause catastrophe". Both this and the previous paper seem like an advancement in our ability to prove things about neural nets, and so could help with that goal. My probably-controversial opinion is that in the long term the harder problem is actually figuring out what you want to prove, and writing down a formal specification of it in a form that is amenable to formal verification that will generalize to the real world, if you want to go down that path. To be clear, I'm excited about this research, both because it can be used both to solve problems that affect current AI systems (eg. to verify that a neural net on a plane will never crash under a mostly-realistic model of the world) and because it can be used as a tool for developing very capable, safer AI systems in the future -- I just don't expect it to be the main ingredient that gives us confidence that our AI systems are aligned with us.

On the methods themselves, it looks like the Raghunathan paper can achieve much tighter bounds if you use their training procedure, which can optimize the neural net weights in tandem with the certificate of robustness -- they compute a bound of 35% on MNIST with perturbations of up to size 26 (where the maximum is 256). However, there are many restrictions on the applicability of the method. The Dvijotham paper lifts many of these restrictions (multilayer neural nets instead of just one hidden layer, any training procedure allowed) but gets much looser bounds as a result -- the bounds are quite tight at perturbations of size 1 or 2, but by perturbations of size 7-10 the bounds are trivial (i.e. a bound of 100%). The training procedure that Raghunathan et al use is crucial -- without it, their algorithm finds non-trivial bounds on only a single small neural net, for perturbations of size at most 1.

A Dual Approach to Scalable Verification of Deep Networks

Krishnamurthy (Dj)Dvijotham, Robert Stanforth, Sven Gowal, Timothy Mann, Pushmeet Kohli

This paper tackles the same problem as the previous one, but can work on general feedforward and recurrent neural nets, though they show the math specifically for nets with fully connected layers and componentwise activations. They start by defining an optimization problem, where the property to be verified is encoded as the optimization objective, and the mechanics of the neural net are encoded as equality constraints. If the optimal value is negative, then the property has been verified. The key idea to solving this problem is to break down the hard problem of understanding a sequence of linear layers followed by nonlinearities into multiple independent problems each involving a single layer and a nonlinearity. They do this by computing bounds on the values coming out of each layer (both before and after activations), and allowing the constraints to be satisfied with some slack, with the slack variables going into the objective with Lagrange multipliers. This dual problem satisfies weak duality -- the solution to the dual problem for any setting of the Lagrange multipliers constitutes an upper bound on the solution to the original problem. If that upper bound is negative, then we have verified the property. They show how to solve the dual problem -- this is easy now that the slack variables allow us to decouple the layers from each other. They can then compute a tighter upper bound by optimizing over the Lagrange multipliers (which is a convex optimization problem, and can be done using standard techniques). In experiments, they show that the computed bounds on MNIST are reasonably good for very small perturbations, even on networks with 2-3 layers.


Certified Defenses against Adversarial Examples

Aditi Raghunathan, Jacob Steinhardt, Percy Liang

Even when defenses are developed to make neural nets robust against adversarial examples, they are usually broken soon after by stronger attacks. Perhaps we could prove once and for all that the neural net is robust to adversarial examples? The abstract summarizes the approach well: "[W]e study this problem for neural networks with one hidden layer. We first propose a method based on a semidefinite relaxation that outputs a certificate that for a given network and test input, no attack can force the error to exceed a certain value. Second, as this certificate is differentiable, we jointly optimize it with the network parameters, providing an adaptive regularizer that encourages robustness against all attacks. On MNIST, our approach produces a network and a certificate that no attack that perturbs each pixel by at most \epsilon = 0.1 can cause more than 35% test error."

To compute the certificate, they consider the optimal attack A. Given a particular input x, the optimal attack A is the one that changes f(A(x)) to a different class, where f is the ML model, and A(x) is restricted to not change x too much. They leverage the structure of f (linear models and neural nets with one hidden layer) and the restrictions on A to compute a bound on f(A(x)) in terms of x. So, for each data point in the training set, the bound either says "guaranteed that it can't be adversarially attacked" or "might be possible to adversarially attack it". Averaging this over the training set or test set gives you an estimate of an upper bound on the optimal adversarial attack success rate.


I really like this as an example of combining the performance of neural networks with the robustness of symbolic approaches. I especially like the fact that the shield is specialized to the current policy and updated over time: I think ML scales so well partly because it only deals with a tiny portion of the input space and can completely ignore the vast majority of possible inputs, and so if you want to add anything on top of ML you need to ensure you preserve this property to ensure scalability. Previous approaches required a shield that is correct across all possible states, failing to preserve this property; in contrast, this approach only requires a shield that is correct for the sequence of learned policies (on whichever states they visit).

I should note that a large portion of why I like this paper is that it feels like it elegantly fits in _both_ the formal verification _and_ the ML fields. (I used to work in programming languages, of which formal verification is a subfield.) On the formal verification side, the guarantees are clean and simple, and the techniques used are canonical. On the ML side, I mentioned above why I like the fact that the shield is policy-specific and updated over time.

As I’ve said before, I think the real challenge in formal verification for AI alignment is how to handle fuzzy specifications. I think this paper shows a path forward: since the safety is established by an inductive invariant that can change over time, we could potentially use human feedback to establish these inductive invariants and update them over time, without requiring a human to fully specify at the outset exactly what is safe and what isn’t. You could think of it as an expanding whitelist of states which the policy is allowed to visit.

Neurosymbolic Reinforcement Learning with Formally Verified Exploration

Greg Anderson, Abhinav Verma, Isil Dillig, Swarat Chaudhuri

A typical approach to formally verified safe exploration in RL is to compute a _shield_, which identifies a safe set of states and actions. After this shield is computed, it is "wrapped" around the environment to ensure that if a potentially unsafe action is about to be taken, it is replaced with a safe one. Then, a policy learning algorithm is applied as normal to learn a good policy.

The key insight of this paper is to compute shields for specific _policies_, rather than creating a one-time shield that must apply to the entire state space. Since any given policy will only visit a small fraction of the state space, the shields are easier to compute and can be more permissive.

They assume access to a _worst-case dynamics model_, which given a state and action outputs a _set_ of states that could be visited. Given a policy π, an _inductive safety invariant_ is a set of safe states that includes all possible initial states and is closed under worst-case transitions: if you start at a state in the set, for any action that π suggests and for any state from the worst-case transition dynamics, that new state will still be in the set. Our algorithm will ensure that any policy we execute will have a corresponding inductive safety invariant.

Formal verification techniques allow us to find inductive safety invariants for restricted classes of policies. This paper uses the space of deterministic, piecewise linear policies as its set of symbolic policies. But how do we apply this to neural nets? The key idea is to start with a safe symbolic policy, convert it to a neurosymbolic policy, take a neural net gradient step, convert back to a safe symbolic policy, and repeat until done. Let's go over each of these steps.

First, let's suppose we have a symbolic policy g with inductive safety invariant ø. Then for any neural net f, we construct the policy h = "f(s) if no matter what we stay within ø, otherwise g(s)". It is easy to see that ø is also an inductive safety invariant for h. Which f should we use to create h? The authors train a neural net to imitate g, and use that as their f. (Note that imitating g only requires executing g in the environment, and we know that g is safe.)

Now that we have our neurosymbolic policy h, we need to take gradient steps on it. We collect data in the environment using h, but then for the gradient we ignore the symbolic part, and take a gradient step as though the data were collected using f. (It seems they used an on-policy algorithm for this, introducing bias; I am not sure why they didn't simply use an off-policy algorithm.) This produces a new neurosymbolic policy h' that is still safe (since g and ø are unchanged, and that's what guarantees safety).

Finally, we need to convert h' back into a symbolic policy g'. This is done by a version of imitation learning that works in the symbolic policy space, where a new inductive safety invariant for g' is found using formal verification techniques.

To start off the whole process, we need an initial symbolic policy, which must be constructed by hand. The authors show using experiments in simple continuous control environments that this method can learn high-reward policies without ever having a safety violation.


Evaluating Robustness of Neural Networks with Mixed Integer Programming

I've only read the abstract so far, but this paper claims to find the _exact_ adversarial accuracy of an MNIST classifier within an L infinity norm ball of radius 0.1, which would be a big step forward in the state of the art for verification.


Since this paper was published at PLDI, it is both longer and goes into a lot more of the details of how to actually perform each of these steps, as well as showing it with a running example on the inverted pendulum (where safety is defined as not going beyond a certain angle). I’m not going to summarize them here but anyone interested in these technical details should check out this paper before the highlighted one (which is constrained by ML page limits and can’t explain the techniques very well).

Just as a reminder that learning programs does not automatically confer interpretability, I present to you the symbolic policy learned by their method for the inverted pendulum:

&&IMAGE HERE&&

An Inductive Synthesis Framework for Verifiable Reinforcement Learning

He Zhu, Zikang Xiong, Stephen Magill, Suresh Jagannathan

This older paper has a pretty similar idea to the one in the highlighted paper. In order to compute a safety shield for a neural network RL agent, we first transform the neural network into a simpler more symbolic policy, prove safety of the symbolic policy, and then use the generated inductive safety invariant as a shield. This paper also uses deterministic piecewise linear policies as its space of symbolic policies. It only proves safety of the final learned RL policy, and so only guarantees safety at deployment, not at training time. (In other words, it does not guarantee safe exploration, and instead assumes that you are training in simulation so that safety is not a concern.)


Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming

Sumanth Dathathri*, Krishnamurthy Dvijotham*, Alexey Kurakin*, Aditi Raghunathan*, Jonathan Uesato*, Rudy Bunel, Shreya Shankar, Jacob Steinhardt, Ian Goodfellow, Percy Liang, Pushmeet Kohli

In parallel with extending verification to sequential settings, as well as learning what specifications to verify, we also need to make verification significantly cheaper in order for it to be feasible to apply it to large neural networks. So far, we have only been able to achieve one of two very desirable properties at a time:

1. The method can scale up to large, independently trained networks. (This has been achieved by methods using linear (LP) relaxations like <@this one@>(@A Dual Approach to Scalable Verification of Deep Networks@).)
2. The method produces tight bounds and thus avoids producing vacuous results. (Achieved by using relaxations based on semidefinite programming (SDP) instead of linear ones.)

This paper shows how you can massage the SDP version such that the resulting algorithm becomes scalable, changing the runtime and memory requirements from O(n^6) and O(n^4) to O(n) per iteration. The resulting algorithm can be applied to larger neural nets than previous SDP approaches and gives much tighter bounds than LP approaches. For example, on an adversarially trained CNN for MNIST (which SDP algorithms haven't previously been applied to), they can verify 87.8% adversarial accuracy, while LP methods can only verify 0.4%.


It's interesting to see other groups also aiming to have what are essentially robustness guarantees, but motivated instead from the perspective of responsibility and liability. The actual method seems reasonable for the impoverished systems we have today, where we must specify everything that we want the system to do.

Ethical Mission Definition and Execution for Maritime Robots Under Human Supervision

Don Brutzman, Curtis L. Blais, Duane T. Davis, Robert B. McGhee

While underwater robots can perform missions that humans cannot, they cannot be held liable for their actions. Our society requires that someone be responsible for (and can be held liable for) the actions of any such robot, leading to a form of the specification problem: how do we program robots such that it is reasonable to hold their operators accountable for their actions?

This paper divides mission execution into three main parts: the execution level (hardware control), the tactical level (low-level behaviors), and the strategic level (what the robot should do). It proposes that, at the strategic level, we use formal methods to specify what the robot should do. The language should be expressive enough to be useful, while still keeping it sufficiently limited to allow exhaustive testing. They propose using state machines augmented with constraints. The constraints can be used to specify things like "the robot must stay at least 10m away from obstacles". The state machine decides which behaviors to execute, and each such behavior can have three results: success, failure, or exception (in the case that a constraint would have been violated had the behavior continued operating).


Verification and Transparency

This post points out that verification and transparency have similar goals. Transparency produces an artefact that allows the user to answer questions about the system under investigation (e.g. "why did the neural net predict that this was a tennis ball?"). Verification on the other hand allows the user to pose a question, and then automatically answers that question (e.g. "is there an adversarial example for this image?").


Algorithms for Verifying Deep Neural Networks

Changliu Liu, Tomer Arnon, Christopher Lazarus, Clark Barrett, Mykel J. Kochenderfer

This is a survey paper about verification of properties of deep neural nets.


As with the previous papers on verification, I'm excited in the improvement in our capability to prove things about neural nets. I do think that the more important problem is how to even state properties that we care about in a way that we could begin to prove them. For example, [last week](https://mailchi.mp/d7b5059d64ed/alignment-newsletter-24) we saw the unrestricted adversarial examples challenge, where humans are the judge of what a legal example is -- how can we formalize that for a verification approach?

On this paper specifically, I wish they had included the number of timeouts that their method has -- it's hard to interpret the provable accuracy numbers without that. Based on the numbers in the paper, I'm guessing this method is still much more computationally expensive than other methods. If so, I'm not sure what benefit it gives over them -- presumably it's that we can compute the exact adversarial accuracy, but if we don't have enough compute, such that other methods can prove better lower bounds anyway, then it doesn't seem worth it.

Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability

Kai Y. Xiao, Vincent Tjeng, Nur Muhammad Shafiullah, Aleksander Madry

The idea behind verification is to consider all possible inputs at the same time, and show that no matter what the input is, a particular property is satisfied. In ML, this is typically applied to adversarial examples, where inputs are constrained to be within the L-infinity norm ball of dataset examples. Prior papers on verification (covered in [AN #19](https://mailchi.mp/4b19d2caa5a9/alignment-newsletter-19)) solve a computationally easier relaxation of the verification problem, that gives a lower bound on the performance of the classifier. This paper aims to use exact verification, since it can compute the exact adversarial performance of the classifier on the test set, and to figure out how to improve its performance.

One easy place to start is to encourage weights to be zero, since these can be pruned from the problem fed in to the constraint solver. (Or more likely, they feed it in anyway, but the constraint solver immediately gets rid of them -- constraint solvers are pretty smart.) This can be done using L1 regularization and pruning small weights. This already gives two orders of magnitude of speedup, making it able to verify that there is no adversarial attack with ϵ = 0.1 on a particular MNIST digit in 11 seconds on average.

Next, they note that verification with linear constraints and functions is easy -- the challenging aspect is the Relu units that force the verifier to branch into two cases. (Since relu(x) = max(x, 0), it is the identity function when x is positive, and the zero function otherwise.) So why not try to ensure that the Relu units are also linear? Obviously we can't just make all the Relu units linear -- the whole point of them is to introduce nonlinearity to make the neural net more expressive. But as a start, we can look at the behavior of the Relu units on the examples we have, and if they are almost always active (inputs are positive) or almost always inactive (inputs are negative), then we replace them with the corresponding linear function (identity and zero, respectively), which is easier to verify. This gets another ~2x speedup.

But what if we could also change the training procedure? Maybe we could augment the loss so that the Relu units are either decisively active or decisively inactive on any dataset example. They propose that _during training_ we consider the L-infinity norm ball around each example, use that to create intervals that each pixel must be in, and then make a forward pass through the neural net using interval arithmetic (which is fast but inexact). Then, we add a term to the loss that incentivizes the interval for the input to each Relu to exclude zero (so that the Relu is either always active or always inactive). They call this the Relu Stability loss, or RS loss.

This leads to a further 4-13x speedup with similar test set accuracy. They then also test on MNIST with ϵ = 0.2, 0.3 and CIFAR with ϵ = 2/255, 8/255. It leads to speedup in all cases, with similar test set accuracy on MNIST but reduced accuracy on CIFAR. The provable accuracy goes up, but this is probably because when there's no RS loss, more images time out in verification, not because the network becomes better at classification. Other verification methods do get better provable accuracies on CIFAR, even though in principle they could fail to detect that a safe example is safe. This could be because their method times out frequently, or because their method degrades the neural net classifier -- it's hard to tell since they don't report number of timeouts.


It is pretty nice to have a program representation. This paper doesn't delve into specifics (besides a motivating example worked out by hand), but I'm excited to see an actual instantiation of this framework in the future!

Towards Mixed Optimization for Reinforcement Learning with Program Synthesis

Surya Bhupatiraju, Kumar Krishna Agrawal, Rishabh Singh

This paper proposes a framework in which policies are represented in two different ways -- as neural nets (the usual way) and as programs. To go from neural nets to programs, you use _program synthesis_ (as done by [VIPER](http://arxiv.org/abs/1805.08328) and [PIRL](https://arxiv.org/abs/1804.02477), both summarized in previous newsletters). To go from programs to neural nets, you use _distillation_ (basically use the program to train the neural net with supervised training). Given these transformations, you can then work with the policy in either space. For example, you could optimize the policy in both spaces, using standard gradient descent in neural-net-space, and _program repair_ in program-space. Having a program representation can be helpful in other ways too, as it makes the policy more interpretable, and more amenable to formal verification of safety properties.


While the approach is straightforward and will certainly increase robustness in many cases, it seems worth mentioning two serious issues. First, they assume that the initial DQN training learns the perfect Q function. Second, the provided certificates are about individual actions, not policy performance: the Q-values approximated in DQN assume optimal performance starting from the next action, which is not a given here. I am a bit concerned that these limitations were not really discussed, while the paper claims that “the resulting policy comes with a certificate of solution quality”.

Certified Adversarial Robustness for Deep Reinforcement Learning

Michael Everett*, Bjorn Lutjens*, Jonathan P. How

<@Certified adversarial robustness@>(@Certified Defenses against Adversarial Examples@) provides guarantees about the effects of small perturbations on a neural network's outputs. This paper uses that approach to make reinforcement learning more robust by training a DQN and acting by choosing the action with the best worst-case Q-value under adversarial perturbations (called the robust-optimal action) estimated from the certificate bounds, instead of the action with the highest Q-value.

The approach is evaluated on Cartpole and a navigation task that requires avoiding collisions, with an adversary perturbing observations in both cases. For small perturbations, this technique actually increases performance, but as perturbations get large the agent's conservatism can lead to a large degradation in performance.


Formal Methods for the Informal Engineer: Workshop Recommendations

Gopal Sarma, James Koppel, Gregory Malecha, Patrick Schultz, Eric Drexler, Ramana Kumar, Cody Roux, Philip Zucker

This is the writeup from the <@Formal Methods for the Informal Engineer@> workshop. The main thrust is a call for increased application of formal methods in order to increase confidence in critical AI/ML systems, especially in the life sciences. They provide five high-level recommendations for this purpose.


Verifiably Safe Exploration for End-to-End Reinforcement Learning

Nathan Hunt, Nathan Fulton, Sara Magliacane, Nghia Hoang, Subhro Das, Armando Solar-Lezama

As we saw in the highlight, applications of formal verification to reinforcement learning and safe exploration often rely on _shielding_, in which any proposed unsafe actions are replaced by randomly chosen safe actions. Typically, this requires having an MDP model in a high-level, symbolic state space, such as by defining the MDP over the Atari simulator state, rather than learning from pixels.

This paper demonstrates that we can relax this requirement and learn policies on low-level observations, while still getting the safety guarantees of the shielding approach. The approach is simple: we define (manually) an abstract model of the environment, with a symbolic state space and dynamics model, and use this to create a shield as usual. Then, to learn the policy (which gets pixels as input), we use an object detector to transform the pixels into a symbolic state, and then use the shield if necessary to select which action to take. The authors show that as long as the error of the object detection step is low, the overall policy learning will remain safe.


Many people believe that ultimately we will need to prove theorems about the safety of our AIs. I don't understand yet what kind of theorems they have in mind, so I don't really want to speculate on how this relates to it. It does seem like the robustness guarantee is the most relevant one, since in general we won't have access to a perfect model of the dynamics.

Verifiable Reinforcement Learning via Policy Extraction

Osbert Bastani, Yewen Pu, Armando Solar-Lezama

Since it is hard to verify properties of neural nets, we can instead first train a decision tree policy to mimic the policy learned by deep RL, and then verify properties about that. The authors generalize [DAGGER](https://www.cs.cmu.edu/~sross1/publications/Ross-AIStats11-NoRegret.pdf) to take advantage of the Q-function and extract decision tree policies. They then prove a correctness guarantee for a toy version of Pong (where the dynamics are known), a robustness guarantee for Pong (with symbolic states, not pixels) (which can be done without known dynamics), and stability of cartpole.


A Geometric Perspective on the Transferability of Adversarial Directions

Zachary Charles, Harrison Rosenberg, Dimitris Papailiopoulos


On the Sensitivity of Adversarial Robustness to Input Data Distributions

Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, Ruitong Huang


International Conference on Learning Representations, 2019.

Are adversarial examples inevitable?

Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, Tom Goldstein


AAAI-2019 Workshop on Engineering Dependable and Secure Machine Learning Systems

Adversarial Reprogramming of Sequence Classification Neural Networks

Paarth Neekhara, Shehzeen Hussain, Shlomo Dubnov, Farinaz Koushanfar


Fortified Networks: Improving the Robustness of Deep Networks by Modeling the Manifold of Hidden Representations

Alex Lamb, Jonathan Binas, Anirudh Goyal, Dmitriy Serdyuk, Sandeep Subramanian, Ioannis Mitliagkas, Yoshua Bengio


Evaluating and Understanding the Robustness of Adversarial Logit Pairing

Logan Engstrom, Andrew Ilyas, Anish Athalye


Adversarial Reprogramming of Neural Networks

Gamaleldin F. Elsayed, Ian Goodfellow, Jascha Sohl-Dickstein


Defense Against the Dark Arts: An overview of adversarial example security research and future research directions


On Evaluating Adversarial Robustness

Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, Alexey Kurakin


Adversarial Logit Pairing

Harini Kannan, Alexey Kurakin, Ian Goodfellow


Learning to write programs that generate images

S M Ali Eslami, Tejas Kulkarni, Oriol Vinyals, Yaroslav Ganin


Intrinsic Geometric Vulnerability of High-Dimensional Artificial Intelligence


On Adversarial Examples for Character-Level Neural Machine Translation


Idealised Bayesian Neural Networks Cannot Have Adversarial Examples: Theoretical and Empirical Study


CDT=EDT=UDT


Two Alternatives to Logical Counterfactuals


Clarifying Logical Counterfactuals


CDT Dutch Book


Anthropic paradoxes transposed into Anthropic Decision Theory


Anthropic probabilities and cost functions


Bounded Oracle Induction


A Rationality Condition for CDT Is That It Equal EDT (Part 2)


A Rationality Condition for CDT Is That It Equal EDT (Part 1)


EDT solves 5 and 10 with conditional oracles


Asymptotic Decision Theory (Improved Writeup)


Counterfactuals and reflective oracles


Cooperative Oracles


A critical agential account of free will, causation, and physics


Approval-directed agency and the decision theory of Newcomb-like problems


No surjection onto function space for manifold X


Failures of UDT-AIXI, Part 1: Improper Randomizing


Oracle Induction Proofs


Dimensional regret without resets


What are Universal Inductors, Again?


When EDT=CDT, ADT Does Well


Ramsey and Joyce on deliberation and prediction


Subjective implication decision theory in critical agentialism


Doubts about Updatelessness


Computing an exact quantilal policy


Logical Counterfactuals & the Cooperation Game


No Constant Distribution Can be a Logical Inductor


Resource-Limited Reflective Oracles


Idea: OpenAI Gym environments where the AI is a part of the environment


Probabilistic Tiling (Preliminary Attempt)


[Logical Counterfactuals for Perfect Predictors](https://www.lesswrong.com/posts/AKkFh3zKGzcYBiPo7/logical-counterfactuals-for-perfect-predictors) and [A Short Note on UDT](https://www.lesswrong.com/posts/EXtzy3v4soZcoZjuH/a-short-note-on-udt)


The Evil Genie Puzzle


Logical uncertainty and Mathematical uncertainty


Forecasting using incomplete models


A Possible Loophole for Self-Applicative Soundness?


Logical Inductor Lemmas


Logical Inductor Tiling and Why it's Hard


Logical Inductors Converge to Correlated Equilibria (Kinda)


Two Notions of Best Response


Quantilal control for finite MDPs


Distributed Cooperation


Abstraction Learning


A Model for General Intelligence


Fractal AI: A fragile theory of intelligence

Sergio Hernandez Cerezo,
Guillem Duran Ballester


Believable Promises


The Foundations of Deep Learning with a Path Towards General Intelligence


Deciphering China's AI Dream


France, China, and the EU All Have an AI Strategy. Shouldn’t the US?


AI policy careers in the EU


Special Report: AI Policy and China – Realities of State-Led Development


FLI Podcast: Beyond the Arms Race Narrative: AI and China


Where are people thinking and talking about global coordination for AI safety?


FLI Podcast: Why Ban Lethal Autonomous Weapons?

Ariel Conn, Emilia Javorsky, Bonnie Docherty, Ray Acheson, and Rasha Abdul Rahim


Ethical and societal implications of algorithms, data, and artificial intelligence: a roadmap for research

Nuffield Foundation and Leverhulme Centre for the Future of Intelligence


Summary of the 2018 Department of Defense Artificial Intelligence Strategy


Bridging near- and long-term concerns about AI


Perspective in Frontiers in Computer Science

Robust Artificial Intelligence and Robust Human Organizations


Game Changers: AI Part III, AI and Public Policy


On the (In)Applicability of Corporate Rights Cases to Digital Minds


China’s Grand AI Ambitions with Jeff Ding


The role of corporations in addressing AI’s ethical dilemmas


Superintelligence Skepticism as a Political Tool


The Advent of Huang's Law


The Deep Roots and Long Branches of Chinese Technonationalism


The Facets of Artificial Intelligence: A Framework to Track the Evolution of AI

Fernando Martınez-Plumed, Bao Sheng Loe, Peter Flach, Sean ́Oh ́Eigeartaigh, Karina Vold and Jose Hernandez-Orallo


AI Policy Challenges and Recommendations


State of AI


Accounting for the Neglected Dimensions of AI Progress

Fernando Martinez-Plumed, Shahar Avin, Miles Brundage, Allan Dafoe, Sean O


AI Algorithms Need FDA-Style Drug Trials

Olaf J. Groth, Mark J. Nitzberg, Stuart J. Russell


Advice to UN High-level Panel on Digital Cooperation

Luke Kemp, Peter Cihon,Matthijs Michiel Maas, Haydn Belfield,Dr Seán Ó hÉigeartaigh, Jade Leung, and Zoe Cremer


Surveying Safety-relevant AI Characteristics

Jose Hernandez-Orallo, Fernando Martınez-Plumed, Shahar Avin, Seán Ó hÉigeartaigh


FLI Signs Safe Face Pledge


Handful of Countries – Including the US and Russia – Hamper Discussions to Ban Killer Robots at UN


Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning


How Europe might matter for AI governance


To stay ahead of Chinese AI, senators want new commission


AI in the UK: ready, willing and able?


EU Member States sign up to cooperate on Artificial Intelligence


Podcast: Six Experts Explain the Killer Robots Debate

Paul Scharre, Toby Walsh, Richard Moyes, Mary Wareham, Bonnie Docherty, Peter Asaro, and Ariel Conn


Narrow AI Nanny: Reaching Strategic Advantage via Narrow AI to Prevent Creation of the Dangerous Superintelligence


Joint Artificial Intelligence Center Created Under DoD CIO


Artificial Intelligence and International Affairs: Disruption Anticipated


India's National Strategy for Artificial Intelligence


Tianjin city in China eyes US$16 billion fund for AI work, dwarfing EU’s plan to spend US$1.78 billion


France's AI strategy

See [Import AI's summary](https://jack-clark.net/2018/04/02/importai-88-nato-designs-a-cyber-defense-ai-object-detection-improves-with-yolov3-france-unveils-its-national-ai-strategy/).


Neural-Augmented Static Analysis of Android Communication

Jinman Zhao, Aws Albarghouthi, Vaibhav Rastogi, Somesh Jha, Damien Octeau


LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations

Michael Schaarschmidt, Alexander Kuhnle, Ben Ellis, Kai Fricke, Felix Gessert, Eiko Yoneki


Applying Deep Learning To Airbnb Search


Learning Scheduling Algorithms for Data Processing Clusters

Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, Mohammad Alizadeh


Predicted Variables in Programming

Victor Carbune, Thierry Coppey, Alexander Daryin, Thomas Deselaers, Nikhil Sarda, Jay Yagnik


The 30-Year Cycle In The AI Debate


Conversation with Ernie Davis


A Comprehensive Survey on Graph Neural Networks

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, Philip S. Yu


Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing


Large Scale GAN Training for High Fidelity Natural Image Synthesis

Andrew Brock, Jeff Donahue, Karen Simonyan


The Unreasonable Effectiveness of Deep Learning


The Lottery Ticket Hypothesis: Training Pruned Neural Networks


Dynamic Control Flow in Large-Scale Machine Learning

Yuan Yu, Martín Abadi, Paul Barham, Eugene Brevdo, Mike Burrows, Andy Davis, Jeff Dean, Sanjay Ghemawat, Tim Harley, Peter Hawkins, Michael Isard, Manjunath Kudlur, Rajat Monga, Derek Murray, Xiaoqiang Zheng


The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation

Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, Macduff Hughes


Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions

Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, Roger Grosse


Random Search and Reproducibility for Neural Architecture Search


Attentive Neural Processes

Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, Yee Whye Teh


Graph Neural Networks: A Review of Methods and Applications

Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, Maosong Sun


Phrase-Based & Neural Unsupervised Machine Translation

Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato


Realistic Evaluation of Deep Semi-Supervised Learning Algorithms

Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, Ian J. Goodfellow


This method uses spatial summarization for increasing convnet accuracy and was discovered around the same time as [this similar work](http://openaccess.thecvf.com/content_ECCV_2018/papers/Sanghyun_Woo_Convolutional_Block_Attention_ECCV_2018_paper.pdf). Papers with independent rediscoveries tend to be worth taking more seriously.

Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks

Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Andrea Vedaldi


Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations

Xander Steenbrugge, Sam Leroux, Tim Verbelen, Bart Dhoedt


Learning Concepts with Energy Functions


Introducing AdaNet: Fast and Flexible AutoML with Learning Guarantees


Relational inductive bias for physical construction in humans and machines

Jessica B. Hamrick, Kelsey R. Allen, Victor Bapst, Tina Zhu, Kevin R. McKee, Joshua B. Tenenbaum, Peter W. Battaglia


Large scale distributed neural network training through online distillation

Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, Geoffrey E. Hinton


Transfer Learning for Estimating Causal Effects using Neural Networks

Sören R. Künzel, Bradly C. Stadie, Nikita Vemuri, Varsha Ramakrishnan, Jasjeet S. Sekhon, Pieter Abbeel


Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures

Gongbo Tang, Mathias Müller, Annette Rios, Rico Sennrich


Analyzing Inverse Problems with Invertible Neural Networks

Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W. Pellegrini, Ralf S. Klessen, Lena Maier-Hein, Carsten Rother, Ullrich Köthe


Proceedings of the Artificial Life Conference 2018

Neural Architecture Search: A Survey

Thomas Elsken, Jan Hendrik Metzen, Frank Hutter


Spherical CNNs

Taco S. Cohen, Mario Geiger, Jonas Koehler, Max Welling


Neural Arithmetic Logic Units

Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, Phil Blunsom


TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing


Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

Arber Zela, Aaron Klein, Stefan Falkner, Frank Hutter


ICML workshop Neural Abstract Machines & Program Induction v2 2018

Automatically Composing Representation Transformations as a Means for Generalization

Michael B. Chang, Abhishek Gupta, Sergey Levine, Thomas L. Griffiths


Universal Transformers

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser


An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, Jason Yosinski


NLP's ImageNet moment has arrived


DARTS: Differentiable Architecture Search

Hanxiao Liu, Karen Simonyan, Yiming Yang


Resource-Efficient Neural Architect

Yanqi Zhou, Siavash Ebrahimi, Sercan Ö. Arık, Haonan Yu, Hairong Liu, Greg Diamos


Improving Language Understanding with Unsupervised Learning


Neural scene representation and rendering


Recent Progress in the Theory of Neural Networks


Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey


2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Learned optimizers that outperform SGD on wall-clock and test loss


Automatic Program Synthesis of Long Programs with a Learned Garbage Collector


Neural Network Quine


The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches

Md Zahangir Alom, Tarek M. Taha, Christopher Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian C Van Esesn, Abdul A S. Awwal, Vijayan K. Asari


Exploring the Limits of Weakly Supervised Pretraining

Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, Laurens van der Maaten


Capsules for Object Segmentation


Skill Rating for Generative Models

Catherine Olsson, Surya Bhupatiraju, Tom Brown, Augustus Odena, Ian Goodfellow


Learning Unsupervised Learning Rules

Luke Metz, Niru Maheswaranathan, Brian Cheung, Jascha Sohl-Dickstein


Generalization Error in Deep Learning

Daniel Jakubovitz, Raja Giryes, Miguel R. D. Rodrigues


Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990-2010)


InfoBot: Transfer and Exploration via the Information Bottleneck

Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, Sergey Levine


Amplifying the Imitation Effect for Reinforcement Learning of UCAV's Mission Execution


Contingency-Aware Exploration in Reinforcement Learning

Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, Honglak Lee


EMI: Exploration with Mutual Information Maximizing State and Action Embeddings

Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, Hyun Oh Song


The Dreaming Variational Autoencoder for Reinforcement Learning Environments

Per-Arne Andersen, Morten Goodwin, Ole-Christoffer Granmo


Learning Exploration Policies for Navigation


World Discovery Models

Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo Avila Pires, Jean-Bastien Grill, Florent Altché, Rémi Munos


Prioritizing Starting States for Reinforcement Learning

Arash Tavakoli, Vitaly Levdik, Riashat Islam, Petar Kormushev


Successor Uncertainties: exploration and uncertainty in temporal difference learning

David Janz, Jiri Hron, José Miguel Hernández-Lobato, Katja Hofmann, Sebastian Tschiatschek


Generative Exploration and Exploitation


Deep Reinforcement Learning with Feedback-based Exploration

Jan Scholten, Daan Wout, Carlos Celemin, Jens Kober


Identifying and Correcting Label Bias in Machine Learning


Path-Specific Counterfactual Fairness


Learning Not to Learn: Training Deep Neural Networks with Biased Data

Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, Junmo Kim


FAT* '19 Proceedings of the Conference on Fairness, Accountability, and Transparency

50 Years of Test (Un)fairness: Lessons for Machine Learning


Providing Gender-Specific Translations in Google Translate


AI Reading List


AI Governance Career Paths for Europeans

Exactly what it sounds like.


FLI Podcast: On Superforecasting


Forecasting Transformative AI: An Expert Survey

Ross Gruetzemacher, David Paradice, Kang Bok Lee


Time for AI to cross the human performance range in diabetic retinopathy


How rapidly are GPUs improving in price performance?


Jeff Hawkins on neuromorphic AGI within 20 years


[Will AI See Sudden Progress?](https://www.lesserwrong.com/posts/AJtfNyBsum6ZzWxKR/will-ai-see-sudden-progress), [Takeoff speeds](https://sideways-view.com/2018/02/24/takeoff-speeds/) and [The Hanson-Yudkowsky AI-Foom Debate](https://intelligence.org/ai-foom-debate/)

Double Cruxing the AI Foom debate


Multi-Agent Imitation Learning for Driving Simulation


Multiagent GAIL


Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradients

Arbaaz Khan, Clark Zhang, Daniel D. Lee, Vijay Kumar, Alejandro Ribeiro


Learning to Teach in Cooperative Multiagent Reinforcement Learning

Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, Jonathan P. How


On the Convergence of Competitive, Multi-Agent Gradient-Based Learning


Learning Social Conventions in Markov Games


CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning

Jiachen Yang, Alireza Nakhaei, David Isele, Hongyuan Zha, Kikuo Fujimura


Generative Multi-Agent Behavioral Cloning


A multi-agent reinforcement learning model of common-pool resource appropriation

Inequity aversion resolves intertemporal social dilemmas

Edward Hughes, Joel Z. Leibo, Matthew G. Phillips, Karl Tuyls, Edgar A. Duéñez-Guzmán, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin R. McKee, Raphael Koster, Heather Roff, Thore Graepel


M^3RL: Mind-aware Multi-agent Management Reinforcement Learning


Negative Update Intervals in Deep Multi-Agent Reinforcement Learning

Gregory Palmer, Rahul Savani, Karl Tuyls


Emergent Communication through Negotiation

Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z Leibo, Karl Tuyls, Stephen Clark


Conference on Robot Learning (CoRL) 2018

Learning to Share and Hide Intentions using Information Regularization

DJ Strouse, Max Kleiman-Weiner, Josh Tenenbaum, Matt Botvinick, David Schwab


Adaptive Mechanism Design: Learning to Promote Cooperation

Tobias Baumann, Thore Graepel, John Shawe-Taylor


1st Workshop on Goal Specifications for Reinforcement Learning

Multi-Agent Deep Reinforcement Learning with Human Strategies

Thanh Nguyen, Ngoc Duy Nguyen, Saeid Nahavandi


Coordination-driven learning in multi-agent problem spaces

Sean L. Barton, Nicholas R. Waytowich, Derrik E. Asher


Multi-winner Voting: a question of Alignment


Data-Efficient Hierarchical Reinforcement Learning

Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine


Hierarchical Reinforcement Learning with Deep Nested Agents


Hierarchical Reinforcement Learning with Hindsight


Model Primitive Hierarchical Lifelong Reinforcement Learning

Bohan Wu, Jayesh K. Gupta, Mykel J. Kochenderfer


Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization


Hierarchical visuomotor control of humanoids

Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Greg Wayne


Unsupervised Discovery of Decision States for Transfer in Reinforcement Learning

Nirbhay Modhe, Prithvijit Chattopadhyay, Mohit Sharma, Abhishek Das, Devi Parikh, Dhruv Batra, Ramakrishna Vedantam


Neural Modular Control for Embodied Question Answering

Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra


SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Maithra Raghu, Justin Gilmer, Jason Yosinski, Jascha Sohl-Dickstein


Deeper Interpretability of Deep Networks

Tian Xu, Jiayu Zhan, Oliver G.B. Garrod, Philip H.S. Torr, Song-Chun Zhu, Robin A.A. Ince, Philippe G. Schyns


GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, Antonio Torralba


Understanding deep learning through neuron deletion


Unsupervised Learning of Neural Networks to Explain Neural Networks

Quanshi Zhang, Yu Yang, Yuchen Liu, Ying Nian Wu, Song-Chun Zhu


Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations

Andrew Slavin Ross, Michael C. Hughes, Finale Doshi-Velez


Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, Jeff Clune


Regularizing Black-box Models for Improved Interpretability

Gregory Plumb, Maruan Al-Shedivat, Eric Xing, Ameet Talwalkar


Please Stop Explaining Black Box Models for High Stakes Decisions


Representer Point Selection for Explaining Deep Neural Networks

Chih-Kuan Yeh, Joon Sik Kim, Ian E.H. Yen, Pradeep Ravikumar


Towards Governing Agent's Efficacy: Action-Conditional β-VAE for Deep Transparent Reinforcement Learning

John Yang, Gyujeong Lee, Minsung Hyun, Simyung Chang, Nojun Kwak


Explaining Explanations in AI

Brent Mittelstadt, Chris Russell, Sandra Wachter


Training Machine Learning Models by Regularizing their Explanations


Towards Better Interpretability in Deep Q-Networks


Transparency and Explanation in Deep Reinforcement Learning Neural Networks

Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, Katia Sycara


Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, Rory Sayres


Measuring abstract reasoning in neural networks

David G. T. Barrett, Felix Hill, Adam Santoro,
Ari Morcos, and Timothy Lillicrap


AIES '18 Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society

Towards Robust Interpretability with Self-Explaining Neural Networks


Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning

Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, Lalana Kagal


Neural Stethoscopes: Unifying Analytic, Auxiliary and Adversarial Network Probing

Fabian B. Fuchs, Oliver Groth, Adam R. Kosiorek, Alex Bewley, Markus Wulfmeier, Andrea Vedaldi, Ingmar Posner


Personalized explanation in machine learning


Sanity Checks for Saliency Maps

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim


Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning


2018 ICML Workshop on Human Interpretability in Machine Learning

Intelligible Artificial Intelligence


Counterfactual States for Atari Agents via Generative Deep Learning

Matthew L. Olson, Lawrence Neal, Fuxin Li, Weng-Keen Wong


Interpretable Reinforcement Learning with Ensemble Methods


The What, the Why, and the How of Artificial Explanations in Automated Decision-Making


IEEE International Conference on Robotics and Automation (ICRA) 2019

Learning Explanations from Language Data

David Harbecke, Robert Schwarzenberg, Christoph Alt


This looks like that: deep learning for interpretable image recognition

Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, Cynthia Rudin


Interpretable Discovery in Large Image Data Sets


Open the Black Box Data-Driven Explanation of Black Box Decision Systems

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Luca Pappalardo, Salvatore Ruggieri, Franco Turini


How Can Neural Network Similarity Help Us Understand Training and Generalization?

Maithra Raghu, Google Brain Team and Ari S. Morcos, DeepMind


How does iterated amplification exceed human abilities?


What are the differences between all the iterative/recursive approaches to AI alignment?


Problems with Amplification/Distillation


Thoughts on "AI safety via debate"


Amplification Discussion Notes


My confusions with Paul's Agenda


Unsupervised Visuomotor Control through Distributional Planning Networks

Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, Chelsea Finn


Learning Task Specifications from Demonstrations

Marcell Vazquez-Chanlatte, Susmit Jha, Ashish Tiwari, Mark K. Ho, Sanjit A. Seshia


Machine Theory of Mind

Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S.M. Ali Eslami, Matthew Botvinick


Policy Shaping: Integrating Human Feedback with Reinforcement Learning

Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L. Isbell and  Andrea L. Thomaz


Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications


Maximum Causal Tsallis Entropy Imitation Learning

Kyungjae Lee, Sungjoon Choi, Songhwai Oh


Planning to Give Information in Partially Observed Domains with a Learned Weighted Entropy Model

Rohan Chitnis, Leslie Pack Kaelbling, Tomás Lozano-Pérez


SafeAI 2019: Proceedings of the AAAI Workshop on Artificial Intelligence Safety 2019

Imitation Learning from Imperfect Demonstration

Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, Masashi Sugiyama


Learning User Preferences via Reinforcement Learning with Spatial Interface Valuing


One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks

Tianhe Yu, Pieter Abbeel, Sergey Levine, Chelsea Finn


International Conference on Innovative Techniques and Applications of Artificial Intelligence 2018

Task-Embedded Control Networks for Few-Shot Imitation Learning

Stephen James, Michael Bloesch, Andrew J. Davison


Few-Shot Goal Inference for Visuomotor Learning and Planning

Annie Xie, Avi Singh, Sergey Levine, Chelsea Finn


Video Imitation GAN: Learning control policies by imitating raw videos using generative adversarial reward estimation

Subhajit Chaudhury, Daiki Kimura, Tu-Hoa Pham, Asim Munawar, Ryuki Tachibana


Proceedings of the AAAI Workshop on Artificial Intelligence Safety 201

Inspiration Learning through Preferences


Sample-Efficient Imitation Learning via Generative Adversarial Nets


Entropic Regret I: Deterministic MDPs


Generative Adversarial Imitation from Observation

Faraz Torabi, Garrett Warnell, Peter Stone


Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning

Xinlei Pan, Eshed Ohn-Bar, Nicholas Rhinehart, Yan Xu, Yilin Shen, Kris M. Kitani


Multi-agent Inverse Reinforcement Learning for General-sum Stochastic Games

Xiaomin Lin, Stephen C. Adams, Peter A. Beling


Regret-based Reward Elicitation for Markov Decision Processes


Robust Online Optimization of Reward-uncertain MDPs


Parenting: Safe Reinforcement Learning from Human Input


Using Causal Analysis to Learn Specifications from Task Demonstrations

Daniel Angelov, Yordan Hristov, Subramanian Ramamoorthy


ReNeg and Backseat Driver: Learning from Demonstration with Continuous Human Feedback

Jacob Beck, Zoe Papakipos, Michael Littman


Reinforcement learning and inverse reinforcement learning with system 1 and system 2


Efficiently Combining Human Demonstrations and Interventions for Safe Training of Autonomous Systems in Real-Time

Vinicius G. Goecks, Gregory M. Gremillion, Vernon J. Lawhern, John Valasek, Nicholas R. Waytowich


Inverse reinforcement learning for video games

Aaron Tucker, Adam Gleave, Stuart Russell


One-Shot Observation Learning

Leo Pauly, Wisdom C. Agboh, Mohamed Abdellatif, David C. Hogg, Raul Fuentes


Using lying to detect human values


Safe Policy Learning from Observations


IRL in General Environments


Training human models is an unsolved problem


ProLoNets: Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning


Zero-Shot Visual Imitation

Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, Trevor Darrell


A Roadmap for the Value-Loading Problem


AAAI FSS-18: Artificial Intelligence in Government and Public Sector

Shared Multi-Task Imitation Learning for Indoor Self-Navigation

Junhong Xu, Qiwei Liu, Hanqing Guo, Aaron Kageza, Saeed AlQarni, Shaoen Wu


Interpretable Latent Spaces for Learning from Demonstration

Yordan Hristov, Alex Lascarides, Subramanian Ramamoorthy


Learning Cognitive Models using Neural Networks

Devendra Singh Chaplot, Christopher MacLellan, Ruslan Salakhutdinov, Kenneth Koedinger


Building Ethics into Artificial Intelligence

Han Yu, Zhiqi Shen, Chunyan Miao, Cyril Leung, Victor R. Lesser, Qiang Yang


Building Ethically Bounded AI


A developmentally-situated approach to teaching normative behavior to AI


On the Convergence of Adam and Beyond

Sashank J. Reddi, Satyen Kale and Sanjiv Kumar


Formal Limitations on The Measurement of Mutual Information


Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, Hugo Larochelle


Online Meta-Learning

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, Sergey Levine


Task-Agnostic Meta-Learning for Few-shot Learning

Muhammad Abdullah Jamal, Guo-Jun Qi, Mubarak Shah


Meta-learning with differentiable closed-form solvers

Luca Bertinetto, João F. Henriques, Philip H.S. Torr, Andrea Vedaldi


CAML: Fast Context Adaptation via Meta-Learning

Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, Shimon Whiteson


Unsupervised Meta-Learning for Reinforcement Learning

Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, Sergey Levine


6th ICML Workshop on Automated Machine Learning (2019)

Alpha MAML: Adaptive Model-Agnostic Meta-Learning

Harkirat Singh Behl, Atılım Güneş Baydin, Philip H. S. Torr


Meta-Learning with Hessian Free Approach in Deep Neural Nets Training


Meta-Learning with Latent Embedding Optimization

Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, Raia Hadsell


Bayesian Model-Agnostic Meta-Learning

Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, Sungjin Ahn


BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova


PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation

Perttu Hämäläinen, Amin Babadi, Xiaoxiao Ma, Jaakko Lehtinen


How To Solve Moral Conundrums with Computability Theory


Podcast: Astronomical Future Suffering and Superintelligence with Kaj Sotala


Clarifying The Malignity of the Universal Prior: The Lexical Update


AI Alignment Podcast: On Consciousness, Qualia, and Meaning

Lucas Perry, Mike Johnson and Andrés Gómez Emilsson


To first order, moral realism and moral anti-realism are the same thing


AAMAS '18 Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems

Human-Centered Artificial Intelligence and Machine Learning


EuroSys '18 Proceedings of the Thirteenth EuroSys Conference, Article No. 18

Does Your Model Know the Digit 6 Is Not a Cat? A Less Biased Evaluation of "Outlier" Detectors

Alireza Shafaei, Mark Schmidt, James J. Little


Against the Linear Utility Hypothesis and the Leverage Penalty

More on the Linear Utility Hypothesis and the Leverage Prior


Beliefs at different timescales


Misbehaving AIs can't always be easily stopped!


Guide Me: Interacting with Deep Networks

Christian Rupprecht, Iro Laina, Nassir Navab, Gregory D. Hager, Federico Tombari


Wireheading is in the eye of the beholder


Building Safer AGI by introducing Artificial Stupidity


Sandboxing by Physical Simulation?


AAAI: AI and Society: Ethics, Safety and Trustworthiness in Intelligent Agents

Papers from AI and Society: Ethics, Safety and Trustworthiness in Intelligent Agents


The Blessings of Multiple Causes


Why the world’s leading AI charity decided to take billions from investors


WAISE 2018


Contest: $1,000 for good questions to ask to an Oracle AI


State of California Endorses Asilomar AI Principles


AI Strategy Project Manager


The AAAI's Workshop on Artificial Intelligence Safety


Learning-by-doing AI Safety workshop


Slides from Human-Level AI 2018


RAISE status report April-June 2018


AI Alignment Podcast: AI Safety, Possible Minds, and Simulated Worlds


Solving the AI Race finalists — $15,000 of prizes


Robust temporal difference learning for critical domains

Richard Klima, Daan Bloembergen, Michael Kaisers, Karl Tuyls


Safe Reinforcement Learning with Model Uncertainty Estimates

Björn Lütjens, Michael Everett, Jonathan P. How


Addressing three problems with counterfactual corrigibility: bad bets, defending against backstops, and overconfidence.


Optimization Regularization through Time Penalty


Stovepiping and Malicious Software: A Critical Review of AGI Containment


A Marauder's Map of Security and Privacy in Machine Learning


Secure Deep Learning Engineering: A Software Quality Assurance Perspective

Lei Ma, Felix Juefei-Xu, Minhui Xue, Qiang Hu, Sen Chen, Bo Li, Yang Liu, Jianjun Zhao, Jianxiong Yin, Simon See


The European Conference on Computer Vision (ECCV) Workshops, 2018

Are You Tampering With My Data?

Michele Alberti, Vinaychandran Pondenkandath, Marcel Würsch, Manuel Bouillon, Mathias Seuret, Rolf Ingold, Marcus Liwicki


The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, Dawn Song


The Secret Sharer: Measuring Unintended Neural Network Memorization & Extracting Secrets


Security and Privacy Issues in Deep Learning

Ho Bae, Jaehee Jang, Dahuin Jung, Hyemi Jang, Heonseok Ha, Sungroh Yoon


Classiﬁcation of global catastrophic risks connected with artiﬁcial intelligence


Predictors as Agents


Superintelligent messiahs are corrigible and probably misaligned


Emergence of Addictive Behaviors in Reinforcement Learning Agents

Vahid Behzadan, Roman V. Yampolskiy, Arslan Munir


Introducing PlaNet: A Deep Planning Network for Reinforcement Learning

Danijar Hafner, Student Researcher, Google AI


Soft Actor-Critic: Deep Reinforcement Learning for Robotics

Tuomas Haarnoja, Student Researcher and Sergey Levine, Faculty Advisor, Robotics at Google


International Conference on Artificial Intelligence in Education 2018

Learning Actionable Representations with Goal-Conditioned Policies


Learning to Explore with Meta-Policy Gradient


Simple random search provides a competitive approach to reinforcement learning


Thinking Fast and Slow with Deep Learning and Tree Search


Learning by Playing


Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning


Meta-Gradient Reinforcement Learning

Zhongwen Xu, Hado van Hasselt, David Silver


GECCO '18 Proceedings of the Genetic and Evolutionary Computation Conference

Evolutionary Reinforcement Learning


Learning Time-Sensitive Strategies in Space Fortress


Reward Estimation for Variance Reduction in Deep Reinforcement Learning

Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavet, Joelle Pineau


Smoothing Policies and Safe Policy Gradients

Matteo Papini, Matteo Pirotta, Marcello Restelli


Towards Symbolic Reinforcement Learning with Common Sense

Artur d'Avila Garcez, Aimore Resende Riquetti Dutra, Eduardo Alonso


Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future

Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, Dhruv Batra


Competitive Experience Replay

Hao Liu, Alexander Trott, Richard Socher, Caiming Xiong


Reward Shaping via Meta-Learning

Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, Jun Zhu


Learning to Generalize from Sparse and Underspecified Rewards

Rishabh Agarwal, Mohammad Norouzi, Chen Liang and Dale Schuurmans


A Geometric Perspective on Optimal Representations for Reinforcement Learning

Marc G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, Clare Lyle


The Value Function Polytope in Reinforcement Learning

Robert Dadashi, Adrien Ali Taïga, Nicolas Le Roux, Dale Schuurmans, Marc G. Bellemare


Visual Hindsight Experience Replay

Himanshu Sahni, Toby Buckley, Pieter Abbeel, Ilya Kuzovkin


Causal Reasoning from Meta-reinforcement Learning

Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, Zeb Kurth-Nelson


Self-supervised Learning of Image Embedding for Continuous Control

Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, Martin Riedmiller


A Study on Overfitting in Deep Reinforcement Learning

Chiyuan Zhang, Oriol Vinyals, Remi Munos, Samy Bengio


Soft Actor Critic—Deep Reinforcement Learning with Real-World Robots

Tuomas Haarnoja, Vitchyr Pong, Kristian Hartikainen, Aurick Zhou, Murtaza Dalal, and Sergey Levine


Foundations and TrendsÂ® in Machine Learning, Vol 11, Issue 3-4

An Introduction to Deep Reinforcement Learning

Vincent Francois-Lavet, Peter Henderson, Riashat Islam, Marc G. Bellemare, Joelle Pineau


Unsupervised Control Through Non-Parametric Discriminative Rewards


Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, Igor Mordatch


VIREL: A Variational Inference Framework for Reinforcement Learning

Matthew Fellows, Anuj Mahajan, Tim G. J. Rudner, Shimon Whiteson


Efficient Eligibility Traces for Deep Reinforcement Learning


CEM-RL: Combining evolutionary and gradient-based methods for policy search


Generalization and Regularization in DQN

Jesse Farebrother, Marlos C. Machado, Michael Bowling


Thirty-Third AAAI Conference On Artificial Intelligence, 2019

Combined Reinforcement Learning via Abstract Representations

Vincent François-Lavet, Yoshua Bengio, Doina Precup, Joelle Pineau


Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning


ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay


SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning

Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, Sergey Levine


Dexterous Manipulation with Reinforcement Learning: Efficient, General, and Low-Cost

Henry Zhu, Abhishek Gupta, Vikash Kumar, Aravind Rajeswaran, and Sergey Levine


Latent Space Policies for Hierarchical Reinforcement Learning

Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, Sergey Levine


ICML 2018 Notes


The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach

Iulian Vlad Serban, Chinnadhurai Sankar, Michael Pieper, Joelle Pineau, Yoshua Bengio


Thirty-Second AAAI Conference on Artificial Intelligence

Counterfactual Multi-Agent Policy Gradients

Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson


The Uncertainty Bellman Equation and Exploration

Brendan O'Donoghue, Ian Osband, Remi Munos, Volodymyr Mnih


Procedural Level Generation Improves Generality of Deep Reinforcement Learning

Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, Sebastian Risi


Ranked Reward: Enabling Self-Play Reinforcement Learning for Combinatorial Optimization

Alexandre Laterre, Yunguan Fu, Mohamed Khalil Jabri, Alain-Sam Cohen, David Kas, Karl Hajjar, Torbjorn S. Dahl, Amine Kerkeni, Karim Beguir


Annual Review of Control, Robotics, and Autonomous Systems 2019

A Tour of Reinforcement Learning: The View from Continuous Control


Evolving simple programs for playing Atari games

Dennis G Wilson, Sylvain Cussat-Blanc, Hervé Luga, Julian F Miller


RUDDER: Return Decomposition for Delayed Rewards

Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, Sepp Hochreiter


Investigating Generalisation in Continuous Deep Reinforcement Learning

Chenyang Zhao, Olivier Sigaud, Freek Stulp, Timothy M. Hospedales


Quantifying Generalization in Reinforcement Learning


Modular Architecture for StarCraft II with Deep Reinforcement Learning

Dennis Lee, Haoran Tang, Jeffrey O Zhang, Huazhe Xu, Trevor Darrell, Pieter Abbeel


Learning Shared Dynamics with Meta-World Models


Learning and Planning with a Semantic Model

Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, Yuandong Tian


Introducing Holodeck


Hierarchical Deep Multiagent Reinforcement Learning

Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Changjie Fan, Li Wang


On Reinforcement Learning for Full-length Game of StarCraft

Zhen-Jia Pang, Ruo-Ze Liu, Zhou-Yu Meng, Yi Zhang, Yang Yu, Tong Lu


Zero-shot Sim-to-Real Transfer with Modular Priors

Robert Lee, Serena Mou, Vibhavari Dasagi, Jake Bruce, Jürgen Leitner, Niko Sünderhauf


Automata Guided Reinforcement Learning With Demonstrations


Sim-to-Real Transfer Learning using Robustified Controllers in Robotic Tasks involving Complex Dynamics

Jeroen van Baar, Alan Sullivan, Radu Cordorel, Devesh Jha, Diego Romeres, Daniel Nikovski


Keep it stupid simple

Erik J Peterson, Necati Alp Müyesser, Timothy Verstynen, Kyle Dunovan


Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning

Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J. Mankowitz, Shie Mannor


World Models


Active Reinforcement Learning with Monte-Carlo Tree Search


Hierarchical Reinforcement Learning: Approximating Optimal Discounted TSP Using Local Policies


Learning Memory Access Patterns


Feedback-Based Tree Search for Reinforcement Learning

Daniel R. Jiang, Emmanuel Ekwedike, Han Liu


Episodic Memory Deep Q-Networks

Zichuan Lin, Tianqi Zhao, Guangwen Yang, Lintao Zhang


Learning Real-World Robot Policies by Dreaming

AJ Piergiovanni, Alan Wu, Michael S. Ryoo


On Learning Intrinsic Rewards for Policy Gradient Methods


USC CLASS Research Papers Series No. CLASS18-10

Improving On-policy Learning with Statistical Reward Accumulation

Yubin Deng, Ke Yu, Dahua Lin, Xiaoou Tang, Chen Change Loy


ViZDoom Competitions: Playing Doom from Pixels

Marek Wydmuch, Michał Kempka, Wojciech Jaśkowski


ExpIt-OOS: Towards Learning from Planning in Imperfect Information Games


Remember and Forget for Experience Replay


Accuracy-based Curriculum Learning in Deep Reinforcement Learning

Pierre Fournier, Olivier Sigaud, Mohamed Chetouani, Pierre-Yves Oudeyer


Adversarial Exploration Strategy for Self-Supervised Imitation Learning

Zhang-Wei Hong, Tsu-Jui Fu, Tzu-Yun Shann, Yi-Hsiang Chang, Chun-Yi Lee


Self-Imitation Learning

Junhyuk Oh, Yijie Guo, Satinder Singh, Honglak Lee


Learning to navigate in cities without a map

Piotr Mirowski, Raia Hadsell and Andrew Zisserman


By default, avoid ambiguous distant situations


AI Alignment Problem: “Human Values” don’t Actually Exist


Utility versus Reward function: partial equivalence


Defining the ways human values are messy


Bayesian Policy Optimization for Model Uncertainty

Gilwoo Lee, Brian Hou, Aditya Mandalika, Jeongseok Lee, Sanjiban Choudhury, Siddhartha S. Srinivasa


Reinforcement Learning with Perturbed Rewards


Push the Student to Learn Right: Progressive Gradient Correcting by Meta-learner on Corrupted Labels

Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, Deyu Meng


Learning Robust Representations by Projecting Superficial Statistics Out

Haohan Wang, Zexue He, Zachary C. Lipton, Eric P. Xing


[Open Category Detection with PAC Guarantees](http://proceedings.mlr.press/v80/liu18e/liu18e.pdf) is the corresponding paper.

Making AI Safe in an Unpredictable World: An Interview with Thomas G. Dietterich

Thomas G. Dietterich and Jolene Creighton


When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks


Adversarial Attacks Against Medical Deep Learning Systems

Samuel G. Finlayson, Hyung Won Chung, Isaac S. Kohane, Andrew L. Beam


39th IEEE Symposium on Security and Privacy

Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning

Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, Bo Li


Treacherous Turn, Simulations and Brain-Computer Interfaces


Evaluating Existing Approaches to AGI Alignment


Phenomenological AI Safety Research Institute

Robustness to fundamental uncertainty in AGI alignment


Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings

Aviral Kumar, Sunita Sarawagi and Ujjwal Jain


Section 4.3 of [this](https://openreview.net/pdf?id=HyxCxhRcY7) paper makes similar observations and ameliorates the issue. [This](https://arxiv.org/abs/1809.04729v1) paper also demonstrates the fragility of density estimators on out-of-distribution data.

Do Deep Generative Models Know What They Don't Know?

Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan


2018 EMNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

Learning deep representations by mutual information estimation and maximization

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio


Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies

Alessandro Achille, Tom Eccles, Loic Matthey, Christopher P. Burgess, Nick Watters, Alexander Lerchner, Irina Higgins


Robots that Learn to Use Improvised Tools

Annie Xie,  Frederik Ebert, Sergey Levine, Chelsea Finn


Toward an AI Physicist for Unsupervised Learning


Unsupervised Learning of Syntactic Structure with Invertible Neural Projections

Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick


Preface to the Sequence on Value Learning

This is a preface, read it if you're going to read the full posts, but not if you're only going to read these summaries.


Certified Adversarial Robustness via Randomized Smoothing

Jeremy M Cohen, Elan Rosenfeld, J. Zico Kolter


Training verified learners with learned verifiers

Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O'Donoghue, Jonathan Uesato, Pushmeet Kohli


Optimization + Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness

Greg Anderson, Shankara Pailoor, Isil Dillig, Swarat Chaudhuri


On a Formal Model of Safe and Scalable Self-driving Cars

Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua


On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models

Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, Pushmeet Kohli


Verification for Machine Learning, Autonomy, and Neural Networks Survey

Weiming Xiang, Patrick Musau, Ayana A. Wild, Diego Manzanas Lopez, Nathaniel Hamilton, Xiaodong Yang, Joel Rosenfeld, Taylor T. Johnson


A Game-Based Approximate Verification of Deep Neural Networks with Provable Guarantees

Min Wu, Matthew Wicker, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska