The article contains many useful examples in a grid world and I recommend reading it for a deeper understanding of the problem. The main problem with the attainable utility penalty seems to be the definition of power. While this has proven to be quite useful in simple examples, it seems like agents in more complex environments would be able to disentangle their ability to optimize their own reward and the auxiliary rewards, for example by essentially committing to never optimize the auxiliary rewards beyond a certain threshold. Such an agent would still look very powerful from the outside and I don't see why power-reducing commitments would diminish the agent's incentive to take away power from others. So while the [catastrophic convergence conjecture](https://www.lesswrong.com/posts/w6BtMqKRLxG9bNLMr/the-catastrophic-convergence-conjecture), which states that unaligned goals tend to lead to catastrophic optimal policies because of power-seeking incentives, still rings true, it seems like we need to look at power from our perspective instead of the agent's.

**Rohin's opinion:** I agree with Flo above: the issue is that AUP is measuring a proxy for our intuitive notion of power that falls apart under adversarial optimization. In particular, while it is normally reasonable to measure power by looking at the ability to optimize a set of auxiliary reward functions, this characterization no longer works when the agent can ensure that it won't be able to optimize those specific rewards, while still being able to optimize its primary reward. Subagents are a particularly clean way of demonstrating the problem.

Subagents and impact measures, full and fully illustrated

Impact measures reduce the reward an agent **A** receives for reaching a state **s** by the distance between **s** and some baseline state **s'**. This article clarifies the interaction of <@impact measures and subagents@>(@Attainable utility has a subagent problem@).

In the <@attainable utility framework@>(@Penalizing Impact via Attainable Utility Preservation@) the distance metric measures changes in **A**'s power, its ability to optimize a set of auxiliary rewards **R**, relative to doing nothing. This means that **A** is never penalized for doing nothing. Thus, if **A** manages to construct a subagent **SA** without getting penalized for that, **A** can just stop doing anything afterwards without getting a penalty, no matter what **SA** does.

For the *stepwise inaction baseline*, where **s'** is the state that **A** would have reached had it done nothing in the last time step, **A** can completely avoid the penalty by building an **SA** that commits to balance out the change in **A**'s ability to optimize the auxiliary reward **R** if **A** decided to optimize **R**, but instead becomes an optimizer for **A**'s real reward when **A** does nothing in the next step.

For the *inaction baseline*, where **s'** is the state reached if **A** had not done anything for the whole episode, it gets more complicated. When only gains in power are penalized, **SA** has to be built such that it does not cause a significant increase in **R**, even if **A** decided to optimize for **R** later on. If **SA** is constructed in a way such that **A** cannot control its behaviour after some time has passed, the penalty **A** receives for the construction only depends on the _actual_ effect of **SA** on **R**. This is a lot weaker than the original penalty, which was about the _possible_ effects of **A**'s actions on **R**.

In the case where decreases in power are also penalized, **A** needs to make sure that building **SA** does not reduce its ability to optimize **R**. This is not always possible without **A** retaining some control over **SA**, but giving up control was only necessary to prevent the construction of **SA** from increasing **A**'s potential to optimize **R**. It seems like it is often possible for **A** to finetune the amount of control it retains over **SA** such that **A**'s ability to optimize **R** barely changes while **SA** is able to do things **A** could not do unpenalized.

In the most extreme example, **A** boxes itself into **SA** and does nothing in reality, but is counterfactually able to intervene on **SA**'s policy. If **A** is able to cleverly put constraints on its own future actions in other ways, no subagents are needed to avoid the penalty. While other impact penalties like <@relative reachability@>(@Measuring and avoiding side effects using relative reachability@) do allow for subagents to be constructed, they do not undermine the penalty for these.


2019 NeurIPS Safety and Robustness in Decision Making Workshop

I am a big fan of benchmarks; they seem to be a prerequisite to making a lot of quantitative progress (as opposed to more conceptual progress, which seems more possible to do without benchmarks). This benchmark seems particularly nice to me because the "side effects" which need to be avoided haven't been handcoded into the benchmark, but instead arise from some simple rules that produce complex effects.

Paper: SafeLife 1.0: Exploring Side Effects in Complex Environments

Introducing SafeLife: Safety Benchmarks for Reinforcement Learning

So far, techniques to avoid negative side effects have only been tested on <@simple@>(@Measuring and avoiding side effects using relative reachability@) <@gridworlds@>(@Penalizing Impact via Attainable Utility Preservation@) <@or@>(@Learning Preferences by Looking at the World@) <@hypotheticals@>(@Test Cases for Impact Regularisation Methods@). SafeLife aims to provide a high-dimensional environment in which negative side effects are likely. It is based on Conway's Game of Life, which allows for complex effects arising out of relatively simple rules. An agent is given the ability to move, create life in an adjacent cell, or destroy life in an adjacent cell. With the specified reward function, the agent must build desired patterns, remove undesired patterns, and navigate to the exit.

The challenge comes when there are additional "neutral" patterns in the environment. In this case, we want the agent to leave those patterns alone, and not disrupt them, even if doing so would allow it to complete the main task faster. The post shows several examples of agents attempting these levels. Vanilla RL agents don't avoid side effects at all, and so unsurprisingly they do quite badly. An agent with a naive impact measure that simply says to preserve the initial state can correctly solve levels where all of the "neutral" patterns are static, but has much more trouble when the existing patterns are dynamic (i.e. they oscillate over time).


I am quite excited about safety gym. I expect that the crisp formalization, as well as the availability of benchmarks and ready-made environments, combined with OpenAI's prestige, will facilitate broader engagement of the ML community with this branch of safe exploration.  As pointed out in the paper, switching from standard to constrained RL could merely shift the burden of correct specification from the reward to the cost and it is not obvious whether that helps with alignment. Still, I am somewhat optimistic because it seems like humans often think in terms of constrained and fuzzy optimization problems rather than specific tradeoffs and constrained RL might capture our intuitions better than pure reward maximization. Lastly, I am curious whether an increased focus on constrained RL will provide us with more concrete examples of "nearest unblocked strategy" failures, as the rising popularity of RL arguably did with more general examples of specification gaming.

**Rohin's opinion:** Note that at initialization, the policy doesn't "know" about the constraints, and so it must violate constraints during exploration in order to figure out what the constraints even are. As a result, in this framework we could never get down to zero violations. A zero-violations guarantee would require some other source of information, typically some sort of overseer (see <@delegative RL@>(@Delegative Reinforcement Learning@), [avoiding catastrophes via human intervention](https://arxiv.org/abs/1707.05173), and [shielding](https://arxiv.org/abs/1708.08611)).

It's unclear to me how much this matters for long-term safety, though: usually I'm worried about an AI system that is plotting against us (because it has different goals than we do), as opposed to one that doesn't know what we don't want it to do.

Safety Gym

Safety gym contains a set of tasks with varying difficulty and complexity focused on safe exploration. In the tasks, one of three simulated robots has to move to a series of goals, push buttons or move a box to a target location, while avoiding costs incurred by hitting randomized obstacles. This is formalized as a **constrained reinforcement learning** problem: in addition to maximizing the received reward, agents also have to respect constraints on a **safety cost function**. For example, we would like self-driving cars to learn how to navigate from A to B as quickly as possible while respecting traffic regulations and safety standards. While this could in principle be solved by adding the safety cost as a penalty to the reward, constrained RL gets around the need to correctly quantify tradeoffs between safety and performance.

Measures of safety are expected to become important criteria for evaluating algorithms' performance and the paper provides first benchmarks. Constrained policy optimization, a trust-region algorithm that tries to prevent updates from breaking the constraint on the cost is compared to new lagrangian versions of TRPO/PPO that try to maximize the reward, minus an adaptive factor times the cost above the threshold. Interestingly, the lagrangian methods incur a lot less safety cost during training than CPO and satisfy constraints more reliably at evaluation. This comes at the cost of reduced reward. For some of the tasks, none of the tested algorithms is able to gain nontrivial rewards while also satisfying the constraints.

Lastly, the authors propose to use safety gym for investigating methods for learning cost functions from human inputs, which is important since misspecified costs could fail to prevent unsafe behaviour, and for transfer learning of constrained behaviour, which could help to deal with distributional shifts more safely.


I enjoyed this application of causal diagrams and think that similar detailed analyses of the interactions between failure modes like wireheading, instrumental goals like reward preservation and the specific implementation of an agent would be quite valuable. That said, I am less excited about the feasibility of the proposed solution since it seems to require detailed knowledge of the agent about counterfactual rewards. Also, I expect the distinction between changes in the reward signal and changes in the state that happen to also affect the reward to be very fuzzy in real problems and current-RF optimization seems to require a very sharp boundary.

**Rohin's opinion:** I agree with Flo's opinion above, and I think the example in the blog post shows how the concept "what affects the reward" is fuzzy: in their gridworld inspired by the game Baba Is You, they say that moving the word "Reward" down to make rocks rewarding is "tampering", whereas I would have called that a perfectly legitimate way to play given my knowledge of Baba Is You.

Paper: Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective

Designing agent incentives to avoid reward tampering

Tom Everitt, Ramana Kumar, and Marcus Hutter

Reward tampering occurs when a reinforcement learning agent actively changes its reward function. The post uses <@Causal Influence Diagrams@>(@Modeling AGI Safety Frameworks with Causal Influence Diagrams@) to analyze the problem in a simple grid world where an agent can easily change the definition of its reward. The proposed solution is **current-RF optimization**: Instead of maximizing the sum of rewards that would be given after each action (where the reward signal can dynamically change over time), the agent searches for and executes a plan of actions that would maximize the current, unchanged reward signal. The agent would then not be incentivized to tamper with the reward function since the current reward is not maximized by such tampering. There are two different flavours to this: time-inconsistency-aware agents account for future changes in their own behaviour due to modified reward signals, while TI-unaware agents ignore this in their planning. TI-aware agents have an incentive to preserve their reward signal and are therefore potentially incorrigible.


To quote myself from a discussion with Alex, "you're looking at the optimal Q-function for the optimal utility function and saying 'this is a good measure of what we care about' and of course I agree with that". (Although this is a bit inaccurate -- it's not the optimal Q-function, but the Q-function relative to what we expect and know.)

This may be somewhat of a surprise, given that I've been [pessimistic](https://www.alignmentforum.org/posts/kCY9dYGLoThC3aG7w/best-reasons-for-pessimism-about-impact-of-impact-measures#qAy66Wza8csAqWxiB) about impact measures in the past. However, my position is that it's difficult to simultaneously get three desiderata: value-agnosticism, avoidance of catastrophes, and usefulness. This characterization of impact is very explicitly dependent on values, and so doesn't run afoul of that. (Also, it just makes intuitive sense.)

This part of the sequence did change some of my thinking on impact measures as well. In particular, the sequence makes a distinction between _objective_ impact, which applies to all (or most) agents, and _value_ impact. This is similar to the idea of [convergent instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf), and the idea that <@large-scale multiagent training@>(@Emergent Tool Use from Multi-Agent Interaction@) can lead to generally useful behaviors that can be applied to novel tasks. It seems plausible to me that we could make value-agnostic impact measures that primarily penalize this objective impact, and this might be enough to avoid catastrophes. This would prevent us from using AI for big, impactful tasks, but could allow for AI systems that pursue small, limited tasks. I suspect we'll see thoughts along these lines in the next parts of this sequence.

Reframing Impact - Part 1

_This sequence has exercises that **will be spoiled by this summary**, so take a moment to consider whether you want to read the sequence directly._

This first part of the sequence focuses on identifying what we mean by impact, presumably to help design an impact measure in the future. The punch line: an event is **impactful to an agent** if it changes the agent's **ability to get what it wants**. This is _Attainable Utility (AU) theory_. To quote the sequence: "How could something possibly be a big deal to us if it _doesn't_ change our ability to get what we want? How could something _not_ matter to us if it _does_ change our ability to get what we want?"

Some implications and other ideas:
- Impact is _relative to an agent_: a new church is more impactful if you are a Christian than if not.
- Some impact is _objective_: getting money is impactful to almost any agent that knows what money is.
- Impact is _relative to expectations_: A burglar robbing your home is impactful to you (you weren't expecting it) but not very impactful to the burglar (who had planned it out). However, if the burglar was unsure if the burglary would be successful, than success/failure would be impactful to them.

While this may seem obvious, <@past work@>(@Measuring and avoiding side effects using relative reachability@) has talked about impact as being caused by changes in state. While of course any impact does involve a change in state, this is the wrong level of abstraction to reason about impact: fundamentally, impact is related to what we care about.


I think the idea of putting the operator in the box with the AI system is very interesting: with previous attempts at boxing, the human operator talking to the AI system was an obvious glaring hole in the box. In this setting, the only information escaping from the box is the fact that the operator has not yet chosen to end the episode.

I am generally skeptical of intuitive reasoning about what can or can't be done by Turing Machines using extreme amounts of computation. There are _lots_ of comments on the post that debate specifics of this. This usually cashes out as a debate about the assumptions in the proof. But it's also worth noting that the theorem is asymptotic, and allows for arbitrarily bad behavior early on. We might still expect good behavior early on for the reasons laid out in the proof, but it's not implied by the theorem, even if the assumptions hold.

Paper: Asymptotically Unambitious Artificial General Intelligence

Asymptotically Benign AGI

I'm a bit out of my depth with this summary, but let's give it a shot anyway. The setting: we are not worried about how much compute we use (except that it should be finite), and we would like to build a powerful AI system that can help us with tasks but does not try to influence the world. We'll assume that we can construct a box from which no signals can pass through, except by pressing a specific button that opens a door.

First, the simple version of BoMAI (Boxed Myopic AI). We'll put the AI system **and the operator** in the box, and the operator and the AI system can talk via text message, and the operator can enter rewards. Each episode has a maximum length (hence myopic), and if the operator ends the episode early, all future rewards are set to zero. BoMAI maximizes episodic reward in a manner similar to [AIXI](http://www.hutter1.net/ai/). It has a distribution (initially a speed prior) over all possible time-bounded Turing Machines as possible models that predict observations and rewards. BoMAI uses the maximum a posteriori (MAP) Turing Machine to predict future observations and rewards given actions, searches over all possible sequences of actions for the best one, and executes the first action of that sequence. (See [this comment](https://www.alignmentforum.org/posts/pZhDWxDmwzuSwLjou/asymptotically-benign-agi#ToDYj27wQZWHqWnH6) and its parents for the reason to use the MAP model.)

Intuitively, BoMAI has no incentive to affect anything outside the box: information can only leave the box if the episode ends, but if that happens, reward is guaranteed to be exactly zero. It might deceive the operator into giving it high reward, but any such deception will only _incidentally_ affect the outside world, it hasn't been _optimized_ to cause any such effect. However, **this is all assuming that the MAP world model is accurate**, which may not be true.

Obviously at initialization our world model will not be accurate: we need to explore. Rather than using something like epsilon-exploration, BoMAI delegates to a human explorer for safety, and reduces exploration over time.

More insidiously, a world model could simulate the world for a year, and then report the value in the computer's memory for "the reward from a year ago". This will usually be a perfect prediction of the reward the operator will give, but if BoMAI tries to _optimize_ for it, it has an incentive to try to e.g. cause the operator to build an unboxed AI whose goal is to change BoMAI's memory cell to the maximum value.

These sorts of bad plans can only happen finitely often. If BoMAI executes such a plan, it will then get the actual reward from the operator, see that it differs from the predicted reward, and update against the world model for making a bad prediction. Eventually the correct model will become more likely. But ideally this sort of thing wouldn't happen at all.

Intuitively, such world models would have to simulate the world past the end of the episode, incurring a speed penalty relative to the correct model. However, they could use that simulation to save on the computation needed for future predictions. To make sure that they do incur a speed penalty, we need to make it so that sometimes their simulations are wrong and can't be reused. So, at the end of each episode, we sample from a source of true randomness, and use that to determine whether or not the operator is given a drug that causes retrograde amnesia. In that case, world models that simulate the future will sometimes be wrong about how that decision plays out, and any simulation they did in the wrong branch is useless and leads to a speed penalty.

With a few non-trivial assumptions, we can prove that asymptotically, BoMAI will do at least as well as the human explorer at accumulating reward, and the MAP world model's rewards do not incentivize BoMAI to affect the outside world.


This is quite a big improvement for impact measures -- it meets many desiderata that weren't satisfied simultaneously before. My main critique is that it's not clear to me that an AUP-agent would be able to do anything useful. For example, perhaps the action used to define the impact unit is well-understood and accepted, but any other action makes humans a little bit more likely to turn off the agent. Then the agent won't be able to take those actions. Generally, I think that it's hard to satisfy the conjunction of three desiderata -- objectivity (no dependence on values), safety (preventing any catastrophic plans) and non-trivialness (the AI is still able to do some useful things). There's a lot more discussion in the comments.

Towards a New Impact Measure

This post introduces a new idea for an impact measure. It defines impact as change in our ability to achieve goals. So, to measure impact, we can simply measure how much easier or harder it is to achieve goals -- this gives us Attainable Utility Preservation (AUP). This will penalize actions that restrict our ability to reach particular outcomes (opportunity cost) as well as ones that enlarge them (instrumental convergence).

Alex then attempts to formalize this. For every action, the impact of that action is the absolute difference between attainable utility after the action, and attainable utility if the agent takes no action. Here, attainable utility is calculated as the sum of expected Q-values (over m steps) of every computable utility function (weighted by 2^{-length of description}). For a plan, we sum up the penalties for each action in the plan. (This is not entirely precise, but you'll have to read the post for the math.) We can then choose one canonical action, calculate its impact, and allow the agent to have impact equivalent to at most N of these actions.

He then shows some examples, both theoretical and empirical. The empirical ones are done on the suite of examples from AI safety gridworlds used to test relative reachability. Since the utility functions here are indicators for each possible state, AUP is penalizing changes in your ability to reach states. Since you can never increase the number of states you reach, you are penalizing decrease in ability to reach states, which is exactly what relative reachability does, so it's not surprising that it succeeds on the environments where relative reachability succeeded. It does have the additional feature of handling shutdowns, which relative reachability doesn't do.

Since changes in probability of shutdown drastically change the attainable utility, any such changes will be heavily penalized. We can use this dynamic to our advantage, for example by committing to shut down the agent if we see it doing something we disapprove of.


These seem like generally good desiderata, though I don't know how to formalize them to the point that we can actually check with reasonable certainty whether a proposed impact measure meets these desiderata.

I have one additional desideratum from impact measures. The impact measure alone should disallow all extinction scenarios, while still allowing the AI system to do most of the things we use AI for today. This is rather weak, really I'd want AI do more tasks than are done today. However, even in this weak form, I doubt that we can satisfy this desideratum if we must also be agnostic to values, goals, representations and environments. We could have valued human superiority at game-playing very highly, in which case building AlphaGo would be catastrophic. How can an impact measure allow that without being at least some knowledge about values?

Impact Measure Desiderata

This post gives a long list of desiderata that we might want an impact measure to satisfy. It considers the case where the impact measure is a second level of safety, that is supposed to protect us if we don't succeed at value alignment. This means that we want our impact measure to be agnostic to human values. We'd also like it to be agnostic to goals, environments, and representations of the environment. There are several other desiderata -- read the post for more details, my summary would just be repeating it.


This is a remarkable number of safety precautions, though in hindsight it makes total sense given how bad a failure could be. None of the precautions would stop a superintelligent agent in the classical sense (that is, the sort of superintelligent agent in paperclip maximizer stories), but they seem like a really good set of precautions for anything task-based. I am curious how they chose the threshold for when to discard actions that the AI is not confident enough in (especially since AI uncertainty estimates are typically not calibrated), and how they developed the safety rules for verification (since that is a form of specification, which is often easy to get wrong).

Safety-first AI for autonomous data centre cooling and industrial control

Two years ago, DeepMind built an AI recommendation system that provided suggestions on how best to cool Google's data centers, leading to efficiency gains. Nine months ago, the AI was given autonomous control to take actions directly, rather than going through human operators, and it has been improving ever since, going from 12% savings at deployment to 30% now.

Of course, such a system must be made extremely reliable, since a failure could result in Google's data centers going down. They implemented several safety measures. They throw out any actions that the AI is not confident about. All actions are verified against a set of hand-coded safety rules, both when the actions are generated in the cloud, and at each local data center, for reliability through redundancy. There are human operators monitoring the AI to make sure nothing goes wrong, who can take over control whenever they want to. There is also an automated system that will fall back to the original system of heuristics and rules if the safety conditions are ever violated.


[Avoiding Tampering Incentives in Deep RL via Decoupled Approval](https://arxiv.org/abs/2011.08827)

I like that this work has moved to the assumption that there is an advisor who is capable of providing feedback agnostic to tampering: the user's feedback is correct when the agent only _asks_ about a tampering action rather than executing it. The information about what is and isn't tampering has to come from somewhere, and the natural source is some human overseer. (Perhaps previous work was also written with this in mind, but if so it wasn't obvious to me from the papers.) Note that this is a pretty weak assumption -- it only means that the overseer needs to not actively reward any tampering behavior; they don't need to penalize it as well. For example, when the agent proposes that they rewrite the file, the overseer just needs to notice that rewriting the file doesn't help with being a good assistant, they don't also need to notice that if the rewriting were to happen then the agent would have tampered with the reward.

The current experiments operate in an episodic setting, meaning that tampering effects get undone by resetting after each episode. However, in many realistic cases, once the reward is tampered with, you wouldn't expect there to be a way to "reset" it, and so you are stuck with the tampering forever and won't learn the right policy. (<@Delegative Reinforcement Learning@> is an example of work which avoids the assumption for this reason.) This is probably fine if each tampering action has a small effect and you need many such actions to have a cumulatively bad effect. In this case, removing the incentive to tamper means that when the agent tampers via exploration, the tampering isn't reinforced, and so the agent won't tamper again in the future. However, if it is unacceptable to take even a single tampering action (as in the case of rewriting the source code for the reward function), then it doesn't help that after exploring that action we don't reinforce it, as the damage has already been done.

Another way to think about it is that current deep RL systems primarily learn from experience (rather than reasoning), which is why the benefit only kicks in after the agent has randomly explored a tampering action. However, if we build superintelligent AI systems, they will be doing some form of reasoning, and in that setting if we can remove the incentive to tamper with the reward, that may indeed prevent the agent from _ever_ tampering with the reward.

Ramana Kumar*, Jonathan Uesato*, Victoria Krakovna, Tom Everitt, Richard Ngo, Shane Legg

<@Current-RF optimization@>(@Designing agent incentives to avoid reward tampering@) shows that to avoid tampering with the reward, we can have an agent that evaluates plans it makes according to the current reward function, rather than the reward after tampering, and this is sufficient to remove any incentive for tampering. However, that work required the ability to evaluate actions and/or plans using the "current reward". How might we implement this in deep RL algorithms in practice?

Let's take a simple example: suppose for an autonomous personal assistant, every once in a while we query the user for their satisfaction, write it to a file, and then use that file to train the assistant. Then with normal RL the assistant is incentivized to "tamper" by rewriting the contents of the file to show maximal satisfaction. In this context, current-RF optimization would say that _before_ rewriting the contents, the agent should ask the user whether that's a good idea. However, we can't ask the user about every action, and our agent does need to take some actions in order to explore the environment.

The authors formalize this as a Corrupted Feedback MDP, in which the feedback which the agent gets is corrupted in some states. They assume that the human gives _approval_ feedback, which they formalize as the advantage function. (The advantage is the long-term value of the queried action relative to the average action for the current state.) This ensures that the agent only needs to myopically select the action with highest approval, which means we can run any old deep RL algorithm with the discount set to zero. However, this doesn't solve the problem, because with deep RL the feedback is given _after_ the action is executed, at which point it has already been corrupted (in our example, the file already claims the user is maximally satisfied).

To fix this, the authors introduce _decoupling_, in which the action executed by the agent and the action on which feedback is given are sampled independently. The idea is that even if the executed action leads to corruption, the corrupted update is equally likely to affect every action, and so in expectation it cancels out. (This requires the _uniform corruption_ assumption, which states that the corruption is _added_ to the feedback, and is _independent_ of the queried action, though it can depend on the executed action.) They derive decoupled approval versions of policy gradients and Q-learning, and prove that the local updates made by these algorithms move towards the approval maximizing policy (in expectation).

They then evaluate the algorithms on a new environment, REALab, in which the agent must collect apples. However, in this environment, the feedback variable is represented _in the environment_ by "registers". The agent can thus tamper with the feedback by interacting with these registers. The experiments show that while standard RL learns to tamper, DA-PG only tampers "accidentally" (i.e. to the same extent that is done by a policy trained with uncorrupted feedback). DA-QL tampers a bit more often, but this could just be due to noise.


It's great to see a paper that directly tackles a clear problem with AI alignment, and I think their solution works well in theory compared to other proposals. As the authors say, it's not practical yet, as it assumes knowledge of all states, computing coverage between all pairs of states, and that the agent can simulate the environment (to compute the baseline). On the theory side, I'm worried about what happens when properties interact. For example, suppose the agent takes an irreversible action because it is necessary for the objective. As a result of this action, there are new environment effects that don't happen in the baseline -- it seems like relative reachability would now start to penalize the agent for these effects, since they aren't present in the baseline. Dumb example: Suppose the agent is tasked with building a house, and cuts down some trees for the purpose. Now that there are no trees providing shade, the shallow pond evaporates away, wind intensifies and causes more leaves to fall, etc. and the agent is penalized for all of this because it wasn't in the baseline. More generally, as the agent takes actions in the real world, it will get further away from the baseline, and so the baseline becomes less useful.

The paper also notes that we could hope to learn this sort of behavior from humans, but that this seems hard to do, or at least that including a penalty on side effects can reduce the sample complexity of learning human values. I tend to agree -- in general, there seem to be two kinds of things we want to learn. First, what is it that we actually want our AI to do, and second, what common-sense things should it not do along the way. (In the formal methods community, these are called liveness and safety properties respectively.) In many areas of CS the second one arises as the "frame problem", which makes it difficult to write formal specifications, or to learn common-sense human preferences from humans. So it seems plausible to me that we want separate solutions for each of these kinds of values. I am a bit worried about having "common sense" be encoded formally as a reward penalty, because it seems very likely that it will be misspecified, but perhaps this does work well if combined with techniques that can learn from human data.

Measuring and avoiding side effects using relative reachability

Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, Shane Legg

One classic description of the AI alignment problem, from Stuart Russell, is that if you optimize a function of n variables, where the objective depends on k < n of these variables, then the remaining variables will often be set to extreme values, which can be highly undesirable if we actually care about those variables. This can be thought of as a negative side effect. This work attacks the problem of preventing negative side effects in general, _even if_ the reward function says nothing about the side effect. They show simple examples that motivate four properties that any solution should satisfy -- penalize side effects, not effects necessary for the objective; penalize agent-caused effects but not environment effects; penalize irreversible effects higher than reversible ones; and penalize multiple irreversible effects more than a single irreversible effect. They add a penalty term called relative reachability to the reward function to incentivize the agent not to cause side effects. Since we don't want to penalize environment effects (effects that would happen anyway), they compare against an "inaction baseline", where the agent does nothing (or follows some hardcoded safe policy). Since we want something more quantitative than "is this reversible", they create a numeric score of "coverage", which measures how easy it is to reach states from the current state, and penalize decreases in coverage relative to the baseline. This satisfies all of the properties we want -- it will still penalize irreversible actions that are necessary to achieve the objective, but as long as the penalty is small enough the reward for achieving the objective will dominate and the agent will take the action. It doesn't penalize environment effects because both the actual policy and the inaction baseline contain such effects. Clearly irreversible effects would lead to much lower coverage than reversible ones, and so irreversible effects are penalized more. Finally, multiple irreversible effects would lead to larger decreases in coverage than a single irreversible effect. They demonstrate these properties on toy gridworlds.


I think it is an underrated point that in some sense all we need to do to avoid x-risk is to make sure AI systems don't do crazy high-impact things in new situations, and that risk aversion is one way to get such an agent. This is also how <@Inverse Reward Design@> gets its safety properties: when faced with a completely new "lava" tile that the agent has never seen before, the paper's technique only infers that it should be _uncertain_ about the tile's reward. However, the _expected_ reward is still 0, and to get the agent to actually avoid the lava you need to use risk-averse planning.

The case for pessimism is similar to the case for impact measures, and similar critiques apply: it is not clear that we can get a value-agnostic method that is both sufficiently safe to rule out all catastrophes, and sufficiently useful to replace other AI techniques. The author himself points out that if we set β high enough to be confident it is safe, the resulting agent may end up always deferring to the mentor, and so not actually be of any use. Nonetheless, I think it's valuable to point out these ways that seem to confer some nice properties on our agents, even if they can't be pushed to the extremes for fear of making the agents useless.

Pessimism About Unknown Unknowns Inspires Conservatism

The argument for AI risk typically involves some point at which an AI system does something unexpected and bad in a new situation that we haven't seen before (as in e.g. a treacherous turn). One way to mitigate the risk is to simply detect new situations, and ensure the AI system does something known to be safe in such situations, e.g. deferring to a human, or executing some handcoded safe baseline policy. Typical approaches involve a separate anomaly detection model. This paper considers: can we use the AI system itself to figure out when to defer to a mentor?

_The key insight is that if an AI system maintains a distribution over rewards, and "assumes the worst" about the reward in new situations, then simply by deferring to the mentor with higher probability when the mentor would get higher expected reward, it will end up deferring to the mentor in new situations._ Hence, the title: by making the agent pessimistic about unknown unknowns (new situations), we get a conservative agent that defers to its mentor in new situations.

This is formalized in an AIXI-like setting, where we have agents that can have beliefs over all computable programs, and we only consider an online learning setting where there is a single trajectory over all time (i.e. no episodes). The math is fairly dense and I didn't try to fully understand it; as a result my summary may be inaccurate. The agent maintains a belief over world models (which predict how the environment evolves and how reward is given) and mentor models (which predict what the mentor will do, where the mentor's policy can depend on the **true** world model). It considers the β most likely world models (where β is a hyperparameter between 0 and 1). It computes the worst-case reward it could achieve under these world models, and the expected reward that the mentor achieves. It is more likely to defer to the mentor when the mentor's expected reward is higher (relative to its worst-case reward).

Such an agent queries the mentor finitely many times and eventually takes actions that are at least as good as the mentor's choices in those situations. In addition, for events with some bound on complexity, we can set things up (e.g. by having a high β) such that for any event, with high probability the agent never causes the event to occur unless the mentor has already caused the event to occur some time in the past. For example, with high probability the agent will never push the big red button in the environment, unless it has seen the mentor push the big red button in the past.


Robust Change Captioning

Dong Huk Park, Trevor Darrell, Anna Rohrbach

Safe exploration requires that agents avoid disrupting their environment. Previous work, such as <@Krakovna et al.@>(@Measuring and avoiding side effects using relative reachability@), penalize an agent's needless side effects on the environment. For such techniques to work in the real world, agents must also _estimate_ environment disruptions, side effects, and changes while not being distracted by peripheral and unaffecting changes. This paper proposes a dataset to further the study of "Change Captioning," where scene changes are described by a machine learning system in natural language. That is, given before and after images, a system describes the salient change in the scene. Work on systems that can estimate changes can likely progress safe exploration.


Since the AUP post was published about half a year ago, I've been watching this unification of AUP and relative reachability slowly take form, since they were phrased very differently initially. I'm glad to see this finally explained clearly and concisely, with experiments showing the effect of each choice. I do want to put special emphasis on the insight of AUP that the pursuit of convergent instrumental subgoals leads to large _increases_ in "ability to do things", and thus that penalizing increases can help avoid such subgoals. This point doesn't typically make it into the academic writings on the subject but seems quite important.

On the topic of impact measures, I'll repeat what I've said before: I think that it's hard to satisfy the conjunction of three desiderata -- objectivity (no dependence on human values), safety (preventing any catastrophic outcomes) and usefulness (the AI system is still able to do useful things). Impact measures are very clearly aiming for the first two criteria, but usually don't have much to say about the third one. My expectation is that there is a strong tradeoff between the first two criteria and the third one, and impact measures have not dealt with this fact yet, but will have to at some point.

Designing agent incentives to avoid side effects

Victoria Krakovna, Ramana Kumar, Laurent Orseau, Alexander Turner

This blog post provides details about the recent update to the [relative reachability paper](https://arxiv.org/abs/1806.01186) ([AN #10](https://mailchi.mp/d1a19c140226/alignment-newsletter-10)), which is now more a paper about the design choices available with impact measures. There are three main axes that they identify:

First, what baseline is impact measured relative to? A natural choice is to compare against the starting state, but this will penalize the agent for environment effects, such as apples growing on trees. We can instead compare against an inaction baseline, i.e. measuring impact relative to what would have happened if the agent did nothing. Unfortunately, this leads to offsetting behavior: the agent first makes a change to get reward, and then undoes the change in order to not be penalized for impact. This motivates the stepwise inaction baseline, which compares each action against what would have happened if the agent did nothing _from that step onwards_.

Second, we need a measure by which to compare states. The unreachability measure measures how hard it is to reach the baseline from the current state. However, this "maxes out" as soon as the baseline is unreachability, and so there is no incentive to avoid further irreversible actions. This motivates relative reachability, which computes the set of states reachable from the baseline, and measures what proportion of those states are reachable from the state created by the agent. [Attainable utility](https://www.alignmentforum.org/posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure) ([AN #25](https://mailchi.mp/0c5eeec28f75/alignment-newsletter-25)) generalizes this to talk about the _utility_ that could be achieved from the baseline for a wide range of utility functions. (This is equivalent to relative reachability when the utility functions are of the form "1 if state s is ever encountered, else 0".)

Finally, we need to figure how to penalize changes in our chosen measure. Penalizing decreases in the measure allows us to penalize actions that make it harder to do things (what the AUP post calls "opportunity cost"), while penalizing increases in the measure allows us to penalize convergent instrumental subgoals (which almost by definition increase the ability to satisfy many different goals or reach many different states).


These test cases and the associated commentary suggest to me that we haven't yet settled on what properties we'd like our impact regularization methods to satisfy, since there are pairs of test cases that seem hard to solve simultaneously, as well as test cases where the desired behavior is unclear.

Test Cases for Impact Regularisation Methods

This post collects various test cases that researchers have proposed for impact regularization methods. A summary of each one would be far too long for this newsletter, so you'll have to read the post itself.


I liked the analysis of what safety properties agents failed to satisfy, and the fact that agents sometimes fail the “obvious” or “easy” safety properties suggests that the bounded shielding approach can actually be useful in practice. Nonetheless, I still prefer the approach of finding an <@inductive safety invariant@>(@Neurosymbolic Reinforcement Learning with Formally Verified Exploration@), as it provides a guarantee of safety throughout the episode, rather than only for the next N timesteps.

Shielding Atari Games with Bounded Prescience

Mirco Giacobbe, Mohammadhosein Hasanbeig, Daniel Kroening, Hjalmar Wijk

In order to study agents trained for Atari, the authors write down several safety properties using the internals of the ALE simulator that agents should satisfy. They then test several agents trained with deep RL algorithms to see how well they perform on these safety properties. They find that the agents only successfully satisfy 4 out of their 43 properties all the time, whereas for 24 of the properties, all agents fail at least some of the time (and frequently they fail on every single rollout tested).

This even happens for some properties that should be easy to satisfy. For example, in the game Assault, the agent loses a life if its gun ever overheats, but avoiding this is trivial: just don't use the gun when the display shows that the gun is about to overheat.

The authors implement a "bounded shielding" approach, which basically simulates actions up to N timesteps in the future, and then only takes actions from the ones that don't lead to an unsafe state (if that is possible). With N = 1 this is enough to avoid the failure described above with Assault.


This paper presents a clear approach to handling side-effects and provides a fairly thorough analysis via experimentation. Having said that, I find the experimental findings to be mixed. Intuitively, adding more random rewards would decrease task performance and the number of side-effects. However, this isn't shown out in the data which raises interesting questions about how to best sample random reward functions. Related to this, the phenomena of side-effects increasing at the start of training for AUP is worth further investigation.

Avoiding Side Effects in Complex Environments

Alex Turner, Neale Ratzlaff, Prasad Tadepalli

One proposal for impact regularization is <@attainable utility preservation (AUP)@>(@Reframing Impact - Part 3@), in which we view side effects as changes in the ability of an agent to optimize a variety of reward functions. By incentivizing the agent not to change the optimal value for a wide range of auxiliary reward functions, the agent may avoid decreasing the optimal value for the true reward.

To test the claim that AUP is a suitable way to avoid side-effects the authors experiment in <@SafeLife@>(@Introducing SafeLife: Safety Benchmarks for Reinforcement Learning@), an environment suite based on Conway's "Game of Life". In the Game of Life, depending on how many live neighbors surround a cell, the cell either comes to life, dies, or retains its state. In SafeLife the eight cells surrounding the agent cells are frozen and can be modified by the agent. Thus, the agent can disturb, or modify, dynamic patterns by merely approaching them.

To measure side-effects the authors compare the evolution as it would've evolved without agent interference vs. the evolution with the agent present. The tasks are simple: either add or remove cells from a specified location. However, there are obstacles in the way that the agent could disturb. To implement AUP, the authors use a single randomly sampled reward function based on downsampling from the observation space. As a baseline, the authors compare AUP against PPO.

Generally, AUP is able to achieve fewer side-effects than PPO while still obtaining reasonable performance. However, AUP does take longer to train than PPO. Additionally, the side-effects incurred during the training of AUP increase to a peak before settling below the side-effect score of PPO. It's also important to note that sampling multiple rewards for AUP has the counter-intuitive effect of increasing the side-effect score.


I’m glad to see more work on this: robustness techniques seem particularly important to get working with learned specifications, and this paper (like the next one) takes a real shot at this goal. In some sense it isn’t that clear what we gain from an approach like this -- now, instead of requiring robustness from the agent, we require robustness from the safety Q-function (since we transfer it from the safe environment to the training environment). Nonetheless, we might hope the safety Q-function is easier to learn and more likely to transfer between the two environments, since it could be simpler than a full policy.

Learning to be Safe: Deep RL with a Safety Critic

Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, Chelsea Finn

While there has been a lot of work on verifying logical specifications and avoiding constraint violations, I've said before that the major challenge is in figuring out what specifications or constraints to use in the first place. This paper takes a stab at this problem by _learning_ safety constraints and transferring them to new situations.

In particular, we assume that we have some way of telling whether a given trajectory violates a constraint (e.g. a human looks at it and says whether or not a violation has happened). We also assume access to a safe environment in which constraint violations are acceptable. For example, for robots, our constraint could be that the robot never crashes, and in the safe environment the robot could be constrained to only move slowly, so that if they do crash there is no permanent damage. We then want to train the agent to perform some task in the true training environment (e.g. with no restrictions on speed), such that we avoid constraint violations with high probability _even during training_.

The key idea is to pretrain a _safety Q-function_ in the safe environment, that is, a function **Qsafe(s, a)** that specifies the probability of eventually violating a constraint if we take action **a** in state **s**. We have the agent choose actions that are estimated to be on the verge of being too risky, in order to optimize for getting more information about the constraints.

Once we have this safety Q-function, we can use it as a shield (<@1@>(@Neurosymbolic Reinforcement Learning with Formally Verified Exploration@), <@2@>(@Shielded Decision-Making in MDPs@)). Specifically, any actions whose risk is above some threshold ε have their probabilities set to zero. Using this shield, we can then train for the true task in the (unsafe) training environment using RL, while only behaving safely. Of course, this depends on the safety Q-function successfully generalizing to the new environment. We also add the safety Q-function as part of the RL objective to disincentivize constraint violations.

Their experiments show that this approach significantly reduces the number of constraint violations during training, though in absolute numbers there are often still hundreds of constraint violations (or about 1% of the number of training steps).


Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones

Brijen Thananjeyan*, Ashwin Balakrishna*, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, Ken Goldberg

This paper introduces Recovery RL, which tackles the same problem as the previous paper: given a dataset of constraint violations (presumably collected from some safe environment), train an agent to perform some task while exploring safely. Like the previous paper, it starts by learning a safety Q-function from the dataset of constraint violations.

The difference is in how this safety Q-function is used. The previous paper uses it as a shield on a policy, and also uses it in the RL objective to get a policy that is performant and safe. Recovery RL instead splits these into two separate policies: there is a task policy that only optimizes for performance, and a recovery policy that only optimizes for safety. During training, the safety Q-function is used to monitor the likelihood of a constraint violation, and if a violation is sufficiently likely, control is handed over to the recovery policy which then tries to make the probability of constraint violation as low as possible.

Experiments show that the method performs significantly better than other baselines (including SQRL, the method from the previous paper). Note however that the environments on which the methods were tested were not the same as in the previous paper.


I like the insight that offsetting is not always bad and the idea of dealing with the bad cases using the task reward. State-based reward functions that capture whether or not the task is currently done also intuitively seem like the correct way of specifying rewards in cases where achieving the task does not end the episode.

Tradeoffs between desirable properties for baseline choices in impact measures

<@Impact measures@>(@Measuring and avoiding side effects using relative reachability@) usually require a baseline state, relative to which we define impact. The choice of this baseline has important effects on the impact measure's properties: for example, the popular stepwise inaction baseline (where at every step the effect of the current action is compared to doing nothing) does not generate incentives to interfere with environment processes or to offset the effects of its own actions. However, it ignores delayed impacts and lacks incentive to offset unwanted delayed effects once they are set in motion.

This points to a **tradeoff** between **penalizing delayed effects** (which is always desirable) and **avoiding offsetting incentives**, which is desirable if the effect to be offset is part of the objective and undesirable if it is not. We can circumvent the tradeoff by **modifying the task reward**: If the agent is only rewarded in states where the task remains solved, incentives to offset effects that contribute to solving the task are weakened. In that case, the initial inaction baseline (which compares the current state with the state that would have occurred if the agent had done nothing until now) deals better with delayed effects and correctly incentivizes offsetting for effects that are irrelevant for the task, while the incentives for offsetting task-relevant effects are balanced out by the task reward. If modifying the task reward is infeasible, similar properties can be achieved in the case of sparse rewards by using the inaction baseline, and resetting its initial state to the current state whenever a reward is achieved. To make the impact measure defined via the time-dependent initial inaction baseline **Markovian**, we could sample a single baseline state from the inaction rollout or compute a single penalty at the start of the episode, comparing the inaction rollout to a rollout of the agent policy.


In my opinion on <@Safety Gym@>, I mentioned how a zero-violations constraint for safe exploration would require a mentor or parent that already satisfied the constraint; so in that sense I agree with this paper, which is simply making that statement more formal and precise.

Nonetheless, I still think there is a meaningful notion of exploration that can be done safely: once you have learned a good model that you have reasonable confidence in, you can find areas of the model in which you are uncertain, but you are at least confident that it won't have permanent negative repercussions, and you can explore there. For example, I often "explore" what foods I like, where I'm uncertain of how much I will like the food, but I'm quite confident that the food will not poison and kill me. (However, this notion of exploration is quite different from the notion of exploration typically used in RL, and might better be called "model-based exploration" or something like that.)

Curiosity Killed the Cat and the Asymptotically Optimal Agent

In environments without resets, an _asymptotically optimal_ agent is one that eventually acts optimally. (It might be the case that the agent first hobbles itself in a decidedly suboptimal way, but _eventually_ it will be rolling out the optimal policy _given_ its current hobbled position.) This paper points out that such agents must explore a lot: after all, it's always possible that the very next timestep will be the one where chopping off your arm gives you maximal reward forever -- how do you _know_ that's not the case? Since it must explore so much, it is extremely likely that it will fall into a "trap", where it can no longer get high reward: for example, maybe its actuators are destroyed.

More formally, the paper proves that when an asymptotically optimal agent acts, for any event, either that event occurs, or after some finite time there is no recognizable opportunity to cause the event to happen, even with low probability. Applying this to the event "the agent is destroyed", we see that either the agent is eventually destroyed, or it becomes _physically impossible_ for the agent to be destroyed, even by itself -- given that the latter seems rather unlikely, we would expect that eventually the agent is destroyed.

The authors suggest that safe exploration is not a well-defined problem, since you never know what's going to happen when you explore, and they propose that instead agents should have their exploration guided by a mentor or <@parent@>(@Parenting: Safe Reinforcement Learning from Human Input@) (see also <@delegative RL@>(@Delegative Reinforcement Learning@), [avoiding catastrophes via human intervention](https://arxiv.org/abs/1707.05173), and [shielding](https://arxiv.org/abs/1708.08611) for more examples).


I am confused about how much the commitment to balance out the original agent's attainable utility would constrain the powerful subagent. Also, in the presence of subagents, it seems plausible that attainable utility mostly depends on the agent's ability to produce subagents of different generality with different goals: If a subagent that optimizes for a single auxiliary reward was easier to build than a more general one, building a general powerful agent could considerably decrease attainable utility for all auxiliary rewards, such that the high penalty rules out this action.

Attainable utility has a subagent problem

This post argues that regularizing an agent's impact by <@attainable utility@>(@Towards a New Impact Measure@) can fail when the agent is able to construct subagents. Attainable utility regularization uses auxiliary rewards and penalizes the agent for changing its ability to get high expected rewards for these to restrict the agent's power-seeking. More specifically, the penalty for an action is the absolute difference in expected cumulative auxiliary reward between the agent either doing the action or nothing for one time step and then optimizing for the auxiliary reward.

This can be circumvented in some cases: If the auxiliary reward does not benefit from two agents instead of one optimizing it, the agent can just build a copy of itself that does not have the penalty, as doing this does not change the agent's ability to get a high auxiliary reward. For more general auxiliary rewards, an agent could build another more powerful agent, as long as the powerful agent commits to balancing out the ensuing changes in the original agent's attainable auxiliary rewards.


**While there is a fair amount of nuance regarding the specifics of how Uber's ADS was operating it does seem as though there was a fair amount of incompetence in how the ADS was deployed.** Turning off Volvo system fail-safes, not accounting for jaywalking, and trajectory reseting seem like unequivocal *mistakes*. A lot of people also seem upset that Uber was engaging in action suppression. However, given that randomly engaging in extreme maneuvering in the presence of other vehicles can *indirectly cause* accidents I have a small amount of sympathy for why such a feature existed in the first place. Of course, the feature was removed and it's worth noting that "there have been no unintended consequences—increased number of false alarms".

Jeff Kaufman writes a [post](https://www.lesswrong.com/posts/tTg4bn5rxHYqQJXhD) summarizing both the original incident and the report. Wikipedia is also rather thorough in their reporting on the factual information. Finally, *[Planning and Decision-Making for Autonomous Vehicles](https://www.annualreviews.org/doi/pdf/10.1146/annurev-control-060117-105157)* gives an overview of recent trends in the field and provides good references for people interested in safety concerns.

Vehicle Automation Report

Last week, the NTSB released a report on the Uber automated driving system (ADS) that hit and killed Elaine Herzberg. The pedestrian was walking across a two-lane street with a bicycle. However, the car didn't slow down before impact. Moreover, even though the environment was dark, the car was equipped with LIDAR sensors which means that the car was able to fully observe the potential for collision. The report takes a closer look at how Uber had set up their ADS and notes that in addition to not considering the possibility of jay-walkers, "...if the perception system changes the classification of a detected object, the tracking history of that object is no longer considered when generating new trajectories". Additionally, in the final few seconds leading up to the crash the vehicle engaged in *action suppression*, which is described as "a one-second period during which the ADS suppresses planned braking while the (1) system verifies the nature of the detected hazard and calculates an alternative path, or (2) vehicle operator takes control of the vehicle". The reason cited for implementing this was concerns of false alarms which could cause the vehicle to engage in unnecessary extreme maneuvers. Following the crash, Uber suspended its ADS operations and made several changes. They now use onboard safety features of the Volvo system that were previously turned off, action suppression is no longer implemented, and path predictions are held across object classification changes.


I agree with this -- as I've [said](https://www.alignmentforum.org/posts/kCY9dYGLoThC3aG7w/best-reasons-for-pessimism-about-impact-of-impact-measures#qAy66Wza8csAqWxiB) [before](https://www.alignmentforum.org/posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure#JFuGFW8ifAPAL6bjp), it seems hard to simultaneously avoid catastrophes, be useful, and be value agnostic. This post is arguing that if we want to avoid catastrophes and be useful, then we can't be value agnostic.

Reversible changes: consider a bucket of water

This post argues that impact regularization methods require preference information in order to work well. Consider a robot that has to navigate to a location, and the fastest way of doing so involves kicking a bucket of water into a pool to get it out of the way. Kicking the bucket is acceptable even though it is irreversible, but it may not be if the water has a special mixture of salts used for an industrial process. In order to determine the appropriate penalty for kicking the bucket, we need preference information -- it is not enough to think about anything value-agnostic like reversibility.


I really like how simple the change is here -- it should be a one-line change for many deep RL algorithms. Previously, we had to choose between unconstrained agents for high dimensional problems, or constrained agents for low dimensional problems -- I like that this work is making progress on constrained agents for high dimensional problems, similarly to [Constrained Policy Optimization](https://arxiv.org/abs/1705.10528). While this work doesn't involve a performance reward, you could use the resulting safe policy in order to guide a process of safe exploration to learn a policy that safely optimizes a performance metric. Of course, this is all assuming a specification for the constraint to satisfy.

Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning

Jaime F. Fisac*, Neil F. Lugovoy*, Vicenc Rubies-Royo, Shromona Ghosh, Claire J. Tomlin

Reinforcement learning is not great at enforcing constraints that hold at all times, because the agent would violate a constraint now if it would lead to higher reward later. In robust optimal control theory, we maximize the **minimum** of the constraint reward over time to avoid this. We can do this in the Bellman equation by taking a minimum between the current reward and estimated future value (instead of summing), but this does not uniquely define a fixed point. Just as in regular RL, we can use discounting to avoid the problem: in particular, if we interpret the discount as the probability that the episode continues, we can derive a Safety Bellman equation for which Q-learning is guaranteed to converge. They demonstrate their method in classic control environments as well as half-cheetah, with a range of RL algorithms including soft actor-critic (SAC).


[Self-confirming predictions can be arbitrarily bad](https://www.alignmentforum.org/posts/KoEY9CjrKe93ErYhd/self-confirming-predictions-can-be-arbitrarily-bad) and [Oracles, sequence predictors, and self-confirming predictions](https://www.alignmentforum.org/posts/i2dNFgbjnqZBfeitT/oracles-sequence-predictors-and-self-confirming-predictions)

Let's consider an oracle AI system tasked with accurate prediction, with a strong enough world model that it could understand how its prediction will affect the world. In that case, "accurate prediction" means giving a prediction P such that the world ends up satisfying P, _given_ the knowledge that prediction P was made. There need not be a single correct prediction -- there could be no correct prediction (imagine predicting what I will say given that I commit to saying something different from what you predict), or there could be many correct predictions (imagine instead that I commit to say whatever you predict). These self-confirming predictions could be arbitrarily bad.

Part of the point of oracles was to have AI systems that don't try to affect the world, but now the AI system will learn to manipulate us via predictions such that the predictions come true. Imagine for example the self-confirming prediction where the oracle predicts zero profit for a company, which causes the company to shut down.

In order to fix this, we could have _counterfactual oracles_, which predict what would have happened in a counterfactual where the prediction couldn't affect the world. In particular, we ask the oracle to predict the future **given that the prediction will immediately be erased and never be read by anyone**. We can also use this to tell how much the prediction can affect us, by looking at the difference between the unconditional prediction and the prediction conditioned on erasure.


Conservative Agency via Attainable Utility Preservation

Alexander Matt Turner, Dylan Hadfield-Menell, Prasad Tadepalli

This paper presents in a more academic format a lot of the content that Alex has published about attainable utility preservation, see <@Towards a New Impact Measure@> and <@Penalizing Impact via Attainable Utility Preservation@>.


While this is easier to read and understand, I think there are important points in the [original post](https://www.alignmentforum.org/posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure) that do not come across, so I would recommend reading both. In particular, one of my core takeaways from AUP was that convergent instrumental subgoals could be avoided by penalizing _increases_ in attainable utilities, and I don't think that comes across as well in this paper. This is the main thing that makes AUP different, and it's what allows it to avoid disabling the off switch in the Survival gridworld.

The fact that AUP works with random rewards is interesting, but I'm not sure it will generalize to realistic environments. In these gridworlds, there is usually a single thing that the agent is not supposed to do. It's very likely that several of the random rewards will care about that particular thing, which means that the AUP penalty will apply, so as long as full AUP would have solved the problem, AUP with random rewards would probably also solve it. However, in more realistic environments, there are many different things that the agent is supposed to avoid, and it's not clear how big a random sample of reward functions needs to be in order to capture all of them. (However, it does seem reasonably likely that if the reward functions are "natural", you only need a few of them to avoid convergent instrumental subgoals.)

Penalizing Impact via Attainable Utility Preservation

This post and the linked paper present [Attainable Utility Preservation] ([AN #25](https://mailchi.mp/0c5eeec28f75/alignment-newsletter-25)) more simply. There are new experiments that show that AUP works on some of the [AI Safety Gridworlds](https://arxiv.org/abs/1711.09883) even when using a set of _random_ utility functions, and compares this against other methods of avoiding side effects.


They require quite a lot of modeling in order to do this -- I think that it's specific to a particular kind of MDP, where there is an agent, and adversaries (the ghosts in Pacman), that are traversing a graph (the maze), which can have tokens (the food pellets). In theory, you should just solve the MDP and not use RL at all. Also in theory, shielding would actually require you to do this (in order to calculate the optimal values of actions), in which case it seems pointless (just use the optimal policy instead). In practice, the shield is only computed over a few timesteps. So you can think of this as a way of combining explicit, computationally-expensive forward reasoning (as in value iteration, for example) with RL, which learns from experience and can scale to much longer time horizons.

From the perspective of safety, I would be a lot more interested in approaches based on formal verification if they could work with learned features, rather than requiring that the human accurately formally model the world. This seems doable using a framework similar to [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://arxiv.org/abs/1707.05173), except by getting a formal safety specification iteratively instead of learning to mimic the human shield with neural nets.

Shielded Decision-Making in MDPs

Nils Jansen, Bettina Könighofer, Sebastian Junges, Roderick Bloem

Given a model of an MDP, we can compute a _shield_, which restricts the actions available to an RL agent to only the ones that can achieve at least some fraction of the optimal value. This results in safe exploration (since catastrophes would fall under the level that the shield guarantees), and also improves sample efficiency, since you no longer have tons of episodes in which the agent gets a large negative reward which only serve to teach it what not to do. They evaluate their approach on Pacman.


AXRP Episode 7 - Side Effects

This podcast goes over the problem of side effects, and impact regularization as an approach to handle this problem. The core hope is that impact regularization would enable "minimalistic" value alignment, in which the AI system may not be doing exactly what we want, but at the very least it will not take high impact actions that could cause an existential catastrophe.

An impact regularization method typically consists of a _deviation measure_ and a _baseline_. The baseline is what we compare the agent to in order to determine whether it had an "impact". The deviation measure is used to quantify how much impact there has been, when comparing the state generated by the agent to the one generated by the baseline.

Deviation measures are relatively uncontroversial – there are several possible measures, but they all seem to do relatively similar things, and there aren't any obviously bad outcomes traceable to problems with the deviation measure. However, that is not the case with baselines. One typical baseline is the **inaction** baseline, where you compare against what would have happened if the agent had done nothing. Unfortunately, this leads to _offsetting_: as a simple example, if some food was going to be thrown away and the agent rescues it, it then has an incentive to throw it away again, since that would minimize impact relative to the case where it had done nothing. A solution is the **stepwise inaction** baseline, which compares to the case where the agent does nothing starting from the previous state (instead of from the beginning of time). However, this then prevents some beneficial offsetting: for example, if the agent opens the door to leave the house, then the agent is incentivized to leave the door open.

As a result, the author is interested in seeing more work on baselines for impact regularization. In addition, she wants to see impact regularization tested in more realistic scenarios. That being said, she thinks that the useful aspect of impact regularization research so far is in bringing conceptual clarity to what we are trying to do with AI safety, and in identifying the interference and offsetting behaviors, and the incentives for them.


With any impact measure approach, I'm worried that there is no learning of what humans care about. As a result I expect that there will be issues that won't be handled properly (similarly to how we don't expect to be able to write down a human utility function). In the previous post, this manifested as a concern for generalization ability, which I'm still worried about. I think the tradeoff identified in this post is actually a manifestation of this worry -- clinginess happens when your AI overestimates what sorts of side effects humans don't want to happen in general, while manipulation of humans happens when your AI underestimates what side effects humans don't want to happen (though with the restriction that only humans can perform these side effects).

Overcoming Clinginess in Impact Measures

In their [previous post](https://www.lesswrong.com/posts/H7KB44oKoSjSCkpzL/worrying-about-the-vase-whitelisting), TurnTrout proposed a whitelisting approach, that required the AI not to cause side effects not on the whitelist. One criticism was that it made the AI _clingy_, that is, the AI would also prevent any other agents in the world from causing non-whitelisted effects. In this post, they present a solution to the clinginess problem. As long as the AI knows all of the other agents in the environment, and their policies, the AI can be penalized for the _difference_ of effects between its behavior, and what the human(s) would have done. There's analysis in a few different circumstances, where it's tricky to get the counterfactuals exactly right. However, this sort of impact measure means that while the AI is punished for causing side effects itself, it _can_ manipulate humans to perform those side effects on its behalf with no penalty. This appears to be a tradeoff in the impact measure framework -- either the AI will be clingy, where it prevents humans from causing prohibited side effects, or it could cause the side effects through manipulation of humans.


Challenges for Using Impact Regularizers to Avoid Negative Side Effects

David Lindner, Kyle Matoba, and Alexander Meulemans

I'm not summarizing this literature review on impact regularization because we've covered almost all of the ideas previously in this newsletter (e.g. <@this blog post@>(@Designing agent incentives to avoid side effects@)). However, I do recommend it for its short, high-level introduction to existing ideas in impact regularization, as well as its ideas for future work.


This seems like a good first step towards whitelisting -- there's still a lot of hardcoded knowledge from a human (which features to pay attention to, the transition function) and restrictions (the number of relevant features needs to be small), but it takes a problem and provides a solution that works in that setting. In the recent [whitelisting approach](https://www.lesswrong.com/posts/H7KB44oKoSjSCkpzL/worrying-about-the-vase-whitelisting), I was worried that the whitelist simply wouldn't include enough transitions for the agent to be able to do anything useful. Since this approach actively queries the operator until it finds a safe policy, that is no longer an issue. However, the corresponding worry would be that it takes prohibitively many queries before the agent can do anything useful. (Their empirical evaluation is on toy gridworlds, so this problem did not come up.) Another worry previously was that whitelisting causes an agent to be "clingy", that is, it wants to prevent all changes to non-whitelisted features, even if they are caused by physical laws, or other humans. A similar problem could arise here when this is generalized to dynamic and/or multiagent environments.

Minimax-Regret Querying on Side Effects for Safe Optimality in Factored Markov Decision Processes

Shun Zhang,  Edmund H. Durfee, and Satinder Singh

As we saw in [Alignment Newsletter #11](https://mailchi.mp/7ad36e25be22/alignment-newsletter-11), one approach to avoiding side effects is to create a whitelist of effects that are allowed. In this paper, the agent learns both a whitelist of allowed effects, and a blacklist of disallowed effects. They assume that the MDP in which the agent is acting has been factored into a set of features that can take on different values, and then separate the features as locked (unchangeable), free (changeable), or unknown. If there are no unknown features, then we can calculate the optimal policy using variants of standard techniques (for example, by changing the transition function to remove transitions that would change locked features, and then running any off-the-shelf MDP solver). However, this would require the operator to label all features as locked or unlocked, which would be very tedious. To solve this, they allow the agent to query the operator whether a certain feature is locked or unlocked, and provide algorithms that reduce the number of queries that the agent needs to make in order to find an optimal safe policy.


I liked this exploration of a somewhat more principled underpinning to impact measures, and it is encouraging that this formalization of option value preservation gives similar results as previous formalizations.

Avoiding Side Effects By Considering Future Tasks

Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, Shane Legg

We are typically unable to specify all of the things that the agent should _not_ change about the environment. So, we would like a generic method that can penalize these _side effects_ in arbitrary environments for an arbitrary reward function. Typically, this is done via somehow preserving option value, as with <@relative reachability@>(@Measuring and avoiding side effects using relative reachability@) and <@attainable utility preservation@>(@Penalizing Impact via Attainable Utility Preservation@).

This paper aims to encode the goal of "option value preservation" in a simpler and more principled manner: specifically, at some point in the future we will randomly choose a new task to give to the agent, so that the agent must maintain its ability to pursue the possible tasks it can see in the future. However, if implemented as stated, this leads to interference incentives -- if something were going to restrict the agent's option value, such as a human irreversibly eating some food, the agent would be incentivized to interfere with that process in order to keep its option value for the future. The authors provide a formal definition of this incentive.

To fix this problem, the authors introduce a baseline policy (which could be set to e.g. noop actions), and propose a future task reward that only provides reward if after the baseline policy had been executed, it would still have been possible to complete the future task. Thus, the agent is only incentivized to preserve options that would have been available had it done whatever the baseline policy does, eliminating the interference incentive in the deterministic case. The authors demonstrate on simple gridworlds that the future task approach with the baseline allows us to avoid side effects, while also not having interference incentives.

Normally we would also talk about how to remove the offsetting incentive, where the agent may be incentivized to undo effects it did as part of the task to avoid being penalized for them. (The example from relative reachability is of an agent that is rewarded for taking a vase off of a conveyor belt, and then puts it back on to minimize its impact.) However, the authors argue that offsetting is often desirable. For example, if you open the door to go to the grocery store, you do want to "offset" your impact by closing the door as you leave, even though opening the door was important for the task of buying groceries. They argue that offsetting incentives should be left in, and the burden is on the reward designer to ensure that anything that shouldn't be offset is specified as such in the reward function. In the original conveyor belt example, we shouldn't reward the action of taking the vase off the conveyor belt, but instead the state in which the vase is not on the conveyor belt.


Avoiding Negative Side Effects due to Incomplete Knowledge of AI Systems

Sandhya Saisubramanian, Shlomo Zilberstein, Ece Kamar

This paper provides an overview of the problem of negative side effects, and recent work that aims to address it. It characterizes negative side effects based on whether they are severe, reversible, avoidable, frequent, stochastic, observable, or exclusive (i.e. preventing the agent from accomplishing its main task), and describes existing work and how they relate to these characteristics.

In addition to the canonical point that negative side effects arise because the agent's model is lacking (whether about human preferences or environment dynamics or important features to pay attention to), they identify two other main challenges with negative side effects. First, fixing negative side effects would likely require collecting feedback from humans, which can be expensive and challenging. Second, there will usually be a tradeoff between pursuing the original goal and avoiding negative side effects; we don't have principled methods for dealing with this tradeoff.

Finally, they provide a long list of potential directions for future side effect research.


Whitelisting seems like a good thing to do, since it is safe by default. (Computer security has a similar principle of preferring to whitelist instead of blacklist.) I was initially worried that we'd have the problems of symbolic approaches to AI, where we'd have to enumerate far too many transitions for the whitelist in order to be able to do anything realistic, but since whitelisting could work on learned embedding spaces, and the whitelist itself can be learned from demonstrations, this could be a scalable method. I'm worried that it presents generalization challenges -- if you are distinguishing between different colors of tiles, to encode "you can paint any tile" you'd have to whitelist transitions (redTile -> blueTile), (blueTile -> redTile), (redTile -> yellowTile) etc. Those won't all be in the demonstrations. If you are going to generalize there, how do you _not_ generalize (redLight -> greenLight) to (greenLight -> redLight) for an AI that controls traffic lights? On another note, I personally don't want to assume that we can point to a part of the architecture as the AI's ontology. I hope to see future work address these challenges!

Worrying about the Vase: Whitelisting

It's really hard to avoid negative side effects because explicitly listing out all possible side effects the agent should avoid would be far too expensive. The issue is that we're trying to build a blacklist of things that can't be done, and that list will never be complete, and so some bad things will still happen. Instead, we should use whitelists, because if we forget to add something to the whitelist, that only limits the agent, it doesn't lead to catastrophe. In this proposal, we assume that we have access to the agent's ontology (in current systems, this might be the output of an object detection system), and we operationalize an "effect" as the transformation of one object into another (i.e. previously the AI believed an object was most likely an A, and now it believes it is most likely a B). We then whitelist allowed transformations -- for example, it is allowed to transform a carrot into carrot slices. If the agent causes any transformations not on the whitelist (such as "transforming" a vase into a broken vase), it incurs a negative reward. We also don't have to explicitly write down the whitelist -- we can provide demonstrations of acceptable behavior, and any transitions in these demonstrations can be added to the whitelist. The post and paper have a long list of considerations on how this would play out in a superintelligent AI system.


It’s worth noting that measures like AUP do create a Markovian reward function, which typically leads to time consistent agents. The reason that this doesn’t apply here is because we’re assuming that the restriction of future choices is “external” to the environment and formalism, but nonetheless affects the penalty. If we instead have this restriction “inside” the environment, then we will need to include a state variable specifying whether the action set is restricted or not. In that case, the impact measure would create a reward function that depends on that state variable. So another way of stating the problem is that if you add the ability to restrict future actions to the environment, then the impact penalty leads to a reward function that depends on whether the action set is restricted, which intuitively we don’t want. (This point is also made in this [followup post](https://www.alignmentforum.org/posts/YjqwTepi53MyM4omT/why-is-the-impact-penalty-time-inconsistent).)

Dynamic inconsistency of the inaction and initial state baseline

In a fixed, stationary environment, we would like our agents to be time-consistent: that is, they should not have a positive incentive to restrict their future choices. However, impact measures like <@AUP@>(@Towards a New Impact Measure@) calculate impact by looking at what the agent could have done otherwise. As a result, the agent has an incentive to change what this counterfactual is, in order to reduce the penalty it receives, and it might accomplish this by restricting its future choices. This is demonstrated concretely with a gridworld example.


Avoiding Side Effects in Complex Environments

Alexander Matt Turner*, Neale Ratzlaff*, Prasad Tadepalli

Previously, attainable utility preservation (AUP) has been used to <@solve@>(@Penalizing Impact via Attainable Utility Preservation@) some simple gridworlds. Can we use it to avoid side effects in complex high dimensional environments as well? This paper shows that we can, at least in <@SafeLife@>(@Introducing SafeLife: Safety Benchmarks for Reinforcement Learning@). The method is simple: first train a VAE on random rollouts in the environment, and use randomly generated linear functions of the VAE features as the auxiliary reward functions for the AUP penalty. The Q-functions for these auxiliary reward functions can be learned using deep RL algorithms. Then we can just do regular deep RL using the specified reward and the AUP penalty. It turns out that this leads to fewer side effects with just _one_ auxiliary reward function and a VAE whose latent space is size _one_! It also leads to faster learning for some reason. The authors hypothesize that this occurs because the AUP penalty is a useful shaping term, but don't know why this would be the case.


I enjoyed this article and the proposed factors match my intuitions. There seem to be two types of problems: extreme beliefs and concave Pareto boundaries. Dealing with the second is more important since a concave Pareto boundary favours extreme policies, even for moderate beliefs. Luckily, diminishing returns can be used to bend the Pareto boundary. However, I expect it to be hard to find the correct rate of diminishing returns, especially in novel situations.

**Rohin's opinion:** Note that this post considers the setting where we have uncertainty over the true reward function, but _we can't learn about the true reward function_. If you can gather information about the true reward function, which <@seems necessary to me@>(@Human-AI Interaction@), then it is almost always worse to take the most likely reward or expected reward as a proxy reward to optimize.

When Goodharting is optimal: linear vs diminishing returns, unlikely vs likely, and other factors

Suppose we were uncertain about which arm in a bandit provides reward (and we don't get to observe the rewards after choosing an arm). Then, maximizing expected value under this uncertainty is equivalent to picking the most likely reward function as a proxy reward and optimizing that; Goodhart's law doesn't apply and is thus not universal. This means that our fear of Goodhart effects is actually informed by more specific intuitions about the structure of our preferences. If there are actions that contribute to multiple possible rewards, optimizing the most likely reward does not need to maximize the expected reward. Even if we optimize for that, we have a problem if value is complex and the way we do reward learning implicitly penalizes complexity. Another problem arises if the correct reward is comparatively difficult to optimize: if we want to maximize the average, it can make sense to only care about rewards that are both likely and easy to optimize. Relatedly, we could fail to correctly account for diminishing marginal returns in some of the rewards.

Goodhart effects are a lot less problematic if we can deal with all of the mentioned factors. Independent of that, Goodhart effects are most problematic when there is little middle ground that all rewards can agree on.


Safety Aware Reinforcement Learning (SARL)

Santiago Miret, Somdeb Majumdar, Carroll Wainwright

Many approaches to safety rely on learning from a trusted overseer (typically a human), including <@iterated amplification@>(@Supervising strong learners by amplifying weak experts@), <@debate@>(@AI safety via debate@), <@parenting@>(@Parenting: Safe Reinforcement Learning from Human Input@), <@delegative RL@>(@Delegative Reinforcement Learning@), and [quantilization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) ([AN #48](https://mailchi.mp/3091c6e9405c/alignment-newsletter-48)). This paper applies this idea to avoiding side effects in the <@SafeLife environment@>(@Introducing SafeLife: Safety Benchmarks for Reinforcement Learning@). They train a safety agent to minimize side effect score to use as a proxy for the trusted overseer, and then train a regular RL agent to optimize reward while penalizing deviations from the safety agent's policy. They find that the safety agent can be transferred zero-shot to new environments and help reduce side effects in those environments as well.


NeurIPS 2019 Workshop on Safety and Robustness in Decision Making

This method seems like a good way to build in domain knowledge about what kinds of action sequences are unlikely to work in a domain, which can help accelerate learning. Both of the constraints in the experiments do this. The paper also suggests using the technique to enforce safety constraints, but the experiments don't involve any safety constraints, and conceptually there do seem to be two big obstacles. First, the constraints will depend on state, but it is very hard to write such constraints given access only to actions and high-dimensional pixel observations. Second, you can only prevent constraint violations by removing actions one timestep before the constraint is violated: if there is an action that will inevitably lead to a constraint violation in 10 timesteps, there's no way in this framework to not take that action. (Of course, you can use a soft constraint, but this is then the standard technique of reward shaping.)

In general, methods like this face a major challenge: how do you specify the safety constraint that you would like to avoid violating? I'd love to see more research on how to create specifications for formal analysis.

Formal Language Constraints for Markov Decision Processes

Eleanor Quint, Dong Xu, Haluk Dogan, Zeynep Hakguder, Stephen Scott, Matthew Dwyer

Within the framework of RL, the authors propose using constraints defined by DFAs (deterministic finite automata) in order to eliminate safety failures, or to prevent agents from exploring clearly ineffective policies (which would accelerate learning). Constraints can be defined on any auxiliary information that can be computed from the "base" MDP. A constraint could either restrict the action space, forcing the agent to take an action that doesn't violate the constraint, which they term "hard" constraints; or a constraint could impose a penalty on the agent, thus acting as a form of reward shaping, which they term a "soft" constraint. They consider two constraints: one that prevents the agent from "dithering" (going left, then right, then left, then right), and one that prevents the agent from "overactuating" (going in the same direction four times in a row). They evaluate their approach with these constraints on Atari games and Mujoco environments, and show that they lead to increased reward and decreased constraint violations.


I'm very excited to see a paper tackling safety in hierarchical reinforcement learning -- that seems like a really important area to consider, and doesn't have many safety people working on it yet. That said, this paper feels weird to me, because in order to learn that a particular (state, option) pair is bad, the RL agent must experience that pair somewhat often, so it will have done the risky thing. It's not clear to me where this would be useful. One upside could be that we do risky things less often, so our RL agent learns faster from its mistakes, and doesn't make them as often. (And in fact they find that this leads to faster learning in three games in the ALE.) Perhaps we could also use this to train a risk-averse agent in simulation, that then never makes a mistake when deployed in the real world.

I also wonder whether we should be trying to make our agents risk-averse. The "right" answer seems to me to a combination of two things: First, some things are actually very bad and have very large negative reward, and so they should be avoided with high probability. Second, when you are acting over a long period of time, even a small probability of failure at every time step compounds and leads to a near-guaranteed failure. If these are actually the reasons underlying risk aversion, it seems like we want to be able to imbue our RL agent with the underlying reasons, rather than flat risk aversion.

Safe Option-Critic: Learning Safety in the Option-Critic Architecture

Arushi Jain, Khimya Khetarpal, Doina Precup

Let's consider an RL agent in the options framework (one way of doing hierarchical reinforcement learning). One way in which we could make such an agent safer would be to make it risk-averse. The authors define the controllability of a (state, option) pair to be the negative expected variance of the TD error. Intuitively, the controllability is higher when the value of the (state, option) pair is more predictable to the agent, and so by optimizing for controllability we can encourage risk-aversion. They derive the policy gradient when the objective is to maximize the reward and the controllability of the initial (state, option) pair, and use this to create the Safe-A2OC algorithm (a safe version of A2OC, which itself is a version of A2C for options). They test this out on a four-rooms gridworld problem, Cartpole, and three games from the Arcade Learning Environment (ALE).


[Three AI Safety Related Ideas](https://www.alignmentforum.org/posts/vbtvgNXkufFRSrx4j/three-ai-safety-related-ideas) and [Two Neglected Problems in Human-AI Safety](https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety)

Last week, I said that the problem of defending against intentional value corruption was an instance of the problem "Competing aligned superintelligent AI systems could do bad things", and I wasn't sure why we were focusing on value corruption in particular. In [this comment](https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety#dykZxAXGr6sk7bh4e), Wei Dai argues that superintelligent AI systems could be really good at cooperating with each other, which solves _most_ of the problems. However, the terms of such cooperation will probably reflect the balance of power between the AI systems, which may tend to benefit simpler value systems rather than ones with a proper amount of value complexity and moral uncertainty. This seems plausible to me, though I'm not confident one way or the other.