I like that this paper subdivides failure modes that can happen in multiparty optimization into several clear categories and provides various models and examples for each of them. I am unsure about the conclusion: on one hand, slowing down deployment to improve the safety of contemporary systems seems very sensible. On the other hand, it seems like there would be some failures of limited scope that are hard to reproduce "in the lab". Widely deployed AI systems might provide us with valuable empirical data about these failures and improve our understanding of the failure modes in general. I guess ideally there would be differential deployment with rapid deployment in noncritical areas like managing local parking lots, but very slow deployment for critical infrastructure.

**Rohin's opinion:** I'm particularly interested in an analysis of how these kinds of failures affect existential risk. I'm not sure if David believes they are relevant for x-risk, but even if so the arguments aren't presented in this paper.

Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence

While [Categorizing Variants of Goodhart's Law](https://arxiv.org/abs/1803.04585) explains failure modes that occur when a single agent's proxy becomes decoupled from the true goal, this paper aims to characterize failures involving multiple agents:

 **Accidental steering** happens when the combined actions of multiple agents facilitate single-agent failures. For example, catching more fish now is usually positively correlated with a fisherman's long term goals, but this relationship inverts once there are lots of fishermen optimizing for short term gains and the fish population collapses.

**Coordination Failure** occurs when agents with mutually compatible goals fail to coordinate. For example, due to incomplete models of other agent's goals and capabilities, two agents sharing a goal might compete for a resource even though one of them is strictly better at converting the resource into progress towards their goal.

**Adversarial optimization** is when an agent **O** steers the world into states where **V**'s proxy goal is positively correlated with **O**'s goal. For example, one could exploit investors who use short term volatility as a proxy for risk by selling them instruments that are not very volatile but still risky.

**Input Spoofing** is the act of one agent manipulating another learning agent's model, either by manufacturing false evidence or by filtering the received evidence systematically, as arguably happened with [Microsoft's Tay](https://en.wikipedia.org/wiki/Tay_(bot)).

Finally, **Goal co-option** happens when agent **O** has (partial) control over the hardware agent **V** runs or relies on. This way, **O** can either modify the reward signal **V** receives to change what **V** optimizes for, or it can directly change **V**'s outputs.

The difficulties in precisely modelling other sophisticated agents and other concerns related to embedded agency make it hard to completely avoid these failure modes with current methods. Slowing down the deployment of AI systems and focussing on the mitigation of the discussed failure modes might prevent limited near term catastrophes, which in turn might cause a slowdown of further deployment and prioritization of safety.


It is good to see an attempt at formalizing reward hacking. The experimental contributions are interesting and the anomaly detection method seems reasonable. However, the proxy rewards chosen to represent reward hacking are questionable. In my opinion, these rewards are obviously 'wrong' so it is less surprising that they result in undesired behavior. I look forward to seeing more comprehensive experiments on this subject.

**Rohin’s opinion:** Note that on OpenReview, the authors say that one of the proxy rewards (maximize average velocity for the driving environment) was actually the default and they only noticed it was problematic after they had trained large neural nets on that environment. I do agree that future proxy objectives will probably be less clearly wrong than most of the ones in this paper.

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

Reward hacking occurs when RL agents exploit the difference between a true reward and a proxy. Reward hacking has been <@observed in practice@>(@Specification gaming examples in AI@), and as reinforcement learning agents are trained with better algorithms, more data, and larger policies, they are at increased risk of overfitting their proxy objectives. However, reward hacking has not yet been systematically studied.

This paper fills this gap by constructing four example environments with a total of nine proxy rewards to investigate how reward hacking changes as a function of optimization power. They increase optimization power in several different ways, such as increasing the size of the neural net, or providing the model with more fine-grained observations.

Overall, the authors find that reward hacking occurs in five of the nine cases. Moreover, the authors observed phase transitions in four of these cases. These are stark transitions where a moderate increase in optimization power leads to a drastic increase in reward hacking behavior. This poses a challenge in monitoring the safety of ML systems. To address this the authors suggest performing anomaly detection to notice reward hacking and offer several baselines.


Why AI alignment could be hard with modern deep learning

This post provides an ELI5-style introduction to AI alignment as a major challenge for deep learning. It primarily frames alignment as a challenge in creating Saints (aligned AI systems), without getting Schemers (AI systems that are <@deceptively aligned@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@)) or Sycophants (AI systems that satisfy only the letter of the request, rather than its spirit, as in <@Another (outer) alignment failure story@>). Any short summary I write would ruin the ELI5 style, so I won't attempt it; I do recommend it strongly if you want an introduction to AI alignment.


I like the low-stakes assumption as a way of saying "let's ignore distributional shift for now". Probably the most salient alternative is something along the lines of "assume that the AI system is trying to optimize the true reward function". The main way that low-stakes alignment is cleaner is that it uses an assumption on the _environment_ (an input to the problem) rather than an assumption on the _AI system_ (an output of the problem). This seems to be a lot nicer because it is harder to "unfairly" exploit a not-too-strong assumption on an input rather than on an output. See [this comment thread](https://www.alignmentforum.org/posts/TPan9sQFuPP6jgEJo/low-stakes-alignment?commentId=askebCP36Ce96ZiJa) for more discussion.

Low-stakes alignment

We often split AI alignment into two parts: outer alignment, or ``finding a good reward function'', and inner alignment, or ``robustly optimizing that reward function''. However, these are not very precise terms, and they don't form clean subproblems. In particular, for outer alignment, how good does the reward function have to be? Does it need to incentivize good behavior in all possible situations? How do you handle the no free lunch theorem? Perhaps you only need to handle the inputs in the training set? But then what specifies the behavior of the agent on new inputs?

This post proposes an operationalization of outer alignment that admits a clean subproblem: _low stakes alignment_. Specifically, we are given as an assumption that we don't care much about any small number of decisions that the AI makes -- only a large number of decisions, in aggregate, can have a large impact on the world. This prevents things like quickly seizing control of resources before we have a chance to react. We do not expect this assumption to be true in practice: the point here is to solve an easy subproblem in the hopes that the solution is useful for solving the hard version of the problem.

The main power of this assumption is that we no longer have to worry about distributional shift. We can simply keep collecting new data online and training the model on the new data. Any decisions it makes in the interim period could be bad, but by the low-stakes assumption, they won't be catastrophic. Thus, the primary challenge is in obtaining a good reward function that incentivizes the right behavior after the model is trained. We might also worry about whether gradient descent will successfully find a model that optimizes the reward even on the training distribution -- after all, gradient descent has no guarantees for non-convex problems -- but it seems like, to the extent that gradient descent doesn't do this, it will probably affect aligned and unaligned models equally.

Note that this subproblem is still non-trivial, and existential catastrophes still seem possible if we fail to solve it. For example, one way that the low-stakes assumption could be made true was if we had a lot of bureaucracy and safeguards that the AI system had to go through before making any big changes to the world. It still seems possible for the AI system to cause lots of trouble if none of the bureaucracy or safeguards can understand what the AI system is doing.


I think this topic is important and the post did it justice. I feel like factors 4 and 5 (dependency and opposition) capture the reasons I expect lock in, with factors 1-3 as less important but still relevant mechanisms. I also really liked the analogy with the British colonization of New Zealand -- it felt like it was in fact quite analogous to how I’d expect this sort of failure to happen.

Clarifying “What failure looks like” (part 1)

The first scenario outlined in <@What failure looks like@> stems from a failure to specify what we actually want, so that we instead build AI systems that pursue proxies of what we want instead. As AI systems become responsible for more of the economy, human values become less influential relative to the proxy objectives the AI systems pursue, and as a result we lose control over the future. This post aims to clarify whether such a scenario leads to _lock in_, where we are stuck with the state of affairs and cannot correct it to get "back on course". It identifies five factors which make this more likely:

1. _Collective action problems:_ Many human institutions will face competitive (short-term) pressures to deploy AI systems with bad proxies, even if it isn't in humanity's long-term interest.
2. _Regulatory capture:_ Influential people (such as CEOs of AI companies) may benefit from AI systems that optimize proxies, and so oppose measures to fix the issue (e.g. by banning such AI systems).
3. _Ambiguity:_ There may be genuine ambiguity about whether it is better to have these AI systems that optimize for proxies, even from a long-term perspective, especially because all clear and easy-to-define metrics will likely be going up (since those can be turned into proxy objectives).
4. _Dependency:_ AI systems may become so embedded in society that society can no longer function without them.
5. _Opposition:_ The AI systems themselves may oppose any fixes we propose.

We can also look at historical precedents. Factors 1-3 have played an important role in climate change, though if it does lead to lock in, this will be "because of physics", unlike the case with AI. The agricultural revolution, which arguably made human life significantly worse, still persisted thanks to its productivity gains (factor 1) and the loss of hunter-gathering skills (factor 4). When the British colonized New Zealand, the Maori people lost significant control over their future, because each individual chief needed guns (factor 1), trading with the British genuinely made them better off initially (factor 3), and eventually the British turned to manipulation, confiscation and conflict (factor 5).

With AI in particular, we might expect that an increase in misinformation and echo chambers exacerbates ambiguity (factor 3), and that due to its general-purpose nature, dependency (factor 4) may be more of a risk.

The post also suggests some future directions for estimating the _severity_ of lock in for this failure mode.


If the last sentence in my summary seems controversial or wrong to you, then you should read this post.

Non-Adversarial Goodhart and AI Risks

Goodhart's Law can be used as a lens to identify potential problems with AI systems. Regressional and extremal Goodhart correspond to the problems of hardcoded utility functions and distributional shift, which we're familiar with already. Causal Goodhart is different -- it is a failure of the AI system to model the world correctly. The post argues that causality is hard to infer, can lead to bad outcomes only over long timescales, and may not be easily found through testing, so we need to focus more on them now.


Classifying specification problems as variants of Goodhart's Law

This post argues that the specification problems from the <@SRA framework@>(@Building safe artificial intelligence: specification, robustness, and assurance@) are analogous to the [Goodhart taxonomy](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy). Suppose there is some ideal specification. The first step is to choose a model class that can represent the specification, e.g. Python programs at most 1000 characters long. If the true best specification within the model class (called the model specification) differs from the ideal specification, then we will overfit to that specification, selecting for the difference between the model specification and ideal specification -- an instance of regressional Goodhart. But in practice, we don't get the model specification; instead humans choose some particular proxy specification, typically leading to good behavior on training environments. However, in new regimes, this may result in optimizing for some extreme state where the proxy specification no longer correlates with the model specification, leading to very poor performance according to the model specification -- an instance of extremal Goodhart. (Most of the classic worries of specifying utility functions, including e.g. negative side effects, fall into this category.) Then, we have to actually implement the proxy specification in code, giving an implementation specification. Reward tampering allows you to "hack" the implementation to get high reward, even though the proxy specification would not give high reward, an instance of causal Goodhart.

They also argue that the ideal -> model -> proxy problems are instances of problems with selection, while the proxy -> implementation problems are instances of control problems (see <@Selection vs Control@>). In addition, the ideal -> model -> proxy -> implementation problems correspond to outer alignment, while inner alignment is a part of the implementation -> revealed specification problem.


You may have noticed that this newsletter doesn't talk about wireheading very much; this is one of the reasons why. It seems like wireheading is a fuzzy subset of specification gaming, and is not particularly likely to be the only kind of specification gaming that could lead to catastrophe. I'd be surprised if we found some sort of solution where we'd say "this solves all of wireheading, but it doesn't solve specification gaming" -- there don't seem to be particular distinguishing features that would allow us to have a solution to wireheading but not specification gaming. There can of course be solutions to particular kinds of wireheading that _do_ have clear distinguishing features, such as <@reward tampering@>(@Designing agent incentives to avoid reward tampering@), but I don't usually expect these to be the major sources of AI risk.

Defining AI wireheading

This post points out that "wireheading" is a fuzzy category. Consider a weather-controlling AI tasked with increasing atmospheric pressure, as measured by the world's barometers. If it made a tiny dome around each barometer and increased air pressure within the domes, we would call it wireheading. However, if we increase the size of the domes until it's a dome around the entire Earth, then it starts sounding like a perfectly reasonable way to optimize the reward function. Somewhere in the middle, it must have become unclear whether or not it was wireheading. The post suggests that wireheading can be defined as a subset of <@specification gaming@>(@Specification gaming examples in AI@), where the "gaming" happens by focusing on some narrow measurement channel, and the fuzziness comes from what counts as a "narrow measurement channel".


The "Commitment Races" problem

When two agents are in a competitive game, it is often to each agent's advantage to quickly make a credible commitment before the other can. For example, in Chicken (both players drive a car straight towards the other and the first to swerve out of the way loses), an agent could rip out their steering wheel, thus credibly committing to driving straight. The first agent to do so would likely win the game. Thus, agents have an incentive to make commitments as quickly as possible, before their competitors can make commitments themselves. This trades off against the incentive to think carefully about commitments, and may result in arbitrarily bad outcomes.


Problems in AI Alignment that philosophers could potentially contribute to

Exactly what it says. The post is short enough that I'm not going to summarize it -- it would be as long as the original.


I have a bunch of complicated thoughts on this post, many of which were said in Paul's comment reply to the post, but I'll say a few things. Firstly, I think that if you want to view the AI alignment problem in the context of the principal-agent literature, the natural way to think about it is with the principal being less rational than the agent. I claim that it is at least conceivable that an AI system could make humans worse off, but the standard principal-agent model cannot accommodate such a scenario because it assumes the principal is rational, which means the principal always does at least as well as not ceding any control to the agent at all. More importantly, although I'm not too familiar with the principal-agent literature, I'm guessing that the literature assumes the presence of norms, laws and institutions that constrain both the principal and the agent, and in such cases it makes sense that the loss that the principal could incur would be bounded -- but it's not obvious that this would hold for sufficiently powerful AI systems.

Agency Failure AI Apocalypse?

This is a response to <@More realistic tales of doom@>, arguing that the scenarios described in the post are unrealistic given what we know about principal-agent problems. In a typical principal-agent problem, the principal doesn't know everything about the agent, and the agent can use this fact to gain "agency rents" where it can gain extra value for itself, or there could be an "agency failure" where the principal doesn't get as much as they want. For example, an employee might spend half of their day browsing the web, because their manager can't tell that that's what they are doing. Our economic literature on principal-agent problems suggests that agency problems get harder with more information asymmetry, more noise in outcomes, etc. but not with smarter agents, and in any case we typically see limited agency rents and failures. So, it's unlikely that the case for AI will be any different, and while it's good to have a couple of people keeping an eye on the problem, it's not worth the large investment of resources from future-oriented people that we currently see.


The Main Sources of AI Risk?

This post lists different causes or sources of existential risk from advanced AI.


This seems pretty similar to the problem of inner optimizers, in which while searching for a good policy for some task T on training distribution D, you end up finding a consequentialist agent that is optimizing some utility function that leads to good performance on D. That agent will have all the standard dangers of goal-directed optimization out of distribution.

Imitation learning considered unsafe?

We might hope that using imitation learning to mimic a corrigible human would be safe. However, this would involve mimicking the human's planning process. It seems fairly likely that slight errors in the imitation of this process could lead to the creation of a goal-directed planning process that does dangerous long-term optimization.


Implicit extortion sounds like a hard problem to solve, and the post argues that humans don't robustly solve it. I'm not sure whether this is a problem we need to solve in order to get good outcomes -- if you can detect that implicit extortion is happening, you can take steps to avoid being extorted, and so it seems that a successful implicit extortion attack would have to be done by a very capable adversary that knows how to carry out the attack so that it isn't detected. Perhaps we'll be in the world where such adversaries don't exist.

Implicit extortion

Explicit extortion occurs when an attacker makes an explicit threat to harm you if you don't comply with their demands. In contrast, in implicit extortion, the attacker always harms you if you don't do the thing that they want, which leads you to learn over time to do what the attacker wants. Implicit extortion seems particularly hard to deal with because you may not know it is happening.


I'm a fan of executable code that demonstrates the problems that we are worrying about -- it makes the concept (in this case, a treacherous turn) more concrete. In order to make it more realistic, I would want the agent to grow in capability organically (rather than simply getting a more powerful weapon). It would really drive home the point if the agent undertook a treacherous turn the very first time, whereas in this post I assume it learned using many episodes of trial-and-error that a treacherous turn leads to higher reward. This seems hard to demonstrate with today's ML in any complex environment, where you need to learn from experience instead of using eg. value iteration, but it's not out of the question in a continual learning setup where the agent can learn a model of the world.

A Gym Gridworld Environment for the Treacherous Turn

An example Gym environment in which the agent starts out "weak" (having an inaccurate bow) and later becomes "strong" (getting a bow with perfect accuracy), after which the agent undertakes a treacherous turn in order to kill the supervisor and wirehead.


Environmental Structure Can Cause Instrumental Convergence

We have <@previously seen@>(@Seeking Power is Provably Instrumentally Convergent in MDPs@) that if you are given an optimal policy for some reward function, but are very uncertain about that reward function (specifically, your belief assigns reward to states in an iid manner), you should expect that the optimal policy will navigate towards states with higher power in some but not all situations. This post generalizes this to non-iid reward distributions: specifically, that "at least half" of reward distributions will seek power (in particular circumstances).

The new results depend on the notion of _environment symmetries_, arising in states in which an action a2 leads to "more options" than another action a1 (we'll assume that a1 and a2 lead to different, disjoint parts of the state space). Specifically, a1 leads to a part of the state space that is isomorphic to a subgraph of the part of the state space that a2 leads to. For example, a1 might be going to a store where you can buy books or video games, and a2 might be going to a supermarket where you can buy food, plants, cleaning supplies, tools, etc. Then, one subgraph isomorphism would be the one that maps "local store" to "supermarket", "books" to "food", and "video games" to "plants". Another such isomorphism would instead map "video games" to "tools", while keeping the rest the same.

Now this alone doesn't mean that an optimal policy is definitely going to take a2. Maybe you really want to buy books, so a1 is the optimal choice! But for every reward function for which a1 is optimal, we can construct another reward function for which a2 is optimal, by mapping it through the isomorphism. So, if your first reward function highly valued books, this would now construct a new reward function that highly values food, and now a2 will be optimal. Thus, at least half of the possible reward functions (or distributions over reward functions) will prefer a2 over a1. Thus, in cases where these isomorphisms exist, optimal policies will tend to seek more options (which in turn means they are seeking power).

If the agent optimizes average reward (i.e. gamma is 1), then we can extend this analysis out in time, to the final cycle that an agent ends up in. (It must end up in a cycle because by assumption the state space is finite.) Any given cycle would only count as one "option", so ending up in any given cycle is not very likely (using a similar argument of constructing other rewards). If shutdown is modeled as a state with a single self-loop and no other actions, then this implies that optimal policies will tend to avoid entering the shutdown state.

We've been saying "we can construct this other reward function under which the power-seeking action is optimal". An important caveat is that maybe we know that this other reward function is very unlikely. For example, maybe we really do just know that we're going to like books and not care much about food, and so the argument "well, we can map the book-loving reward to a food-loving reward" isn't that interesting, because we assign high probability to the first and low probability to the second. We can't rule this out for what humans actually do in practice, but it isn't as simple as "a simplicity prior would do the right thing" -- for any non-power-seeking reward function, we can create a power-seeking reward function with only slightly higher complexity by having a program that searches for a subgraph isomorphism and then applies it to the non-power-seeking reward function to create a power-seeking version.

Another major caveat is that this all relies on the existence of these isomorphisms / symmetries in the environment. It is still a matter of debate whether good models of the environment will exhibit such isomorphisms.


Eight claims about multi-agent AGI safety

This post clearly states eight claims about multiagent AGI safety, and provides brief arguments for each of them. Since the post is itself basically a summary, I won't go into detail here.


"Unsupervised" translation as an (intent) alignment problem

We have previously seen that a major challenge for alignment is that our models may learn <@inaccessible information@>(@Inaccessible information@) that we cannot extract from them, because we do not know how to provide a learning signal to train them to output such information. This post proposes unsupervised translation as a particular concrete problem to ground this out.

Suppose we have lots of English text, and lots of Klingon text, but no translations from English to Klingon (or vice versa), and no bilingual speakers. If we train GPT on the text, it will probably develop a good understanding of both English and Klingon, such that it "should" have the ability to translate between the two (at least approximately). How can we get it to actually (try to) do so? Existing methods (both in unsupervised translation and in AI alignment) do not seem to meet this bar.

One vague hope is that we could train a helper agent such that a human can perform next-word prediction on Klingon with the assistance of the helper agent, using a method like the one in [Learning the prior](https://www.alignmentforum.org/posts/SL9mKhgdmDKXmxwE4/learning-the-prior) ([AN #109](https://mailchi.mp/ee62c1c9e331/an-109teaching-neural-nets-to-generalize-the-way-humans-would)).


I think that the problems of generalization and ambiguity out of distribution are extremely important and fundamental to AI alignment, so I’m glad to see work on them. It seems like model splintering could be a fruitful approach for those looking to take a more formal approach to these problems.

Model splintering: moving from one imperfect model to another

This post introduces the concept of _model splintering_, which seems to be an overarching problem underlying many other problems in AI safety. This is one way of more formally looking at the out-of-distribution problem in machine learning: instead of simply saying that we are out of distribution, we look at the model that the AI previously had, and see what model it transitions to in the new distribution, and analyze this transition.

Model splintering in particular refers to the phenomenon where a coarse-grained model is "splintered" into a more fine-grained model, with a one-to-many mapping between the environments that the coarse-grained model can distinguish between and the environments that the fine-grained model can distinguish between (this is what it means to be more fine-grained). For example, we may initially model all gases as ideal gases, defined by their pressure, volume and temperature. However, as we learn more, we may transition to the van der Waal's equations, which apply differently to different types of gases, and so an environment like "1 liter of gas at standard temperature and pressure (STP)" now splinters into "1 liter of nitrogen at STP", "1 liter of oxygen at STP", etc.

Model splintering can also apply to reward functions: for example, in the past people might have had a reward function with a term for "honor", but at this point the "honor" concept has splintered into several more specific ideas, and it is not clear how a reward for "honor" should generalize to these new concepts.

The hope is that by analyzing splintering and detecting when it happens, we can solve a whole host of problems. For example, we can use this as a way to detect if we are out of distribution. The full post lists several other examples.


As far as I could tell, the problems identified were ones that we had seen before, but I’m glad someone has gone through the more systematic exercise, and the resulting list is more organized and easier to understand than previous lists.

An Architectural Risk Analysis of Machine Learning Systems: Towards More Secure Machine Learning

Gary McGraw, Harold Figueroa, Victor Shepardson, Richie Bonett

One systematic way of identifying potential issues in a system is to perform an _architectural risk analysis_, in which you draw an architecture diagram showing the various components of the system and how they interact, and then think about each component and interaction and how it could go wrong. (<@Last week's highlight@>(@The Alignment Problem for Bayesian History-Based Reinforcement Learners@) did this for Bayesian history-based RL agents.) This paper performs an architectural risk analysis for a generic ML system, resulting in a systematic list of potential problems that could occur.


Unsurprisingly I agreed with Stuart’s responses, but nevertheless I found this illuminating, especially in illustrating the downsides of examples with simplistic goals. I did find it frustrating that Steven didn’t respond to the point about multiple goals not helping, since that seemed like a major crux, though they were discussing many different aspects and that thread may simply have been dropped by accident.

Steven Pinker and Stuart Russell on the Foundations, Benefits, and Possible Existential Threat of AI

Lucas Perry, Steven Pinker and Stuart Russell

Despite their disagreements on AI risk, Stuart and Steven agree on quite a lot. They both see the development of AI as depending on many historical ideas. They are both particularly critical of the idea that we can get general intelligence by simply scaling up existing deep learning models, citing the need for reasoning, symbol manipulation, and few-shot learning, which current models mostly don't do. They both predict that we probably won't go extinct from superintelligent AI, at least in part because we'll notice and fix any potential failures, either via extensive testing or via initial failures that illustrate the problem.

On the AI risk side, while they spent a lot of time discussing it, I'll only talk about the parts where it seems to me that there is a real disagreement, and not mention anything else. Steven's position against AI risk seems to be twofold. First, we are unlikely to build superintelligent AI soon, and so we should focus on other clear risks like climate change. In contrast, Stuart thinks that superintelligent AI is reasonably likely by the end of the century and thus worth thinking about. Second, the idea of building a super-optimizer that focuses on a single goal is so obviously bad that AI researchers will obviously not build such a thing. In contrast, Stuart thinks that goal-directed systems are our default way of modeling and building intelligent systems. It seemed like Steven was particularly objecting to the especially simplistic goals used in examples like maximizing paperclips or curing cancer, to which Stuart argued that the problem doesn't go away if you have multiple goals, because there will always be some part of your goal that you failed to specify.

Steven also disagrees with the notion of intelligence that is typically used by AI risk proponents, saying "a super-optimizer that pursued a single goal is self-evidently unintelligent, not superintelligent". I don't get what he means by this, but it seems relevant to his views.


I am curious to see people's answers to this, I think it might be a good question to reveal major differences in worldviews between optimistic and pessimistic safety researchers.

Metaphilosophical competence can't be disentangled from alignment

Would you be comfortable taking a single human, and making them a quadrillion times more powerful?


I like this framing. I'm not sure if it is actually a good model for [act-based agents](https://ai-alignment.com/act-based-agents-8ec926c79e9c), but it's another way to think about what problems could arise from an AI system that is superintelligent in some domains and subhuman in others.

Reframing misaligned AGI's: well-intentioned non-neurotypical assistants

Another way to think about problems from AGI is to imagine the AI as a well-intentioned but neuroatypical friend, who learned all about humans from Wikipedia, and who has access to _immense_ resources. You would worry a lot about principal-agent problems in such a scenario.


Specification gaming: the flip side of AI ingenuity

Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, Shane Legg

This post on the DeepMind website explains the concept of <@specification gaming@>(@Specification gaming examples in AI@), and illustrates three problems that arise within it. First and most obviously, we need to capture the human concept of a given task in a reward function. Second, we must design agents without introducing any mistaken implicit assumptions (e.g. that the physics simulation is accurate, when it isn't). Finally, we need to ensure that agents don't tamper with their reward functions.


This version of the argument seems to be more a statement about our epistemic state than about actual AI risk. For example, I know many people without technical expertise who anthropomorphize their laptops as though they were pursuing some goal, but they don't (and shouldn't) worry that their laptops are going to take over the world. More details in [this comment](https://www.lesswrong.com/posts/EnN7cm3KaRrEAuWfa/comment-on-coherence-arguments-do-not-imply-goal-directed#CsRXodmiBfZ9wCZwr).

Comment on Coherence arguments do not imply goal directed behavior

I <@have argued@>(@Coherence arguments do not imply goal-directed behavior@) that coherence arguments that argue for modeling rational behavior as expected utility maximization do not add anything to AI risk arguments. This post argues that there is a different way in which to interpret these arguments: we should only model a system to be an EU maximizer if it was the result of an optimization process, such that the EU maximizer model is the best model we have of the system. In this case, the best way to predict the agent is to imagine what we would do if we had its goals, which leads to the standard convergent instrumental subgoals.


While I appreciate and agree with the thesis of this post -- that machine learning models don't have to be extremely competent to be manipulative -- I would still prefer a different example to convince skeptical researchers. I suspect many people would reply that we could easily patch the issue without doing dedicated safety work. In particular, it is difficult to see how this strategy arises if we train the system via supervised learning rather than training it to maximize the number of articles users label unbiased (which requires RL).

**Rohin's opinion:** I certainly agree with this post that ML models don't need to be competent to be manipulative. However, it's notable that this happened via the model randomly manipulating people (during exploration) and noticing that it helps it achieve its objective. It seems likely that to cause _human extinction_ via a treacherous turn, the model would need to do zero-shot manipulation. This seems much less likely to happen (though I wouldn't say it's impossible).

An Increasingly Manipulative Newsfeed

An early argument for specialized AI safety work is that misaligned systems will be incentivized to lie about their intentions while weak, so that they aren't modified. Then, when the misaligned AIs are safe from modification, they will become dangerous. Ben Goertzel [found the argument unlikely](https://jetpress.org/v26.1/goertzel.pdf), pointing out that weak systems won't be good at deception. This post asserts that weak systems can still be manipulative, and gives a concrete example. The argument is based on a machine learning system trained to maximize the number of articles that users label "unbiased" in their newsfeed. One way it can start being deceptive is by seeding users with a few very biased articles. Pursuing this strategy may cause users to label everything else unbiased, as it has altered their reference for evaluation. The system is therefore incentivized to be dishonest without necessarily being capable of pure deception.


The more general argument underlying this post is that aligning AGI will be hard partly because ethics is hard [(as discussed here)](https://www.alignmentforum.org/posts/w6d7XBCegc96kz4n3/the-argument-from-philosophical-difficulty). I agree that using uncertain objective functions might help with this problem. However, I'm not convinced that it's useful to frame this issue in terms of impossibility theorems and narrow AI, and would like to see these ideas laid out in a philosophically clearer way.

Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)

This paper discusses some impossibility theorems related to the Repugnant conclusion in population ethics (i.e. theorems showing that no moral theory simultaneously satisfies certain sets of intuitively desirable properties). Peter argues that in the context of AI it's best to treat these theorems as uncertainty results, either by allowing incommensurate outcomes or by allowing probabilistic moral judgements. He hypothesises that "the emergence of instrumental subgoals is deeply connected to moral certainty", and so implementing uncertain objective functions is a path to making AI safer.


This is an important point about Goodhart's Law. If you take some "random" or unoptimized environment, and then try to optimize some proxy for what you care about, it will probably work quite well. It's only when the environment is already optimized that Goodhart effects are particularly bad.

Constructing Goodhart

This post makes the point that Goodhart's Law is so common in practice because if there are several things that we care about, then we are probably at or close to a Pareto-optimal point with respect to those things, and so choosing any one of them as a proxy metric to optimize will cause the other things to become _worse_, leading to Goodhart effects.


Two More Decision Theory Problems for Humans

The first problem is that any particular human's values only make sense for the current environment. When considering different circumstances (eg. an astronomically large number of very slightly negative experiences like getting a dust speck in your eye), many people will not know how to evaluate the value of such a situation.

The second problem is that for most formalizations of values or utility functions, the values are defined relative to some way of making decisions in the world, or some ontology through which we understand the world. If this decision theory or ontology changes, it's not clear how to "transfer" the values to the new version.


There’s definitely room to quibble with some of these arguments as stated, but I think this sort of argument basically works. Note that it only establishes that it is worth looking into AI risk; to justify the specific things people are doing (especially in AI alignment) you need significantly more specific and detailed arguments.

AI Risk for Epistemic Minimalists

This post makes a case for working on AI risk using four robust arguments:

1. AI is plausibly impactful because it is the first system that could plausibly have long-term influence or power _without_ using humans as building blocks.
2. The impact is plausibly concerning because in general, when humans gain power quickly (as they would with AI), this tends to increase existential risk.
3. We haven't already addressed the concern: we haven't executed a considered judgment about the optimal way to roll out AI technology.
4. It seems possible to take actions that decrease the concern, simply because there are so many possible actions that we could take; at least some of them should have some useful effect.


Acta Baltica Historiae et Philosophiae Scientiarum

I agree that successful AI alignment does not rule out the possibility of malicious use of AI systems. This paper is proposing putting rules inside the AI system that handle this problem. But as the paper itself notes, it seems quite challenging to even figure out what rules we would want.

I personally am more optimistic about the alternate route, where we put rules _outside_ the AI system to handle the problem, that is, we create laws and regulations around the use of AI, such that malicious uses can be blamed on the human operator, and enforcement mechanisms that ensure that we actually catch and punish such malicious uses.

Challenges of Aligning Artificial Intelligence with Human Values

This paper argues that since immoral humans could use AI systems to do harm, we must build ethical rules into AI systems. For this purpose, the traditional notion of "value alignment" is not enough, as it only requires that the AI system do what the user wants, which might not be ethical. But we also cannot embed a single theory of ethics into an AI system, as there is no agreement on such a theory. Instead, we should focus on what we _don't_ want an AI system to do, and _rule out_ that behavior, while remaining uncertain or agnostic on what should be done.


While there are many points that I disagree with (eg. "what [AIs] do uniquely is not thinking as heretofore conceived and experienced. Rather, it is unprecedented memorization and computation"), overall there was a surprising amount of familiar material said in a different way (such as explainability and unintended consequences).

How the Enlightenment Ends

This is an article about the dangers of AI written by a non-technologist, hitting some points that are relatively familiar.


When Bots Teach Themselves How To Cheat

A media article about specification gaming in AI that I actually just agree with, and it doesn't even have a Terminator picture!


The simple picture on AI safety

Argues that we should distill the problem of AI safety into a simple core. The author proposes it be distilled into two simple (but not easy) problems -- the technical engineering problem of how to build a safe superintelligence, and the coordination problem of how to prevent an unaligned superintelligence from being built first.


I really liked this paper: it seems like it took a real stab at trying to align today’s recommender systems, and might have made substantial progress.

I am somewhat dubious of the use of causal graphical models here: if you create a model with some conditional independence relation that then doesn’t hold in practice, your model can have some pretty bad inferences. This actually happened: when they only modeled the relationships based on Twitter’s UI elements, and in particular did not model the dependence of SLO on clicks, they were getting bad results, where clicking on a post was interpreted as evidence that the post was _not_ valuable.

As the paper mentions, we can drop the causal interpretation of the Bayes net. This lets us draw edges between more nodes in B in order to make our model more expressive and partially prevent this sort of misspecification, while also letting us express more complex relationships. For example, I think (but am not sure) that with their current graph, liking and retweeting a post would be treated as independent sources of evidence. If we drew an edge between them, you would gain the ability for the model to learn that if a user likes _and_ retweets a post, that is more than the sum of the contributions of liking and retweeting separately. Note that you still can’t connect everything to the anchor A, because they require that A has no children, and if you add parents to A, those must then be estimated by the heuristic method above that is probably not very good. So you still need to model conditional independence for A, and this may get significantly harder the more complex B becomes.

This also makes sense given the motivation: since the idea is to have information about value “flow through the anchor” to the variables in B, it seems like you shouldn’t need to worry too much about the relationships between variables in B, and it would be fine to model arbitrarily complex relationships between them. Besides dropping the causal interpretation and adding lots of edges in B, another option is to add many more features to the graph: for example, perhaps you also want to include the number of likes that the tweet gets overall, or whether or not it is currently the weekend. You do need to make sure your model does not become so expressive that it can now overfit to the dataset, e.g. “tweets that are posted on Sep 24 that get exactly 38756 likes will not be valuable to Alice”. However, the datasets they are using are presumably huge, and the current graphical models are tiny (16 nodes with low degree), so we can increase it at least somewhat before we get to that point.

Is this paper relevant to alignment of superintelligent AI systems, the topic of this newsletter? I don’t think it is that relevant, since it seems like the main technique involves us effectively hardcoding our knowledge (of the anchor variable) into the objective, in a way that makes sense for recommender systems but probably would not work for more general systems. I’m highlighting it anyway because I think it is particularly novel and interesting, it seems to solve an analog of the alignment problem for existing AI systems, and it is part of a research lineage that I do think will be relevant for alignment: how to create good objectives for AI systems. Even if you only care about alignment of superintelligent systems, it seems worth following the techniques used today and the problems that come up in their application, as the lessons learned may continue to be relevant when intelligence is scaled up.

From Optimizing Engagement to Measuring Value

This paper takes a stab at creating a better objective for existing recommender systems than engagement, in a way that could be applied at existing companies like Twitter. The basic approach is to treat the variable to be optimized (user value) as a latent variable, and use probabilistic inference to infer how likely it is that a particular recommendation was valuable.

Usually a major challenge with such an approach is specifying the _observation model_: how the observed data is caused by the latent variable. In the case of Twitter, this would require you to answer questions like, "if the user does not value a tweet, how likely is a user to hit the like button anyway?" This is a hard question to answer, since perhaps users like tweets in order to stop conversations, or because they are addicting at the moment but are not actually valuable, etc.

One simple heuristic is to take two datasets where we know one dataset has more valuable recommendations than the other. Differences in user behavior between these datasets can then be assumed to be correlations with value. The authors provide a quantitative method for inferring the observation model from such datasets, which I won't go into here since it is primarily a heuristic baseline. One obvious problem is that if the "better" dataset was produced by optimizing (say) clicks, then the clicks may have increased for reasons other than improved value, but this heuristic approach will attribute the entire increase to improved value.

How can we do better? The key insight of this paper is that if you have a bunch of historical data, then you can get a lot of mileage by identifying an _anchor_: a type of feedback that when given provides unequivocal evidence of the latent value. On Twitter, this is taken to be the "See Less Often" (SLO) button: if this is clicked, then we know with effective certainty that this was not valuable, regardless of any other actions the user took. The connection between value and other behaviors such as liking a tweet can then be inferred by looking at the connection between those behaviors and the anchor, which we can estimate from historical data.

Formally, the authors assume access to a graph describing the relationships between the various possible behaviors (almost all of which have the latent value V as a parent). One of these is identified as the anchor node A, for which P(V = 1 | A = 1) is assumed to be known and independent of all other variables. However, P(V = 1 | A = 0) is not independent of other variables: intuitively, if the SLO button is _not_ clicked, then we need to fall back to looking at other variables to estimate value.

The authors then show that under some reasonable assumptions on the anchor variable, if you have a dataset of historical data to estimate P(A, B) (where B consists of all the other tracked behaviors), then instead of specifying observation models P(B | V) for all behaviors, you only need to specify observation models for the parents of A, that is P(parents(A) | V). Everything else is uniquely determined, allowing us to calculate our final objective P(V | A, B). (There are algorithmic details on how to do this efficiently; see the paper for details.) In this case, they use the heuristic method outlined above to estimate P(parents(A) | V).

They unfortunately don't have a great way to evaluate their method: they clearly can't evaluate it by seeing if it leads to higher clicks, since the whole point was to move away from clicks as an optimization target. (I assume a user study on Twitter was infeasible.) Their primary form of evaluation is to run the model and report the learned probabilities, and show that they seem reasonable, whereas those output by a Naive Bayes model do not.


Even though I often feel like an <@optimist@>(@Conversation with Rohin Shah@) about incentives towards alignment, even I was surprised to see the amount of effort that it seems Facebook has put into trying to align its recommendation algorithm with well-being. To the extent that the recommendation algorithm is still primarily harmful (which might be true or false, idk), this suggests to me that it might just be really hard to give good recommendations given the sparse feedback you get. Of course, there are more cynical explanations, e.g. Facebook just wants to look like they care about well-being, but if they really cared they could do way better. I lean towards the first explanation, but it's very hard to distinguish between these hypotheses.

While this post claimed that narrow value alignment should be easier than AGI alignment, I'm actually not so sure. With AGI alignment, you have the really powerful assumption that the AI system you are trying to align is _intelligent_: this could plausibly help a lot. For example, maybe the recommender systems that Facebook is using are just incapable of predicting what will and won't improve human well-being, in which case narrow alignment is doomed. This wouldn't be the case with an AGI (depending on your definition of AGI) -- it should be capable of doing at least as well as humans do. The challenge is in ensuring that the AI systems are actually <@motivated@>(@Clarifying "AI Alignment"@) to do so, not whether they are capable of doing so; with narrow alignment you need to solve both problems.

Aligning AI to Human Values means Picking the Right Metrics

There has been a lot of attention recently on the flaws of recommender systems, especially when optimizing for simple metrics like engagement -- an example of what we might call "narrow value alignment". This post reconstructs how Facebook and YouTube have been incorporating better metrics into their algorithms from 2015 and 2017 respectively. For example, Facebook found that academic research suggested that well-being was improved by "meaningful social interactions", but worsened by passive consumption of content. As a result, they changed the metric for the recommendation algorithm to better track this. How did they measure it? It seems that they simply asked a survey of thousands of people what the most meaningful content was (both on and off Facebook), and used this to train a model to predict "meaningful interactions". They estimated that this resulted in a 5% decrease in time spent on Facebook, at least in the short term. The story with YouTube is similar, though sparser on details (and it's not clear if there was input from end users in YouTube's case).

The author then contrasts this sort of narrow value alignment with AGI alignment. His main take is that narrow alignment should be easier to address, since we can learn from how existing systems behave in the real world, and the insights we gain may be critical for AGI alignment. I'll end with a quote from the conclusion: "My argument is not so much that one should use AI to optimize for well-being. Rather, we live in a world where large-scale optimization is already happening. We can choose not to evaluate or adjust these systems, but there is little reason to imagine that ignorance and inaction would be better."


I really liked this paper as an example of an agenda about how to change recommender systems. It didn’t rely on armchair speculation or anecdotes to determine what problems exist and what changes should be made, and it didn’t just assume that polarization must be bad and instead considered both costs and benefits. The focus on “making conflict healthy” makes a lot of sense to me. I especially appreciated the emphasis on a strategy for evaluating particular changes, rather than pushing for a specific change; specific changes all too often fail once you test them at scale in the real world.

Designing Recommender Systems to Depolarize

This paper agrees with the post above that "available evidence mostly disfavors the hypothesis that recommender systems are driving polarization through selective exposure, aka 'filter bubbles' or 'echo chambers'". Nonetheless, social media is a huge part of today's society, and even if it isn't actively driving polarization, we can ask whether there are interventions that would decrease polarization. That is the focus of this paper.

It isn't clear that we _should_ intervene to decrease polarization for a number of reasons:

1. Users may not want to be "depolarized".
2. Polarization may lead to increased accountability as each side keeps close watch of politicians on the other side. Indeed, in 1950 people used to worry that we weren't polarized _enough_.
3. More broadly, we often learn through conflict: you are less likely to see past your own biases if there isn't someone else pointing out what they are. To the extent that depolarization removes conflict, it may be harmful.

Nonetheless, polarization also has some clear downsides:

1. It can cause gridlock, preventing effective governance.
2. It erodes norms against conflict escalation, leading to outrageous behavior and potentially violence.
3. At current levels, it has effects on all spheres of life, many of which are negative (e.g. harm to relationships across partisan lines).

Indeed, it is plausible that _most_ situations of extreme conflict were caused in part by a positive feedback loop involving polarization; this suggests that reducing polarization could be a very effective method to prevent conflict escalation. Ultimately, it is the _escalation_ and _violence_ that we want to prevent. Thus we should be aiming for interventions that don't eliminate conflict (as we saw before, conflict is useful), but rather transform it into a version that doesn't lead to escalation and norm violations.

For this purpose we mostly care about _affective polarization_, which tells you how people feel about "the other side". (In contrast, issue polarization is less central, since we don't want to discourage disagreement on issues.) The paper's central recommendation is for companies to measure affective polarization and use this as a validation metric to help decide whether a particular change to a recommender system should be deployed or not. (This matches current practice at tech companies, where there are a few high-level metrics that managers use to decide what to deploy, and importantly, the algorithms do _not_ explicitly optimize for those metrics.) Alternatively, we could use reinforcement learning to optimize the affective polarization metric, but in this case we would need to continuously evaluate the _metric_ in order to ensure we don't fall prey to Goodhart effects.

The paper also discusses potential interventions that could reduce polarization. However, it cautions that they are based on theory or studies with limited ecological validity, and ideally should be checked via the metric suggested above. Nonetheless, here they are:

1. **Removing polarizing content.** In this case, the polarizing content doesn't make it to the recommender system at all. This can be done quite well when there are human moderators embedded within a community, but is much harder to do at scale in an automated way.

2. **Changing recommendations.** The most common suggestion in this category is to increase the diversity of recommended content (i.e. go outside the "filter bubble"). This can succeed, though usually only has a modest effect, and can sometimes have the opposite effect of increasing polarization. A second option is to penalize content for incivility: studies have shown that incivility tends to increase affective polarization. Actively promoting civil content could also help. That being said, it is not clear that we want to prevent people from ever raising their voice.

3. **Changing presentation.** The interface to the content can be changed to promote better interactions. For example, when the Facebook "like" button was replaced by a "respect" button, people were more likely to "respect" comments they disagreed with.


This article is interesting because it formalizes and experimentally demonstrates an intuitive concern many have regarding recommender systems.  I also found the formalization of instrumental goals to be of independent interest. The most surprising result was that the agents who exploit tampering are not particularly more effective than policies that avoid tampering. This suggests that the instrumental incentive is not really pointing at what is actually optimal, which I found to be an illuminating distinction.

User Tampering in Reinforcement Learning Recommender Systems

Large-scale recommender systems have emerged as a way to filter through large pools of content to identify and recommend content to users. However, these advances have led to social and ethical concerns over the use of recommender systems in applications. This paper focuses on the potential for social manipulability and polarization from the use of RL-based recommender systems. In particular, they present evidence that such recommender systems have an instrumental goal to engage in user tampering by polarizing users early on in an attempt to make later predictions easier.

To formalize the problem the authors introduce a causal model. Essentially, they note that predicting user preferences requires an exogenous, non-observable variable, that models click-through rates. They then introduce a notion of instrumental goal that models the general behavior of RL-based algorithms over a set of potential tasks. The authors argue that such algorithms will have an instrumental goal to influence the exogenous/preference variables whenever user opinions are malleable. This ultimately introduces a risk for preference manipulation.

The author's hypothesis is tested using a simple media recommendation problem. They model the exogenous variable as either leftist, centrist, or right-wing. User preferences are malleable in the sense that a user shown content from an opposing side will polarize their initial preferences. In experiments, the authors show that a standard Q-learning algorithm will learn to tamper with user preferences which increases polarization in both leftist and right-wing populations. Moreover, even though the agent makes use of tampering it fails to outperform a crude baseline policy that avoids tampering.


I’m glad to see a simple summary of the evidence we currently have on the effects of social media on polarization. I feel like for the past year or two I’ve constantly heard people speculating about massive harms and even existential risks based on a couple of anecdotes or armchair reasoning, without bothering to check what has actually happened; whenever I talk to someone who seems to have actually studied the topic in depth, it seems they think that there are problems with recommender systems, but they are different from what people usually imagine. (The post also notes a reason to expect our intuitions to be misguided: we are unusual in that we get most of our news online; apparently every age group, starting from 18-24, gets more news from television than online.)

Note that there have been a few pieces arguing for these harms; I haven't sent them out in the newsletter because I don't find them very convincing, but you can find links to some of them along with my thoughts [here](https://www.alignmentforum.org/posts/TmHRACaxXrLbXb5tS/rohinmshah-s-shortform?commentId=EAKEfPmP8mKbEbERv).

How Much Do Recommender Systems Drive Polarization?

There is a common worry that social media (and recommender systems in particular) are responsible for increased polarization in recent years. This post delves into the evidence for this claim. By "polarization", we mostly mean _affective polarization_, which measures how positive your feelings towards the opposing party are (though we also sometimes mean _issue polarization_, which measures the correlation between your opinions on e.g. gun control, abortion, and taxes). The main relevant facts are:

1. Polarization in the US has increased steadily since 1980 (i.e. pre-internet), though arguably there was a slight increase from the trend around 2016.
2. Since 2000, polarization has only increased in some Western countries, even though Internet use has increased relatively uniformly across countries.
3. Polarization in the US has increased most in the 65+ age group (which has the least social media usage).

(2) could be partly explained by social media causing polarization only in two-party systems, and (3) could be explained by saying that social media changed the incentives of more traditional media (such as TV) which then increased polarization in the 65+ age group. Nevertheless, overall it seems like social media is probably not the main driver of increased polarization. Social media may have accelerated the process (for instance by changing the incentives of traditional media), but the data is too noisy to tell one way or the other.


Beyond Engagement: Aligning Algorithmic Recommendations With Prosocial Goals

To decide what item to show a user, a recommender system needs to have some metric by which to rank items. Since this metric must usually be automated, it often contains in large part some operationalization of "engagement". Unfortunately, this metric may not be able to differentiate between clickbait or extremist content on the one hand, and actually valuable posts on the other. In a workshop on the topic, participants brainstormed four main approaches for improvement:

1. **Build better controls:** Offer users more and better ways to control their feed.
2. **Develop standardized survey-based metrics:** Surveys should be able to get a significantly higher quality signal to optimize than engagement.
3. **Pay users for better data,** such as survey data.
4. **Recommend feeds, not items:** If we rank items individually, it is quite likely that all the posts of the same type (e.g. controversial posts) will get high scores. By ranking entire feeds, we can also optimize for the diversity of items within the feed.
5. **Incentivize the creation of different feeds,** so that users can choose which ones they prefer all things considered.


Both this paper and the previous post seem like meaningful progress in ideas for making recommendation engines better (though as a caveat, I don’t follow this space and so don’t know to what extent this has been said before). I’m glad that we’re getting to the point of actually proposing technical solutions; I hope to see more papers implementing such solutions (we’ve seen one from Twitter <@recently@>(@From Optimizing Engagement to Measuring Value@)).

[Aligning AI Optimization to Community Well-Being](https://link.springer.com/article/10.1007/s42413-020-00086-3) (an expanded version of <@Aligning AI to Human Values means Picking the Right Metrics@>)

What are you optimizing for? Aligning Recommender Systems with Human Values

Jonathan Stray, Steven Adler, Dylan Hadfield-Menell

While the previous blog post focused on societal-level approaches to recommender systems, this paper looks at what can be done at a technical level. By analyzing existing case studies of improvements to recommender systems (some of which we've <@seen before@>(@Aligning AI to Human Values means Picking the Right Metrics@)), the authors identify a typical approach taken in industry today.

First, engineers identify a problem with an already-deployed recommendation engine, perhaps from user feedback, or through monitoring. Second, they develop a concrete procedure to identify instances of this problem in the recommendations -- a typical approach is to curate a dataset and train an ML classifier to identify these instances, though it is also possible to use manual review. Finally, the recommender system is adjusted to avoid the problem, for example by adding a term to the objective when training the recommender system, or by filtering its outputs based on the classifier's output.

The authors then propose four high-level technical approaches to recommender alignment:
1. Develop better measures of what we want out of a recommendation engine, for example, an operationalization of "well-being" rather than "engagement".
2. Allow users to collaboratively design the recommendation engine (called _participatory design_). Rather than have a company decide on how to trade off between different objectives, allow the users to settle upon the appropriate tradeoffs themselves.
3. Interactively learn about the user's values. While this could look like building better controls as suggested in the previous post, it could also involve e.g. using <@Inverse Reward Design@> to maintain appropriate uncertainty over what the user cares about.
4. Design around "time well spent", as evaluated by users on reflection or after consideration, rather than revealed preferences or immediate judgments. For example, we could show users a summary of their activity over the past month and ask how happy they are about it.


It seems to me that this is how we'll have to solve exploration in practice if we don't want to have a huge sample complexity, though I know other researchers are optimistic about solving exploration using curiosity or diversity. It's pretty exciting that they could use a source of data that was already present in the real world.

Playing hard exploration games by watching YouTube

Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, Nando de Freitas

There are many YouTube videos demonstrating how to play levels of eg. Montezuma's Revenge. Can we use these demonstrations to solve the hard exploration tasks in Atari? One challenge is that the videos have slightly different visual properties (like color and resolution). They propose to learn a shared feature space by using an auxiliary loss where the network must predict the number of timesteps between two frames of a video, or to predict the delay between a video and audio clip from the same trajectory. Using this shared feature space, they can define a reward function that encourages the agent to take trajectories whose features match those of the demonstrations. In experiments they exceed human performance on Atari games with hard exploration problems.


I feel similarly to my opinion on <@AlphaStar@>(@AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning@) here. The result is definitely impressive and a major step up in complexity from shorter, discrete games like chess or go. However, I don’t see how the approach of just running PPO at a large scale brings us closer to AGI because we can’t run massively parallel simulations of real world tasks. Even for tasks that can be simulated, this seems prohibitively expensive for most use cases (I couldn’t find the exact costs, but I’d estimate this model cost tens of millions of dollars). I’d be quite excited to see an example of deep RL being used for a complex real world task without training in simulation.

Dota 2 with Large Scale Deep Reinforcement Learning

OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, Susan Zhang

In April, <@OpenAI Five@>(@How to Train Your OpenAI Five@) defeated the world champion Dota 2 team, OG. This paper describes its training process. OpenAI et al. hand-engineered the reward function as well as some features, actions, and parts of the policy. The rest of the policy was trained using PPO with an LSTM architecture at a massive scale. They trained this in a distributed fashion as follows:

-   The *Controller* receives and distributes the updated parameters.
-   The *Rollout Worker CPUs* simulate the game, send observations to the *Forward Pass GPUs* and publish samples to the *Experience Buffer*.
-   The *Forward Pass GPUs* determine the actions to use and send them to the *Rollout Workers*.
-   The *Optimizer GPUs* sample experience from the *Experience Buffer*, calculate gradient updates, and then publish updated parameters to the *Controller*.

The model trained over 296 days. In that time, OpenAI needed to adapt it to changes in the code and game mechanics. This was done via model "surgery", in which they would try to initialize a new model to maintain the same input-output mapping as the old one. When this was not possible, they gradually increased the proportion of games played with the new version over time.


This general idea has been proposed in robotics as well, in [Reverse Curriculum Generation for Reinforcement Learning](https://arxiv.org/abs/1707.05300), where there is a single goal state. However, in this setting we have the added benefit of perfect inverse dynamics, that is, for any action a that moves us from state s to s', we can find the inverse action a' that moves us from state s' to s. This allows the authors to start from the goal state, generate nearby states, and automatically know the value of those states (or at least a very good approximation to it). [Hindsight Experience Replay](https://arxiv.org/abs/1707.01495) also tackles similar issues -- I'd be interested to see if it could solve the Rubik's cube. Overall, the problem of sparse rewards is very difficult, and it seems like we now have another solution in the case where we have a single goal state and perfect (or perhaps just sufficiently good?) inverse dynamics.

Solving the Rubik's Cube Without Human Knowledge

Stephen McAleer, Forest Agostinelli, Alexander Shmakov, Pierre Baldi

This paper proposes _Autodidactic Iteration_ (ADI), which is a technique that can be combined with the techniques in AlphaGo and expert iteration to solve problems with only one goal state, such as the Rubik's cube. MCTS with value and policy networks will not suffice, because when starting from a randomly scrambled cube, MCTS will never find a path to the goal state, and so there will never be any reward signal. (Whereas with Go, even if you play randomly the game will end relatively quickly, giving you some reward signal.) To get around this, they start _from the goal state_ and generate states that are near the goal state. This gives them a training dataset of states for which they know (a good approximation to) the value and the best action, which they can use to train a value and policy network. They then use this with MCTS to solve the full problem, as in AlphaGo.


This is clearly a major step for model-based RL, becoming the state of the art on a very popular benchmark and enabling planning approaches to be used in domains with unknown rules or dynamics. I am typically optimistic about model-based approaches as progress towards safe AGI. They map well to how humans think about most complex tasks: we consider the likely outcomes of our actions and then plan accordingly. Additionally, model-based RL typically has the safety property that the programmers know what states the algorithm expects to pass through and end up in, which aids with interpretability and auditing. However, *MuZero* loses that property by using a learned model whose internal states are not constrained to have any semantic meaning. I would be quite excited to see follow up work that enables us to understand what the model components are learning and how to audit them for particularly bad inaccuracies.

**Rohin's opinion:** _Note: This is more speculative than usual._ This approach seems really obvious and useful in hindsight (something I last felt for [population-based training](https://deepmind.com/blog/article/population-based-training-neural-networks) of hyperparameters). The main performance benefit (that I see) of model-based planning is that it only needs to use the environment interactions to learn how the environment _works_, rather than how to _act optimally_ in the environment -- it can do the "act optimally" part using some MDP planning algorithm, or by simulating trajectories from the world model rather than requiring the actual environment. Intuitively, it should be significantly easier to learn how an environment works -- consider how easy it is for us to learn the rules of a game, as opposed to playing it well. However, most model-based approaches force the learned model to learn features that are useful for predicting the state, which may not be the ones that are useful for playing well, which can handicap their final performance. Model-free approaches on the other hand learn exactly the features that are needed for playing well -- but they have a much harder learning task, so it takes many more samples to learn, but can lead to better final performance. Ideally, we would like to get the benefits of using an MDP planning algorithm, while still only requiring the agent to learn features that are useful for acting optimally.

This is exactly what MuZero does, similarly to [this previous paper](https://arxiv.org/abs/1707.03497): its "model" only predicts actions, rewards, and value functions, all of which are much more clearly relevant to acting optimally. However, the tasks that are learned from environment interactions are in some sense "easier" -- the model only needs to predict, _given a sequence of actions_, what the immediate reward will be. It notably _doesn't_ need to do a great job of predicting how an action now will affect things ten turns from now, as long as it can predict how things ten turns from now will be _given_ the ten actions used to get there. Of course, the model does need to predict the policy and the value function (both hard and dependent on the future), but the learning signal for this comes from MCTS, whereas model-free RL relies on credit assignment for this purpose. Since MCTS can consider multiple possible future scenarios, while credit assignment only gets to see the trajectory that was actually rolled out, we should expect that MCTS leads to significantly better gradients and faster learning.

Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, David Silver

Up until now, model-free RL approaches have been state of the art at visually rich domains such as Atari, while model-based RL has excelled for games which require planning many steps ahead, such as Go, chess, and shogi. This paper attains state of the art performance on Atari using a model-based approach, *MuZero*, while matching <@AlphaZero@>(@AlphaZero: Shedding new light on the grand games of chess, shogi and Go@) at Go, chess, and shogi while using less compute. Importantly, it does this without requiring any advance knowledge of the rules of the game.

*MuZero*'s model has three components:

1. The *representation* function produces an initial internal state from all existing observations.
2. The *dynamics* function predicts the next internal state and immediate reward after taking an action in a given internal state.
3. The *prediction* function generates a policy and a value prediction from an internal state.

Although these are based on the structure of an MDP, **the internal states of the model do not necessarily have any human-interpretable meaning**. They are trained end-to-end only to accurately predict the policy, value function, and immediate reward. This model is then used to simulate trajectories for use in MCTS.


I really liked this paper: it was creative in combining conceptual components from variational methods and imitation learning, and it was pragmatic in trying to address the problem of how to get viable human-demonstration data in a way that avoids having to get distinct datasets for a huge set of different discrete tasks.

Learning Latent Plans from Play

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, Pierre Sermanet

This paper collects unsupervised data of humans playing with robotic control systems, and uses that data to thread a needle between two problems in learning. One problem is that per-task demonstration data is costly, especially as number of tasks grows; the other is that randomly sampled control actions will rarely stumble across complex motor tasks in ways that allow robots to learn. The authors argue that human play data is a good compromise because humans at play tend to explore different ways of manipulating objects in ways that give robots nuggets of useful information like "how do I move this block inside a drawer", which can be composed into more complicated and intentional tasks.

The model works by learning to produce vectors that represent plans (or sequences of actions), and jointly learning to decode those vectors into action sequences. This architecture learns to generate plan vectors by using an autoencoder-like structure that uses KL divergence to align (1) a distribution of plan vectors predicted from the start and end state of a window of play data, and (2) a distribution of plan vectors predicted by looking back at all the actions taken in that window. Because we're jointly learning to unroll the (2) lookback-summarized vector such that it matches the actions actually taken, we'll ideally end up with a system that can take in a given plan vector and produce a sequence of actions to execute that plan. And, because we're learning to predict a vector that aligns with actions successfully taken to get to an end state from a starting one, the model at test time should be able to produce a play vector corresponding to feasible actions that will get it from its current state to a goal state we'd like it to reach.  The authors found that their Play-trained model was able to outperform single-task models on a range of manipulation tasks, even though those single-task models were trained with explicit demonstrations of the task.


I really liked this paper: it's a rare instance where I actually wanted to read the theory in the paper because it felt important for getting the high level insight. The theory is particularly straightforward and easy to understand (which usually seems to be true when it leads to high level insight). The design of the algorithm seems more principled than others, and the experiments suggest that this was actually fruitful. The algorithm is probably more computationally expensive per step compared to other algorithms, but that could likely be improved in the future.

One thing that felt strange is that the proposed solution is basically to prevent generalization between (s, a) pairs, but the whole point of DQL algorithms is to generalize between (s, a) pairs since you can't get experience from all of them. Of course, since they are only preventing generalization within a batch, they still generalize between (s, a) pairs that are not in the batch, but presumably that was because they only could prevent generalization within the batch. Empirically the algorithm does seem to work, but it's still not clear to me why it works.

Towards Characterizing Divergence in Deep Q-Learning

Joshua Achiam, Ethan Knight, Pieter Abbeel

Q-Learning algorithms use the Bellman equation to learn the Q\*(s, a) function, which is the long-term value of taking action a in state s. Tabular Q-Learning collects experience and updates the Q-value for each (s, a) pair independently. As long as each (s, a) pair is visited infinitely often, and the learning rate is decayed properly, the algorithm is guaranteed to converge to Q\*.

Once we get to complex environments where you can't enumerate all of the states, we can't explore all of the (s, a) pairs. The obvious approach is to approximate Q\*(s, a). Deep Q-Learning (DQL) algorithms use neural nets for this approximation, and use some flavor of gradient descent to update the parameters of the net such that it is closer to satisfying the Bellman equation. Unfortunately, this approximation can prevent the algorithm from ever converging to Q\*.

This paper studies the first-order Taylor expansion of the DQL update, and identifies three factors that affect the DQL update: the distribution of (s, a) pairs from which you learn, the Bellman update operator, and the _neural tangent kernel_, a property of the neural net that specifies how information from one (s, a) pair generalizes to other (s, a) pairs. The theoretical analysis shows that as long as there is limited generalization between (s, a) pairs, and each (s, a) pair is visited infinitely often, the algorithm will converge. Inspired by this, they design PreQN, which explicitly seeks to minimize generalization across (s, a) pairs _within the same batch_. They find that PreQN leads to competitive and stable performance, despite not using any of the tricks that DQL algorithms typically require, such as target networks.


I don't care very much about whether AlphaStar is superhuman or not -- it clearly is very good at StarCraft at both the micro and macro levels. Whether it hits the rather arbitrary level of "top human performance" is not as interesting as the fact that it is anywhere in the ballpark of "top human performance".

It's interesting to compare this to [OpenAI Five](https://blog.openai.com/openai-five/) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13)). While OpenAI solved the exploration problem using a combination of reward shaping and domain randomization, DeepMind solved it by using imitation learning on human games. While OpenAI relied primarily on self-play, DeepMind used population-based training in order to deal with catastrophic forgetting and in order to be robust to many different strategies. It's possible that this is because of the games they were playing -- it's plausible to me that StarCraft has more rock-paper-scissors-like cyclic mechanics than Dota, and so it's more important to be robust to many strategies in StarCraft. But I don't know either game very well, so this is pure speculation.

Exploring the full strategy space rather than finding the Nash equilibrium seems like the right thing to do, though I haven't kept up with the multiagent RL literature so take that with a grain of salt. That said, it doesn't seem like the full solution -- you also want some way of identifying what strategy your opponent is playing, so that you can choose the optimal strategy to play against them.

I often think about how you can build AI systems that _cooperate_ with humans. This can be significantly harder: in competitive games, if your opponent is more suboptimal than you were expecting, you just crush them even harder. However, in a cooperative game, if you make a bad assumption about what your partner will do, you can get significantly worse performance. (If you've played Hanabi, you've probably experienced this.) Self-play does not seem like it would handle this situation, but this kind of population-based training could potentially handle it, if you also had a method to identify how your partner is playing. (Without such a method, you would play some generic strategy that would hopefully be quite robust to playstyles, but would still not be nearly as good as being able to predict what your partner does.)

[Open-ended Learning in Symmetric Zero-sum Games](https://arxiv.org/abs/1901.08106), [AMA with AlphaStar creators and pro players](https://www.reddit.com/r/MachineLearning/comments/ajgzoc/we_are_oriol_vinyals_and_david_silver_from), and [Vox: StarCraft is a deep, complicated war strategy game. Google’s AlphaStar AI crushed it.](https://www.vox.com/future-perfect/2019/1/24/18196177/ai-artificial-intelligence-google-deepmind-starcraft-game)

AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

The AlphaStar system from DeepMind has beaten top human pros at StarCraft. You can read about the particular details of the matches in many sources, such as the blog post itself, this [Vox article](https://www.vox.com/future-perfect/2019/1/24/<PHONE>/ai-artificial-intelligence-google-deepmind-starcraft-game), or [Import AI](https://jack-clark.net/2019/01/28/import-ai-131-ibm-optimizes-ai-with-ai-via-neunets-amazon-reveals-its-scout-delivery-robot-google-releases-300k-natural-questions-dataset/). The quick summary is that while there are some reasons you might not think it is conclusively superhuman yet (notably, it only won when it didn't have to manipulate the camera, and even then it may have had short bursts of very high actions per minute that humans can't do), it is clearly extremely good at StarCraft, both at the technically precise micro level and at the strategic macro level.

I want to focus instead on the technical details of how AlphaStar works. The key ideas seem to be a) using imitation learning to get policies that do something reasonable to start with and b) training a population of agents in order to explore the full space of strategies and how to play against all of them, without any catastrophic forgetting. Specifically, they take a dataset of human games and train various agents to mimic humans. This allows them to avoid the particularly hard exploration problems that happen when you start with a random agent. Once they have these agents to start with, they begin to do population-based training, where they play agents against each other and update their weights using an RL algorithm. The population of agents evolves over time, with well-performing agents splitting into two new agents that diversify a bit more. Some agents also have auxiliary rewards that encourage them to explore different parts of the strategy space -- for example, an agent might get reward for building a specific type of unit. Once training is done, we have a final population of agents. Using their empirical win probabilities, we can construct a Nash equilibrium of these agents, which forms the final AlphaStar agent. _(Note: I'm not sure if at the beginning of the game, one of the agents is chosen according to the Nash probabilities, or if at each timestep an action is chosen according to the Nash probabilities. I would expect the former, since the latter would result in one agent making a long-term plan that is then ruined by a different agent taking some other action, but the blog post seems to indicate the latter -- with the former, it's not clear why the compute ability of a GPU restricts the number of agents in the Nash equilibrium, which the blog posts mentions.)_

There are also a bunch of interesting technical details on how they get this to actually work, which you can get some information about in this [Reddit AMA](https://www.reddit.com/r/MachineLearning/comments/ajgzoc/we_are_oriol_vinyals_and_david_silver_from/). For example, "we included a policy distillation cost to ensure that the agent continues to try human-like behaviours with some probability throughout training, and this makes it much easier to discover unlikely strategies than when starting from self-play", and "there are elements of our research (for example temporally abstract actions that choose how many ticks to delay, or the adaptive selection of incentives for agents) that might be considered "hierarchical"". But it's probably best to wait for the journal publication (which is currently in preparation) for the full details.

I'm particularly interested by this [Balduzzi et al paper](https://arxiv.org/abs/1901.08106) that gives some more theoretical justification for the population-based training. In particular, the paper introduces the concept of "gamescapes", which can be thought of as a geometric visualization of which strategies beat which other strategies. In some games, like "say a number between 1 and 10, you get reward equal to your number - opponent's number", the gamescape is a 1-D line -- there is a scalar value of "how good a strategy is", and a better strategy will beat a weaker strategy. On the other hand, rock-paper-scissors is a cyclic game, and the gamescape looks like a triangle -- there's no strategy that strictly dominates all other strategies. Even the Nash strategy of randomizing between all three actions is not the "best", in that it fails to exploit suboptimal strategies, eg. the strategy of always playing rock. With games that are even somewhat cyclic (such as StarCraft), rather than trying to find the Nash equilibrium, we should try to explore and map out the entire strategy space. The paper also has some theoretical results supporting this that I haven't read through in detail.


I know that a lot of effort has gone into this project, and I expect that as a result this is probably the best educational resource on deep RL out there. The main other resource I know of is the [deep RL bootcamp](https://sites.google.com/view/deep-rl-bootcamp/), which probably supplements this resource nicely, especially with the lectures (though it is a year out of date).

Spinning Up in Deep RL

OpenAI has released an educational resource aimed to help software engineers become skilled at deep reinforcement learning. It includes simple implementations of many deep RL algorithms (as opposed to the relatively complex, highly optimized implementations in Baselines), educational exercises, documentation, and tutorials. OpenAI will host a workshop on the topic at their headquarters on Feb 2nd, and are also planning to hold a workshop at CHAI some time in early 2019.


This is a simple yet powerful idea which significantly reduces the number of samples required for good performance; I'm excited to see follow-up work. There are some confusing references in the paper to applications of MB-MPO to real-world robotics; it's unclear whether the authors already have results in this direction.

Model-Based Reinforcement Learning via Meta-Policy Optimization

Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, Pieter Abbeel

This paper introduces a new approach to model-based RL, called Model-Based Meta-Policy-Optimisation (MB-MPO), which doesn't require the dynamics models to be as accurate. It does so by learning an ensemble of dynamics models each trained on different subsets of the data, and then using meta-learning (specifically MAML) to find a policy which adapts well to any of these models within one step of gradient descent. This approach is a form of regularisation of policy learning, and achieves much greater sample efficiency without compromising performance: MB-MPO does just as well as top model-free algorithms in various Mujoco continuous-control environments, while requiring between 10 and 100 times fewer samples. Experiments suggest that it does so by having higher plasticity in regions with high dynamics model uncertainty. See also [Import AI](https://jack-clark.net/2018/09/25/import-ai-113-why-satellitesai-gives-us-a-global-eye-industry-pays-academia-to-say-sorry-for-strip-mining-it-and-kindred-researchers-seek-robot-standardization/).


This is an impressive result, since it's the first time a single agent has performed so well on a range of Atari games. It doesn't seem to have required any novel techniques except for a straightforward extension of PopArt to the multi-task setting, but this is still interesting since the results from the previous PopArt paper were very mixed (performance increased and decreased dramatically on different games, with the average remaining roughly stable).

One confusing aspect was that PopArt still benefitted slightly from being trained with reward clipping (110% vs. 101% in the unclipped case), even though the point of PopArt was to normalize rewards so that clipping wasn't necessary. I'm assuming the clipping happens after PopArt normalization, since if it happens before then you lose information as in the Pacman example. In this case, maybe it's that the reward distribution is fat-tailed, and so even after normalization there could be some extreme rewards that after normalization are still large enough that they would cause updates that are too large, and clipping alleviates this problem.

Preserving Outputs Precisely while Adaptively Rescaling Targets

When an agent is trained on multiple tasks across which the sizes of rewards vary greatly, it usually focuses on tasks which provide the largest or most frequent rewards at the expense of performance on others. Previous work dealt with this by clipping rewards outside a certain range, but this changes the optimal policy (eg. in Pacman, eating pellets is just as rewarding as eating ghosts). This paper uses PopArt (introduced in [this 2016 paper](https://arxiv.org/abs/1602.07714)) to normalise rewards from each task before using them to update the policy in an actor-critic RL algorithm. The authors use PopArt to train a single IMPALA agent which can play all 57 Atari games, achieving a median performance slightly higher than human performance.

To delve into more detail about PopArt, let's consider training a policy with an actor-critic algorithm. In this case, we need a critic that produces estimates of values V, and an actor that produces probabilities of actions. Both of these networks are trained by taking gradients of their outputs, and weighting them based on the observed rewards. Now, a key empirical fact about deep learning is that it works better if all the things are normalized, especially the gradients. (If nothing else, it makes it easier to choose the learning rate.) For the actor, this is easy -- probabilities are already normalized, and the weight of gradient is proportional to the reward, so we can just rescale the weight of the gradient based on the mean and standard deviation of the rewards we have observed so far. This is a bit harder for the critic, since it has to predict values, so we have to normalize both the outputs and the gradient weights. We can normalize the gradient weights in the same way as before. However, normalizing the outputs is tricky, because as time goes on the means and standard deviations change. To do this, at every timestep we modify the weights of the critic that is equivalent to unnormalizing based on the old statistics and then normalizing based on the new statistics. This gives the PopArt method.

Here's a simple example where I butcher types, ignore the difference between states and trajectories, and throw away the standard deviation. Suppose the first reward we see is 10, so we say that our mean is 10 and train our net to output a normalized reward of 0 for this state and action. Then, we see a reward of 100, so we update our mean to 55. On our previous (state, action) pair, we still output a normalized reward of 0, which now corresponds to a real reward of 55, even though it should correspond to 10! We then do the unnormalize-and-renormalize trick. After unnormalization, the critic would output 10, and after renormalization, the network would output -45, which when combined with the mean of 55 would give us the desired 10 reward.