AI Alignment Research Overview

---
identifier: 045a6d13-a46c-4743-aaf8-cd1490df6e98
language: en
title: AI Alignment Research Overview \[Oct 2019\]
---

[]{#AIAlignmentResearchOverviewOct2019.xhtml}

[AI Alignment Research Overview]{.c15} {#AIAlignmentResearchOverviewOct2019.xhtml#h.n1wk9bxo847o .c28}
--------------------------------------

[Jacob Steinhardt]{.c4}

Introduction {#AIAlignmentResearchOverviewOct2019.xhtml#h.mwq7iwlo7m1q .c20}
------------

This document gives an overview of different areas of technical work that seem necessary, or at least desirable, for creating safe and aligned  AI systems. The focus is on safety and alignment of powerful [ AI systems, i.e. systems that may exceed human capabilities in a broad variety of domains, and which likely act on a large scale. Correspondingly, there is an emphasis on approaches that seem scalable to such systems.]{.c4}

[]{.c4}

By "aligned", I mean that the actions it pursues move the world towards states that humans want, and away from states that humans don't want.  Some issues with this definition are that different humans might have different preferences (I will mostly ignore this issue), and that there are differences between stated preferences, "revealed" preferences as implied by actions, and preferences that one endorses upon reflection (I won't ignore this issue).

[]{.c4}

I think it is quite plausible that some topics are missing, and I welcome comments to that regard. My goal is to outline a critical mass of topics in enough detail that someone with knowledge of ML and some limited familiarity with AI alignment as an area ^[\[1\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt1){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref1}^ [ would have a collection of promising research directions, a mechanistic understanding of why they are promising, and some pointers for what work on them might look like.]{.c4}

[]{.c4}

To that end, below I outline four broad categories of technical work: [technical alignment]{.c6}  (the overcoming of conceptual or engineering issues needed to create aligned AI), [detecting failures ]{.c6} (the development of tools for proactively assessing the safety/alignment of a system or approach), [methodological understanding]{.c6}  (best practices backed up by experience), and [system-building]{.c6} [ (how to tie together the three preceding categories in the context of many engineers working on a large system). These are described in more detail in the next section.]{.c4}

[]{.c4}

[In each section I give examples of problems we might want to solve. I imagine these in the context of future powerful AI systems, which means that most of the concrete scenarios are speculative, vague, and likely incorrect if interpreted as a prediction about the future. If I were to give the strongest justification for the research topics below, I would instead focus on near-future and existing systems, which already exhibit many of the issues I discuss. Nevertheless, I think this imaginative exercise can be helpful both for stimulating research and for keeping the focus on scalable solutions.]{.c4}

[]{.c4}

[Caveats.]{.c6}  I found it difficult to write a research overview of a field as nascent as AI alignment, as anything I could write sounded either too authoritative relative to my confidence, or so full of caveats and qualifications as to be unreadable. I settled for eliding many of the qualifications and providing this single caveat up front: that this document reflects an imperfect snapshot of my  current thinking, that it expresses many ideas more sloppily than I would usually feel comfortable putting into writing, and that I hope readers will forgive this sloppiness in the service of saying [something]{.c12} [ about a topic that I feel is important.]{.c4}

[]{.c4}

This document is not meant to be a description of [my personal interests]{.c12} [, but rather of potentially promising topics within a field I care about. My own interests are neither a subset nor superset of the topics in this document, although there is high overlap. Even confined to AI alignment, this document is out-of-date and omits some of my recent thinking on economic aspects of ML.]{.c4}

[]{.c4}

[Finally, I make a number of claims below about what research directions I think are promising or un-promising. Some of these claims are likely wrong, and I could even imagine changing my mind after 1 hour of conversation with the right person. I decided that this document would be more informative and readable if I gave my unfiltered take (rather than only opinions I thought I would likely defend upon consideration), but the flip side is that if you think I'm wrong about something, you should let me know!]{.c4}

[]{.c4}

[Categories of technical work]{.c15} {#AIAlignmentResearchOverviewOct2019.xhtml#h.skxebvaouvc4 .c20}
------------------------------------

[In this document, I will discuss four broad categories of technical work:]{.c4}

[]{.c4}

[ [Technical alignment problem.](#AIAlignmentResearchOverviewOct2019.xhtml#h.3yux0ysd0o07){.c0} ]{.c1 .c6} [ ]{.c6} Research on the "technical alignment problem" either addresses conceptual obstacles to making AI aligned with humans (e.g. robustness, reward mis-specification),  or creates tools and frameworks that aid in making AI [aligned (e.g. scalable reward generation).]{.c4}

[]{.c4}

[ [Detecting failures in advance.](#AIAlignmentResearchOverviewOct2019.xhtml#h.bcg9ehclyjjp){.c0} ]{.c1 .c6} [ Independently of having solved various alignment problems, we would like to have ways of probing systems / blueprints of systems to know whether they are likely to be safe. Example topics include interpretability, red-teaming, or accumulating checklists of failure modes to watch out for.]{.c4}

[]{.c4}

[ [Methodological understanding.](#AIAlignmentResearchOverviewOct2019.xhtml#h.gelap16ik8p5){.c0} ]{.c1 .c6}   There is relatively little agreement or first-hand knowledge of how to make systems aligned or safe, and even less about which methods for doing so will scale to very powerful AI systems. I am personally skeptical of our ability to get alignment right based on purely abstract arguments without also having a lot of methodological experience, which is why I think work in this category is important. An example of a methodology-focused document is Martin Zinkevich's [ [Rules of Reliable ML](https://www.google.com/url?q=http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf&sa=D&source=editors&ust=1646871528082273&usg=AOvVaw2pHGYIo_Wk4vUlcrHJ9Yi9){.c0} ]{.c1} [, which addresses reliability of existing large systems.]{.c4}

[]{.c4}

[ [System-building.](#AIAlignmentResearchOverviewOct2019.xhtml#h.aazn4b1tanc){.c0} ]{.c1 .c6}  It is possible that building powerful AI will involve a large engineering effort (say, 100+ engineers, 300k+ lines of code ^[\[2\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt2){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref2}^ [). In this case we need a framework for putting many components together in a safe way.]{.c4}

[]{.c4}

[Each category has its own section (and sometimes subsections for more specific sub-areas) where I describe why we want to solve the problem(s) in that category, different things one could mean by "solving the problem" and which ones I am most excited about, some short descriptions of concrete research questions, and some brief pointers to related work.]{.c4}

[]{.c4}

[One might also care about forecasting, AI policy, or higher-level questions around field-building and culture. These are all important but not addressed in this document.]{.c4}

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.tvb3weyiwyyw}

[Technical alignment problem]{.c15} {#AIAlignmentResearchOverviewOct2019.xhtml#h.3yux0ysd0o07 .c20}
-----------------------------------

[We would ideally like to build AI that acts according to some specification of human values, and that is robust both to errors in the specification and to events in the world. To achieve this robustness, the system likely needs to represent uncertainty about both its understanding of human values and its beliefs about the world, and to act appropriately in the face of this uncertainty to avoid any catastrophic events.]{.c4}

[]{.c4}

[I split the technical alignment problem correspondingly into four sub-categories:]{.c4}

[]{.c4}

[ [Scalable reward generation.](#AIAlignmentResearchOverviewOct2019.xhtml#h.l5anuokbtii){.c0} ]{.c1 .c6} [ ]{.c6} Powerful [ AI systems will potentially have to make decisions in situations that are foreign to humans or otherwise difficult to evaluate\-\--for instance, on scales far outside human experience, or involving subtle but important downstream consequences. Since modern ML systems are primarily trained through human-labeled training data (or more generally, human-generated reward functions), this presents an obstacle to specifying which decisions are good in these situations. Scalable reward generation seeks to build processes for generating a good reward function.]{.c4}

[]{.c4}

[ [Reward learning.](#AIAlignmentResearchOverviewOct2019.xhtml#h.xz2ufqknm62){.c0} ]{.c1 .c6} [ ]{.c6} [Many autonomous agents seek to maximize the expected value of some reward function (or more broadly, to move towards some specified goal state / set of states). Optimizing against the reward function in this way can cause even slight errors in the reward to lead to large errors in behavior\--typically, increased reward will be well-correlated with human-desirability for a while, but will become anti-correlated after a point. Reward learning seeks to reason about differences between the observed (proxy) reward and the true reward, and to converge to the true reward over time.]{.c4}

[]{.c4}

[ [Out-of-distribution robustness](#AIAlignmentResearchOverviewOct2019.xhtml#h.r9qxvavhaw9){.c0} ]{.c1 .c6} [ ]{.c6} [is the problem of getting systems to behave well on inputs that are very different from their training data. This might be done by a combination of transfer learning (so the system works well in a broader variety of situations) and having more uncertainty in the face of unfamiliar/atypical inputs (so the system can at least notice where it is likely to not do well).]{.c4}

[]{.c4}

[ [Acting conservatively.](#AIAlignmentResearchOverviewOct2019.xhtml#h.wrd99evb3row){.c0} ]{.c1 .c6} [ ]{.c6} [Safe outcomes are more likely if systems can notice situations where it is unclear how to act, and either avoid encountering them, take actions that reduce the uncertainty, or take actions that are robustly good. This would, for instance, allow us to specify an ambiguous reward function that the system could clarify as needed, rather than having to think about every possible case up-front. ]{.c4}

[]{.c4}

Acting conservatively interfaces with reward learning and out-of-distribution robustness, as the latter two focus on noticing uncertainty while the former focuses on what to do [given]{.c12}  the uncertainty. Unfortunately, current methods for constructing uncertainty estimates seem inadequate to drive such decisions, and even given a good uncertainty estimate little work has been done on how the system should use it to shape its actions.

[]{.c4}

[A toy framework. ]{.c6} Conceptually [, it may be useful to think in terms of the standard rational agent model, where an agent has a value function or utility function V, and beliefs P, and then takes actions A that maximize the expected value of V under P (conditioned on the action A). Failures of alignment could come from incorrect beliefs P, or a value function V that does not lead to what humans want. Out-of-distribution robustness seeks to avoid or notice problems with P, while scalable reward generation seeks to produce accurate information about some value function V that is aligned with humans. Reward learning seeks to correct for inaccuracies in the reward generation process, as well as the likely limited amount of total data about rewards. Finally, acting conservatively takes into account the additional uncertainty due to acting out-of-distribution and having a learned reward function, and seeks to choose actions in a correspondingly conservative manner.]{.c4}

[]{.c4}

[ ![](images/image1.png) ]{style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 234.67px;"}

[]{.c4}

[In an RL setting where we take actions via a learned policy, we can tell the same story but with a slightly modified diagram. Instead of an action A we have a learned policy θ, and instead of P\* and \\tilde{P} denoting beliefs, they denote distributions over environments (P\* is the true on-policy environment at deployment time, while \\tilde{P} is the distribution of training environments).]{.c4}

[]{.c4}

[Other topics. ]{.c6} Beyond the topics above, the problem of [ [counterfactual reasoning](#AIAlignmentResearchOverviewOct2019.xhtml#h.oj6ckreyf3ff){.c0} ]{.c1 .c6} [ cuts across multiple categories and seems worth studying on its own. There may be other important categories of technical work as well.]{.c4}

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.qvpzi1f3el5s}

### [Out-of-distribution Robustness]{.c22} {#AIAlignmentResearchOverviewOct2019.xhtml#h.r9qxvavhaw9 .c16}

[The]{.c6} [ Problem]{.c5}

[Out-of-distribution robustness is the problem of getting systems to behave well on inputs that are very different from their training data. This might be done by a combination of transfer learning (so the system works well in a broader variety of situations) and having more uncertainty in the face of unfamiliar/atypical inputs (so the system can at least notice where it is likely to not do well).]{.c4}

[ ]{.c4}

[Why We Want to Solve It]{.c5}

Powerful  AI will likely take actions that bring it into regimes very different from what it saw at training time. Reasons for this include ^[\[3\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt3){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref3}^ :

1.  [Creating new technologies enabling capabilities that didn't exist at training time.]{.c4}
2.  [Going from being less capable than humans to being more capable than humans.]{.c4}
3.  [Going from actions/consequences that humans can reasonably understand/interpret to ones that they cannot.]{.c4}
4.  [Potential competition with other AIs.]{.c4}

```{=html}
<!-- -->
```
1.  [Additionally, feedback effects with other AIs. If one AI does something really weird, it could potentially cause a cascade.]{.c4}

```{=html}
<!-- -->
```
5.  [More generally, the world tends to change. Maybe there is a natural disaster, or a viral youtube video with the same name as a government organization.]{.c4}

[]{.c4}

[Without out-of-distribution robustness, each of the above changes could lead to failures:]{.c4}

1.  [An AI fails to predict bad consequences of some technology (doesn't realize that some useful but dangerous synthetic organism will lead to a bad outcome), or predicts a consequence but incorrectly thinks it is good (doesn't realize that genetically engineering humans to be happy all the time is not universally endorsed).]{.c4}
2.  When an AI is less powerful than humans, (A) "do what humans want" and (B) "cause the reward input by humans to be high" lead to the same actions, so the AI could equally well generalize according to A or B. When an AI is more powerful than humans, if it generalized according to B it would seize control over the reward input [.]{.c4}
3.  Similarly to the above, the AI could (A) take actions that humans would endorse, or (B) take actions whose consequences appear superficially good to humans. See discussion starting at "Strategy \#2" in [ [this](https://www.google.com/url?q=https://ai-alignment.com/an-unaligned-benchmark-b49ad992940b&sa=D&source=editors&ust=1646871528099549&usg=AOvVaw3KWM6MTYByOz5P-n-LD2XG){.c0} ]{.c1} [ article.]{.c4}
4.  In a multi-polar world where there are multiple powerful AIs competing with each other, each AI has an incentive to drive other AIs out-of-distribution (assuming that the other AIs perform poorly out-of-distribution). Additionally, while we might hope that if one AI does something erratic the other AIs would respond to neutralize the erraticness, it could instead lead to a feedback loop where all of the AIs behave erratically due to being out-of-distribution.
5.  [There are probably additional failures modes and it seems desirable in the abstract for AI to be robust to changes in distribution.]{.c4}

[]{.c4}

[What A Solution Looks Like]{.c6}

Ideally, a system solving the out-of-distribution robustness problem would [:]{.c4}

-   [Make good predictions in novel situations and anticipate potential novel consequences of actions (as in the synthetic organism example).]{.c4}
-   Notice when multiple generalizations are supported by the data and act ^[\[4\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt4){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref4}^ [ to safely disambiguate between them (as in the A vs. B scenarios above).]{.c4}
-   [Be robust to strategic or adversarial inputs to the system.]{.c4}

It is possible that we can make do without some of these, or that some of these are impossible or infeasible in full generality [. My current guess is that each of these is in expectation quite useful for improving safety and alignment, and that at least some important subclass of each problem can be solved.]{.c4}

[]{.c4}

Traditionally, terms like "distributional shift", "transfer learning", and "domain adaptation" point to a cluster of problems, some of which seem safety-relevant and some of which do not. I am primarily interested in [detecting]{.c6}  changes in distribution that might compromise performance, and devising training methods that make systems [robust]{.c6}  to a larger family of inputs. I am less interested in methods that decrease the cost of training in new domains (e.g. word vectors, image vectors, the [ [retro contest](https://www.google.com/url?q=https://contest.openai.com/2018-1/&sa=D&source=editors&ust=1646871528104084&usg=AOvVaw0vM2OYUPGe1FvdqdQdUBOA){.c0} ]{.c1} ), since these primarily make it cheaper to train new AI systems, rather than preventing the failure of a deployed system . I am also less interested in methods that transfer a capability from one situation to a new situation (e.g. [ [sim2real](https://www.google.com/url?q=https://people.eecs.berkeley.edu/~pabbeel/publications-sim2real.html&sa=D&source=editors&ust=1646871528104913&usg=AOvVaw1TXkE8mGEfB8D253TWFRFW){.c0} ]{.c1} ), although there are exceptions\-\--sim2real could potentially be used to harden a system across a wide range of inputs in simulation, which could then be transferred to the real world. This is useful because it is about [transferring robustness]{.c12}  (which may be easier to achieve in simulation) rather than   [transferring capabilities]{.c12} [.]{.c4}

[]{.c4}

There are some methods that impart robustness to a specific type of distributional change (such as "strategic agents" in the case of algorithmic game theory, or "L-infinity perturbations" in the case of some adversarial examples research). I am excited about these methods as a starting point, but  we should keep in mind the requirement of scalability\-\--we eventually want to scale to much broader families of distributional changes and have arguments for having achieved sufficient robustness for the system to be safe. Relatedly, one proposed approach is to explicitly train the system on a wide variety of situations in the hope that this will impart robustness.  This approach may work, but it should be accompanied by arguments for why it provides sufficient safety in unanticipated situations, and we should develop methods to empirically evaluate whether this approach can yield systems that generalize to radically new situations.

[]{.c4}

[Possible Directions]{.c5}

For [detecting]{.c6}  changes, some concrete problems  (not especially selected) are [ [unsupervised risk estimation](https://www.google.com/url?q=https://arxiv.org/abs/1606.05313&sa=D&source=editors&ust=1646871528108406&usg=AOvVaw3wttVCRH4soHtp7EBTM-wt){.c0} ]{.c1} , [ [detecting out-of-domain inputs](https://www.google.com/url?q=https://arxiv.org/pdf/1610.02136.pdf&sa=D&source=editors&ust=1646871528109536&usg=AOvVaw3kzgK-eR5QImgrFRVcOAur){.c0} ]{.c1} , and [ [change-point detection](https://www.google.com/url?q=https://www.sciencedirect.com/science/article/pii/0005109888900738&sa=D&source=editors&ust=1646871528110227&usg=AOvVaw1SNJfmr1moXlzza4gxp8UJ){.c0} ]{.c1} . More broadly, it seems useful to have systems that have good estimates of uncertainty that continue to be valid even on out-of-distribution examples . For any of these tasks, we wish to accomplish them with high reliability, little to no human input, and in a way that scales to large, powerful, and complex systems [.]{.c4}

[]{.c4}

For being [robust]{.c6}  to changes, some concrete problems are [ [unsupervised transfer learning](https://www.google.com/url?q=http://proceedings.mlr.press/v15/blitzer11a/blitzer11a.pdf&sa=D&source=editors&ust=1646871528111724&usg=AOvVaw3ASXNtH6h3dv5RizRE5hVW){.c0} ]{.c1} , as well as certain types of [ [robust optimization](https://www.google.com/url?q=https://arxiv.org/abs/1010.5445&sa=D&source=editors&ust=1646871528112175&usg=AOvVaw2tWgDALKhRM7u_JsKQcAxH){.c0} ]{.c1} , and certain work on [ [adversarial examples](https://www.google.com/url?q=https://drive.google.com/file/d/1T0yiu9LPv_Qh-qYhYFLj9dxjnkca8fkG/view&sa=D&source=editors&ust=1646871528112629&usg=AOvVaw2EQojKmkYE0gdLgr7Gxm2A){.c0} ]{.c1} . Research should aim to eventually scale to very broad families of perturbations\--see e.g. [ [these](https://www.google.com/url?q=https://arxiv.org/abs/1807.01697&sa=D&source=editors&ust=1646871528113039&usg=AOvVaw2e3m3BqlQNIX1-Z693eslY){.c0} ]{.c1}   [ [papers](https://www.google.com/url?q=https://arxiv.org/abs/1808.08750&sa=D&source=editors&ust=1646871528113365&usg=AOvVaw1ANMq9jO3GNNiUHdso7B55){.c0} ]{.c1}  on stochastic perturbations and [ [these](https://www.google.com/url?q=https://arxiv.org/abs/1809.08352&sa=D&source=editors&ust=1646871528113802&usg=AOvVaw0IIoLuKFBtCQrcVwPte48D){.c0} ]{.c1}   [ [two](https://www.google.com/url?q=https://arxiv.org/abs/1908.08016&sa=D&source=editors&ust=1646871528114259&usg=AOvVaw37PgXZklAasYHyM8hH4QvF){.c0} ]{.c1}  on adversarial perturbations, which are a starting point in this direction . [ [Causal inference](https://www.google.com/url?q=http://jmlr.org/papers/v14/bottou13a.html&sa=D&source=editors&ust=1646871528114779&usg=AOvVaw2kPqVXYhmK8LfzNqDgowE1){.c0} ]{.c1}  is also relevant, and [ [algorithmic game theory](https://www.google.com/url?q=https://arxiv.org/abs/1710.07887&sa=D&source=editors&ust=1646871528115211&usg=AOvVaw1WtsNkPp2ODZ-ZD0yoUKxN){.c0} ]{.c1}  may help as well (insofar as it outlines a specific class of perturbations and methods for being robust to them). Note however that many approaches are limited to either  specific narrow forms of robustness, or can only impart robustness to very small perturbations . Different researchers will have differing intuitions about which of these approaches can scale to richer forms of robustness, but it is important to keep this in mind as the end goal. Similarly, empirical work should make sure that experiments are testing for scalability  by including large perturbations as well as perturbation types that were not explicitly anticipated at training time [.]{.c4}

[]{.c4}

For both detection and robustness, many difficulties disappear when the model is [ [well-specified](https://www.google.com/url?q=https://jsteinhardt.wordpress.com/2017/01/10/latent-variables-and-model-mis-specification/&sa=D&source=editors&ust=1646871528116928&usg=AOvVaw2zsSW6w3f0YIWt4-jt4c-o){.c0} ]{.c1}  (meaning that nature's distribution lies within the model family) , but it is not clear that we can expect to have well-specified models as systems scale up [. Approaches should either grapple with model mis-specification (either theoretically or by running experiments that test behavior under mis-specification), or argue that a particular model family should be scalably well-specified and show how to leverage this.]{.c4}

[]{.c4}

[My Personal Take (as of May 2019)]{.c5}

[For detecting changes, I currently feel that building appropriate ensembles of models and looking at when they start to disagree with each other could be a good and robust way of detecting changes. However, how the ensemble is built is important\-\--I do not think that simply training multiple models with different random initialization will provide enough diversity to do a good job.]{.c4}

[]{.c4}

For robustness, I am excited by my [ [current line of work](https://www.google.com/url?q=https://cs.stanford.edu/~jsteinhardt/publications/thesis/paper.pdf&sa=D&source=editors&ust=1646871528119226&usg=AOvVaw15DJznUGeRyw_1heoA60NT){.c0} ]{.c1}  on robustness to training data corruptions in high dimensions, which designs procedures handling perturbations whose size remains constant even as the dimension grows (in contrast, most older work can only handle fairly small perturbations when the dimension is large). While this focuses on training-time robustness rather than test-time robustness, the two seem related to me. For instance, under distributional shift we can think of the shifted distribution as the "real" distribution and the original training distribution as a "corrupted" version of this real distribution (see my [ [class notes](https://www.google.com/url?q=https://www.stat.berkeley.edu/~jsteinhardt/stat260/index.html&sa=D&source=editors&ust=1646871528120092&usg=AOvVaw0QKa4uD1XmyktU4AKP3V6o){.c0} ]{.c1}  or [ [this](https://www.google.com/url?q=https://arxiv.org/abs/1909.08755&sa=D&source=editors&ust=1646871528120492&usg=AOvVaw2EWQFV-Ljgg4gVfLKZtgdw){.c0} ]{.c1} [ recent paper).]{.c4}

[]{.c4}

[I am also excited by methodological work that measures robustness across a wide variety of perturbations and attempts to understand when we should expect to get robustness to unforeseen changes. This includes some of my own work on transfer of adversarial robustness, as well as work on measuring corruption robustness (both linked above).]{.c4}

[]{.c5}

[Additional References]{.c6}

[These are not meant to be exhaustive or even representative, but include papers that are relevant to the topics above (some already linked in-text):]{.c4}

[]{.c4}

[ [Deep Anomaly Detection using Geometric Transformations](https://www.google.com/url?q=https://arxiv.org/abs/1805.10917&sa=D&source=editors&ust=1646871528122551&usg=AOvVaw1IbeaqniT9YCK-GcOySiFj){.c0} ]{.c1}  uses one-class learning to detect out-of-domain inputs.

[ [Unsupervised Risk Estimation Using Only Conditional Independence Structure](https://www.google.com/url?q=https://arxiv.org/abs/1606.05313&sa=D&source=editors&ust=1646871528123330&usg=AOvVaw0LsiNTcdwBuvw3fZrJS8Vc){.c0} ]{.c1} [ shows that the "three-view" assumption can be used to estimate the performance of ML models on new input distributions from only unlabeled data.]{.c4}

[ [Domain Adaptation from Coupled Subspaces](https://www.google.com/url?q=http://proceedings.mlr.press/v15/blitzer11a/blitzer11a.pdf&sa=D&source=editors&ust=1646871528124077&usg=AOvVaw3RmLGreRGg1_5S4Y8wcc4O){.c0} ]{.c1} [ uses a two-view assumption to do unsupervised transfer learning.]{.c4}

[ [Deep Anomaly Detection with Outlier Exposure](https://www.google.com/url?q=https://arxiv.org/abs/1812.04606&sa=D&source=editors&ust=1646871528124722&usg=AOvVaw3m3kw-d6slFR13GM7F5jXk){.c0} ]{.c1}  seeks to improve outlier detection by training on auxiliary data.

[]{.c4}

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.zftide3dzx4}

### Reward   Learning {#AIAlignmentResearchOverviewOct2019.xhtml#h.xz2ufqknm62 .c16}

[The Problem]{.c5}

Many autonomous agents seek to maximize the expected value of some reward function (or to move towards some set of specified goal states ). Unfortunately, it is difficult to specify a reward function by hand such that increasing the reward function leads to human-desirable outcomes. (Typically, increased reward will be well-correlated with human-desirability for a while, but will become anti-correlated after a point.) Reward learning is the problem of constructing processes for generating a good reward function from potentially noisy and biased information about the true reward. ^[\[5\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt5){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref5}^

[]{.c4}

[Another way of putting this is that reward learning is about receiving (information about) rewards and faithfully incorporating them into the systems\' actions, in a way that is ideally robust to getting the answer slightly wrong and is trying hard to \"do what the specifier meant\" rather than \"optimize against the literal content of the specification\".]{.c4}

[]{.c4}

[Why We Want To Solve It]{.c5}

Most easy-to-specify rewards lead to bad outcomes if maximized. For instance, "take actions that are rated highly by humans" incentivizes not just doing things that are good for humans, but also deceiving humans into thinking the action is good, as well as physically hijacking the rating process [.]{.c4}

[]{.c4}

To give a sense for how hard hand-specifying good rewards is, let's consider a limiting case. Imagine that we have a system that is in charge of optimizing the global economy. Because it produces so much economic value, we can devote many resources to providing a faithful reward signal\-\--for instance, we might have [\~1-10 million people providing rewards to the system many times a day based on detailed rubrics. Here are some ways that an AI system would be incentivized to exploit this reward signal:]{.c26 .c9}

-   [Trying to improve people's moods shortly before they provide the reward signal (or even trying to bribe people, although that carries the risk of a very negative reward from people who don't approve of bribes).]{.c26 .c9}
-   [Suppose that people in city A are more skeptical of the system and harder to satisfy than people in city B. Increasing economic prosperity of B relative to A might lead to higher reward (because it would likely increase the population of B relative to A, and thus increase the fraction of non-skeptical reward providers).]{.c9}
-   [The system would also be incentivized to hack the devices raters use to enter rewards, create Potemkin villages giving the appearance that the world is more prosperous, etc. These behaviors would be easily detected until the point that the system is powerful enough to consistently fool humans, so we might expect a tipping point where these behaviors only appear once the system is sufficiently powerful.]{.c26 .c9}

[]{.c4}

Paul Christiano talks about additional failures of alignment [ [here](https://www.google.com/url?q=https://ai-alignment.com/an-unaligned-benchmark-b49ad992940b&sa=D&source=editors&ust=1646871528130912&usg=AOvVaw0mzMpX3SeBqYTiUI6ycIKe){.c0} ]{.c1} [. Paul also wrote a ]{.c9} [ [post](https://www.google.com/url?q=https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/more-realistic-tales-of-doom&sa=D&source=editors&ust=1646871528131550&usg=AOvVaw3_9-5wqUtd4LBpXs5EJTB_){.c0} ]{.c1} [ ]{.c9} [describing]{.c9} [ how alignment failures might actually arise and cause catastrophes in practice.]{.c9}

[]{.c4}

A solution to reward learning should avoid the above failure modes. The solution likely needs a way to distinguish the actual preferences of humans in the environment from simple proxies such as "the number input into this terminal" that correlate with human preferences but can be gamed (see the above example of the global economic system). Some challenges are that "humans" and "preferences" are fuzzy concepts that seem difficult to specify by hand, humans are inconsistent in many ways, and their values probably fluctuate over time (and might not even be defined regarding far-future events) .  Moreover, there are differences between "verbally stated preferences", "revealed preferences", "urges", etc. A final issue is that different humans might have contradictory values, although I expect this to be only a small part of the problem since the values of two randomly sampled people are much closer to each other than to the value functions used in modern AI systems [.]{.c4}

[]{.c4}

We might not need to fully capture all aspects of human values (which might not even be a well-defined concept) but at minimum need to capture ideas such as "humans should still have sovereignty over what is happening". Paul Christiano discusses this [ [here](https://www.google.com/url?q=https://ai-alignment.com/ambitious-vs-narrow-value-learning-99bd0c59847e&sa=D&source=editors&ust=1646871528178773&usg=AOvVaw2ZOKlObWcEdikWuynU-bRw){.c0} ]{.c1} .

[]{.c4}

In the economic scenario above, we might hope that the system doing reward learning would understand that (Y) "the reward the human enters into the system" is a noisy proxy of (Z) "the extent to which their preferences are satisfied", and that influencing their mood affects the proxy Y but not the true value Z, and is thus undesirable because it destroys information. In circumstances where the system gains new capabilities ( e.g., it figures out how to affect which political parties are in control ), and it doesn't have enough information to know whether a given use of those capabilities would be endorsed, it should notice this and seek additional information (this is closely related to acting conservatively and out-of-distribution robustness, as it is essentially about detecting and responding to out-of-distribution inputs) .

[]{.c4}

[What A Solution Looks Like]{.c6}

The most obvious way to learn rewards in an ML framework is to generate a lot of data about what the reward should be, and build a model based on that [. However, the data about the rewards will inevitably be a proxy for what we actually care about (since we don't have access to a human's "true reward function", which is likely not even a coherent concept). We thus would ideally like a system that can learn about complex rewards that cannot be hand-coded. We desire that the system:]{.c4}

1.  Understands what   errors tend to occur when specifying the reward function
2.  [Takes actions in light of this understanding, reasoning about its reward signal being a proxy for the true reward]{.c4}
3.  Reliably seeks out information about the true reward and [ [converges to it over time](https://www.google.com/url?q=https://ai-alignment.com/corrigibility-3039e668638&sa=D&source=editors&ust=1646871528183010&usg=AOvVaw1cP1n5fAAqGB-nxkXLlEXN){.c0} ]{.c1} [.]{.c4}
4.  [Is robust to the fact that the information it seeks out will also be a proxy.]{.c4}
5.  [Avoids doing anything catastrophic from the perspective of the true reward, even in cases where different alternative rewards conflict strongly on recommended actions.]{.c4}

[]{.c4}

[This leaves aside the question of how to provide information about the reward function in the first place, which is discussed in Scalable Reward Generation.]{.c4}

[]{.c4}

In addition to the above, our solution should be scalable\--there should be minimal hand-coding involved, and it should work even for fairly sophisticated agents that can do e.g. planning and meta-learning ^[\[6\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt6){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref6}^ [.]{.c4}

[]{.c4}

[The more correct the information about the reward is, the less we need to rely on \#1 through \#5. Indeed, one could potentially obviate \#1-\#5 entirely by solving reward generation completely. Relying entirely on this feels brittle to me, but I know other people such as Paul Christiano who are more optimistic about this (note that Paul's approach, based on amplification, involves a lot of machinery to ensure that the reward continues to be sufficiently correct as the system becomes more capable; more on this below under "Alternative Approaches"). For most of the text below I will assume that we do need to care substantially about \#1-\#5 above.]{.c4}

[]{.c4}

A common framework for reward learning is inverse reinforcement learning, where we observe actions of an agent and impute the reward function that leads to those actions. However, this approach must grapple with a number of issues such as those outlined in [ [Model Mis-Specification and Inverse Reinforcement Learning](https://www.google.com/url?q=https://jsteinhardt.wordpress.com/2017/02/07/model-mis-specification-and-inverse-reinforcement-learning/&sa=D&source=editors&ust=1646871528186217&usg=AOvVaw1Ygn-hK_LkrJuY5XaG1-ut){.c0} ]{.c1} [.]{.c4}

[]{.c4}

Note that inverse reinforcement learning sometimes refers instead to a form of imitation learning , where the goal is to [ [imitate the trajectory of the agent](https://www.google.com/url?q=https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf&sa=D&source=editors&ust=1646871528187145&usg=AOvVaw0goOhqsypZgIgnJ-CRLsii){.c0} ]{.c1} . This form of inverse reinforcement learning does not seek to learn the underlying reward function; it instead posits a reward function as a nuisance variable in a model [ that predicts trajectories (note though that imitation learning may be useful for amplification, discussed below).]{.c4}

[]{.c4}

Some work tries to bypass some of the difficulties of inverse reinforcement learning by using a more direct signal of human preferences (such as directly asking humans what they want, e.g. via pairwise comparisons  \[refs. [ [1](https://www.google.com/url?q=https://arxiv.org/abs/1706.03741&sa=D&source=editors&ust=1646871528188341&usg=AOvVaw3nRpEddxQMrRv9PpZlUUUy){.c0} ]{.c1} , [ [2](https://www.google.com/url?q=http://iliad.stanford.edu/pdfs/publications/sadigh2017active.pdf&sa=D&source=editors&ust=1646871528188704&usg=AOvVaw14mdVRZF3612F9bsdmxQ56){.c0} ]{.c1} , [ [3](https://www.google.com/url?q=http://iliad.stanford.edu/pdfs/publications/biyik2018batch.pdf&sa=D&source=editors&ust=1646871528189026&usg=AOvVaw3easQ-KVhQXPn9bQGteoD1){.c0} ]{.c1} \] ).  I like this because it makes less stringent assumptions about the data\-- it only assumes that people can say what they prefer, rather than that they are (almost) perfectly rational agents [. However, I expect this signal to still have errors, so I am excited about scaling to more complex tasks where these errors are likely to occur so that we can study them (e.g. tasks that are harder for humans to specify by hand, where there are more opportunities for reward hacking, where humans have limited intuition about what the optimal policy looks like, where out-of-distribution robustness is a larger issue, etc.).]{.c4}

[]{.c4}

For both inverse reinforcement learning and learning from preferences, we want to explicitly account for the fact that the data is likely noisy/biased (per \#1-\#5 above). One line of thought is to maintain uncertainty about rewards , which can help us to be more robust to incorrect reward specification. This idea seems fruitful in principle, but it is important to be careful about what we mean by uncertainty. Even with infinite data, the directly observable rewards are only a proxy for the true reward, and so statistical uncertainty ^[\[7\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt7){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref7}^  does not suffice as a measure in the context of reward mis-specification (since it goes to zero with infinite data). Accounting for mismatch between the true reward function and whatever model family we use, or for errors from distributional shift, also is not sufficient, since the observed reward can again be a proxy even if the model family is correct  and there is no distributional shift ^[\[8\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt8){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref8}^ . We thus need something more nuanced in addition to these uncertainty measures\-\--perhaps a model of how rewards tend to be mis-specified  that was obtained via empirical investigation  (although we might worry about the scalability of this). I discuss the need for more nuanced forms of uncertainty in more detail in "Acting Conservatively".

[]{.c4}

[Possible Directions]{.c6}

Below I discuss some possible concrete directions for approaching \#1-\#5 above. I see much of this space as uncharted [ with much of the question being how to approach the problem in the first place. There are also approaches (discussed next) that question this entire framework.]{.c4}

[]{.c4}

[On \#1 (characterizing errors in specification):]{.c14}  There is some work investigating ways in which the reward signal might be biased or otherwise incorrect. For instance, D. Sculley, Leon Bottou , and their collaborators identify feedback loops as one source of this (see [ [here](https://www.google.com/url?q=https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf&sa=D&source=editors&ust=1646871528192818&usg=AOvVaw1cE-6QeTtFlI5FqriNhPhf){.c0} ]{.c1} , [ [here](https://www.google.com/url?q=https://leon.bottou.org/publications/pdf/tr-2012-09-12.pdf&sa=D&source=editors&ust=1646871528193132&usg=AOvVaw2G7Jyw-UTp4gPb-__5ZT6N){.c0} ]{.c1} , and [ [here](https://www.google.com/url?q=https://leon.bottou.org/talks/2challenges&sa=D&source=editors&ust=1646871528193405&usg=AOvVaw2G0bAtrUdt7PGPGQCyWkJB){.c0} ]{.c1} ). I suspect that industry engineers at large companies have experience handling various reward specification issues (for instance, Google and probably other companies already pay people to fill out detailed rubrics rating search results etc., and Martin Zinkevich discusses it in rule \#38 [ [here](https://www.google.com/url?q=http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf&sa=D&source=editors&ust=1646871528193779&usg=AOvVaw2vn8KJtbUKU72DYtTPX0HN){.c0} ]{.c1} ). Distilling this knowledge would be valuable [.]{.c4}

[]{.c4}

One could also perform human subject experiments on soliciting rewards or preferences to gather information about the sorts of systematic errors that humans make. I expect neither this nor distilling industry knowledge to yield a comprehensive list of potential failures of reward specification, but I still think it would be valuable by virtue of providing empirically-grounded models to work with for \#2 (right now the models used for \#2 come from the armchair), as well as providing a concrete set of tasks to work on for reward learning more generally .

[]{.c4}

Some existing work discusses errors from [ [mis-specified objective spaces](https://www.google.com/url?q=https://arxiv.org/abs/1810.05157&sa=D&source=editors&ust=1646871528194783&usg=AOvVaw3dUudy6ThrWrzBwhlkY01M){.c0} ]{.c1}  as well as [ [myopia and time-inconsistency](https://www.google.com/url?q=https://agentmodels.org/chapters/5-biases-intro.html&sa=D&source=editors&ust=1646871528195067&usg=AOvVaw0OG-_MTioJceEV6Rdx3Rab){.c0} ]{.c1} . See [ [here](https://www.google.com/url?q=https://www.fhi.ox.ac.uk/wp-content/uploads/nips-workshop-2015-website.pdf&sa=D&source=editors&ust=1646871528195353&usg=AOvVaw2uPJpNIGwb_DBj5XbR78dU){.c0} ]{.c1}  for a brief overview of possible sources of systematic error and [ [here](https://www.google.com/url?q=https://arxiv.org/abs/1512.05832&sa=D&source=editors&ust=1646871528195622&usg=AOvVaw1Pvfc6xq7nDmc3ZEEBl5Wd){.c0} ]{.c1} [ for a model and empirical study of how humans infer preferences of other humans.]{.c4}

[]{.c4}

[On \#2 (reasoning about errors):]{.c14}  There are two aspects of \#2\-\--(A)  representing our uncertainty about the true reward , and (B) acting appropriately conservatively given that uncertainty. As noted in \#1, we do not currently have good empirically grounded models for addressing A, which somewhat limits our ability to work on B. Perhaps the closest  to A is work from the cog sci direction looking at ways in which people tend to behave irrationally (see references above from \#1), and [ [inverse reward design](https://www.google.com/url?q=https://arxiv.org/abs/1711.02827&sa=D&source=editors&ust=1646871528196411&usg=AOvVaw2rcZ7yrrM3_VeKUSyKkngH){.c0} ]{.c1}  also proposes an abstract model of reward uncertainty based on assuming that the proxy reward function imputes nearly-optimal policies on the distribution it was designed for. For B , [ [robust optimization](https://www.google.com/url?q=https://pdfs.semanticscholar.org/2749/46a974bc2bbbfe89c7f6fd3751396f295625.pdf&sa=D&source=editors&ust=1646871528196852&usg=AOvVaw1bXavT-KMS8_pKPV0oQ7dk){.c0} ]{.c1}  can provide [ [conservative](https://www.google.com/url?q=http://web.hec.ca/pages/erick.delage/PreferenceRobust4.pdf&sa=D&source=editors&ust=1646871528197226&usg=AOvVaw0mAmhxOiraVXVau7yI-2yn){.c0} ]{.c1} [ [decision rules](https://www.google.com/url?q=http://web.hec.ca/pages/erick.delage/PreferenceRobust4.pdf&sa=D&source=editors&ust=1646871528197568&usg=AOvVaw3J4Ahq1cEGFk7WxS9UWpZ8){.c0} ]{.c1}  under Knightian uncertainty about the reward function  (e.g. we do not know the true reward and want to do well under all possible rewards, so we apply some [ [minimax](https://www.google.com/url?q=https://projecteuclid.org/euclid.aos/1091626173&sa=D&source=editors&ust=1646871528198062&usg=AOvVaw2FqKb3l05yVrDrPpSFhgLb){.c0} ]{.c1} [ [ decision rule](https://www.google.com/url?q=https://projecteuclid.org/euclid.aos/1091626173&sa=D&source=editors&ust=1646871528198415&usg=AOvVaw1vvmHRLmKUYuQoP87z626z){.c0} ]{.c1} ). These conservative rules that take into account worst-case uncertainty also tend to "optimize to the boundary of the space" less than non-conservative rules. There is also a large literature on risk-averse planning ( [ [example paper](https://www.google.com/url?q=https://arxiv.org/abs/1502.03919&sa=D&source=editors&ust=1646871528198940&usg=AOvVaw3v3cuJwTL1xujbnx-Heske){.c0} ]{.c1} [) although I am not sure whether the types of robustness sought there map correctly onto the reward uncertainty scenario.]{.c4}

[]{.c4}

[On \#3 (seeking out/converging to true reward):]{.c14}   Paul Christiano   discusses one formalization of this   [ [here](https://www.google.com/url?q=https://ai-alignment.com/semi-supervised-reinforcement-learning-cf7d5375197f&sa=D&source=editors&ust=1646871528199994&usg=AOvVaw0SCWBBIVshvBuyuZScVV3H){.c0} ]{.c1}  (see "Application to AI control: measuring success"), although in a different context where the assumption is that the true reward is correctly specified in principle, but very difficult to evaluate. His idea is that the agent might then be incentivized to seek out reliable proxies in the environment for the expensive-to-evaluate reward, and query those (e.g. ask people clarification questions, and try to pose the questions such that the answer is as informative as possible); it would then fall back to the expensive-to-evaluate true reward only when necessary. Shah et al. consider how [ [preferences implicit in the state of the world](https://www.google.com/url?q=https://arxiv.org/abs/1902.04198&sa=D&source=editors&ust=1646871528200563&usg=AOvVaw263fCOSNgLs3rtlvm5fBdE){.c0} ]{.c1} [ could provide such information.]{.c4}

[]{.c4}

It is less clear how to do this if we only ever get a proxy reward; one might hope to find somewhat "independent" proxies that are wrong in complementary ways , for instance asking people questions phrased in a number of different ways, or giving them different context when asking the question. Such independent proxies could be an approach to \#4 (realizing the information sought out could also be a proxy). The idea of independent proxies has been explored elsewhere in the context of multi-view learning  (see my paper on [ [unsupervised risk estimation](https://www.google.com/url?q=https://arxiv.org/abs/1606.05313&sa=D&source=editors&ust=1646871528201897&usg=AOvVaw2Uwm3oiyB6nvxHOIbdLAnA){.c0} ]{.c1}   and the references  within).

[]{.c4}

The [ [Off-Switch Game](https://www.google.com/url?q=https://arxiv.org/abs/1611.08219&sa=D&source=editors&ust=1646871528202809&usg=AOvVaw1u3kX54fFpWuVWirp0euX-){.c0} ]{.c1}  provides a simple Bayesian model in which an agent would seek out information about the true reward, and [ [Bajcsy et al.](https://www.google.com/url?q=http://proceedings.mlr.press/v78/bajcsy17a/bajcsy17a.pdf&sa=D&source=editors&ust=1646871528203326&usg=AOvVaw2eKfyesNg2RqR7izwNuW_3){.c0} ]{.c1} [ provide a model of informative feedback in human-robot interaction.]{.c4}

[]{.c4}

[On \#5 (avoiding anything catastrophic]{.c14} [):]{.c14}  One way in which different reward functions could strongly conflict is when an agent gains a new ability that allows it to hack a proxy reward at the expense of the true reward, leading to a so-called "treacherous turn" . I am interested in constructing plausible environments where something like a treacherous turn might occur (this might require the agent to be doing some sort of planning). So far most concrete examples of treacherous turns are in toy environments, e.g. [ [this](https://www.google.com/url?q=https://www.greaterwrong.com/posts/cKfryXvyJ522iFuNF/a-gym-gridworld-environment-for-the-treacherous-turn&sa=D&source=editors&ust=1646871528204774&usg=AOvVaw06vzmBVEtZk9UeuLeQNGva){.c0} ]{.c1}  or [ [this](https://www.google.com/url?q=https://www.greaterwrong.com/posts/xt5Z2Kgp8HXTRKmQf/a-toy-model-of-the-treacherous-turn&sa=D&source=editors&ust=1646871528205252&usg=AOvVaw2RIJ4djNiJ382-6o0oK6wa){.c0} ]{.c1} .

[]{.c4}

[Alternative Approaches]{.c5}

[Some work questions the need/desirability of an agent that maximizes expected reward, and proposes alternative criteria for agents to choose actions. One approach is imitation learning\--if an agent simply seeks to imitate another agent, then it seems unlikely to perform problematic optimization as long as the original agent does not.]{.c4}

[]{.c4}

Another approach is [ [quantilization](https://www.google.com/url?q=https://intelligence.org/files/QuantilizersSaferAlternative.pdf&sa=D&source=editors&ust=1646871528206755&usg=AOvVaw3jYnImjGUKjNro-jT2eRIB){.c0} ]{.c1} , which plays actions in the qth quantile of reward rather than the maximum reward (this quantile must be defined relative to some base distribution; the naive approach would be a uniform distribution over actions, while a better approach might be to take a base distribution arising from a stochastic reference policy such as imitation of a human expert).   [ [This](https://www.google.com/url?q=https://arxiv.org/abs/1611.02796&sa=D&source=editors&ust=1646871528207340&usg=AOvVaw0dhhPqRrVFp7D9fNTmihSD){.c0} ]{.c1} [ paper implements something roughly along these lines.]{.c4}

[]{.c4}

If successful, these approaches could obviate the need for some or all of \#1-6 above, but we should attend to their scalability. Quantilization in particular may not work in high-dimensional action spaces, since the qth quantile of actions is not much better than a random action unless q is exponentially small ; using a reference policy rather than a uniformly random policy as the base distribution may alleviate this, but I suspect it still runs into difficulties eventually. Similarly, straightforward imitation learning has the issue that the resulting agent is usually not better than the agent it imitates [ (although it may be faster and cheaper). Amplification, which we discuss next, provides one way of getting past this obstacle.]{.c4}

[]{.c4}

[ [Amplification](https://www.google.com/url?q=https://arxiv.org/abs/1810.08575&sa=D&source=editors&ust=1646871528208994&usg=AOvVaw12W9518wkONAYeRIgN8L1i){.c0} ]{.c1}  is a strategy for creating a reward that is correctly specified "enough" that we can train a system against it while mostly ignoring \#2-\#6. ^[\[9\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt9){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref9}^  The basic intuition behind amplification is that how correct the reward function needs to be depends on how capable the system optimizing the reward function is\--with a weak optimizer you can get by with a somewhat correct reward whereas a very strong optimizer requires an almost perfectly correct reward. Under this perspective, what we want is a way for the correctness of the reward function to scale with the capabilities of AI systems. One way of doing this is to use AI systems themselves (in conjunction with humans) to specify the reward function. An obstacle to this is how to ensure this system actually specifies the human's preferences, rather than something based on the idiosyncracies of the AI system that the human is working with. I am fairly worried about this obstacle, although Paul has written about strategies for trying to avoid this by decomposing tasks into human-answerable questions (see [ [this](https://www.google.com/url?q=https://ai-alignment.com/meta-execution-27ba9b34d377&sa=D&source=editors&ust=1646871528210197&usg=AOvVaw3vnMQu2omLefo0YUvUq9B8){.c0} ]{.c1}  blog post). Paul has more generally written extensively about amplification on his [ [blog](https://www.google.com/url?q=https://ai-alignment.com/&sa=D&source=editors&ust=1646871528210636&usg=AOvVaw0s0JJ3P465mjR0J19Pdu_R){.c0} ]{.c1} [.]{.c4}

[]{.c4}

Another approach that somewhat questions the reward learning framework is goal specification via natural language. David McAllester writes about this [ [here](https://www.google.com/url?q=https://machinethoughts.wordpress.com/2014/08/10/friendly-ai-and-the-servant-mission/&sa=D&source=editors&ust=1646871528211473&usg=AOvVaw3kSLYKlyeihmu3KObSHRey){.c0} ]{.c1} [, and at least one other NLP person who I respect seems optimistic about this idea. My impression is that not much has been done to articulate the advantages/disadvantages of this approach relative to others (or even what the approach itself would look like beyond a very high-level description). I feel excited about someone creating such an articulation.]{.c4}

[]{.c4}

Finally, while not an alternative approach, an important idea is [ [corrigibility](https://www.google.com/url?q=https://ai-alignment.com/corrigibility-3039e668638&sa=D&source=editors&ust=1646871528212456&usg=AOvVaw04k2vVf3hVbe7EVwxKohsc){.c0} ]{.c1} [, which is the idea that an agent that is "sufficiently" aligned will tend to become more aligned over time, meaning that it suffices to create agents that have goals which are in the basin of attraction of being aligned with humans, rather than being perfectly aligned from the beginning. If this intuition is correct, we could try to figure out what is required for an agent to fall into this basin, and then seek to use reward learning to hit a target in this basin. This intuition is in the background of both my \#1-\#6 above (especially \#4) as well as amplification, the dialog approach, and probably other approaches as well.]{.c4}

[]{.c4}

[My Personal Take (as of May 2019)]{.c5}

[]{.c5}

We are currently held back by the lack of a concrete way of testing different approaches to reward learning\--existing benchmarks are either gridworld or simple MuJoCo environments,  and/or based on qualitative evaluation. The opportunities for reward hacking in most of these cases seem limited \--hand-coded policies or hand-coded reward functions would plausibly suffice to solve most of the existing tasks, and I don't feel these tasks can distinguish between promising and unpromising approaches to reward learning. Consequently, I am interested in designing better tasks, which ideally have the following properties:

-   [Success and failure are clearly separated and quantitatively measurable.]{.c4}
-   [Simple baselines (such as hand-coded policies or hand-coded reward functions) do poorly.]{.c4}
-   [Many approaches succumb to clear instances of reward hacking.]{.c4}

[]{.c4}

Separately, I feel there are connections between reward learning and robustness to distributional shift. This is most clear for \#2 (reasoning about errors)\-\--often one can construct a reward-generating process that [ [comes close to giving the true reward](https://www.google.com/url?q=https://arxiv.org/abs/1711.02827&sa=D&source=editors&ust=1646871528215594&usg=AOvVaw1t1NBUmXwr4lb5jSisWQka){.c0} ]{.c1}  for a simple enough fixed distribution of examples, and the problem is that it diverges sharply once we go out-of-distribution  (as will likely happen once the system is deployed, or even earlier if we are trying to assign rewards to complex enough actions ^[\[10\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt10){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref10}^ ). Similarly, for inverse reinforcement learning the learned reward function is trained to reproduce human actions in-distribution, with the main problem being that it might not generalize well out-of-distribution  (and is moreover forced to act out-of-distribution since it has to take actions on [behalf of]{.c12}  a human, rather than [as]{.c12} [ a human).]{.c4}

[]{.c4}

Reward learning is also related to the problem of [ [robustly imputing latent variable models](https://www.google.com/url?q=https://jsteinhardt.wordpress.com/2017/01/10/latent-variables-and-model-mis-specification/&sa=D&source=editors&ust=1646871528217394&usg=AOvVaw3d4U4RDWUehVK0GKTZbYyT){.c0} ]{.c1} \-\--we can think of the true reward as a latent variable that we are trying to impute given observations of the proxy reward. My preferred way of thinking about this is that we want to impute latent variables (such as a value function) that are accurate not just on the distribution they were estimated from, but also on many other counterfactual distributions (see the section below on [ [counterfactual reasoning](#AIAlignmentResearchOverviewOct2019.xhtml#h.oj6ckreyf3ff){.c0} ]{.c1}  for how we might do this) .

[]{.c4}

Finally, I have been thinking about goal specification via natural language and am pretty excited about trying to flesh it out more concretely, but have not come up with approaches that seem plausibly scalable yet.

[]{.c4}

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.tul6bggynrvc}

### Scalable   [Reward Generation]{.c22} {#AIAlignmentResearchOverviewOct2019.xhtml#h.l5anuokbtii .c16}

[The]{.c6} [ Problem]{.c5}

Powerful [ AI systems will potentially have to make decisions in situations that are difficult for humans to evaluate\-\--for instance, on scales far outside human experience, or involving technology that is foreign to present-day humans. Modern ML systems are primarily trained through human-labeled training data (or more generally, human-generated reward functions), which presents an obstacle to specifying which decisions are good in such situations.]{.c4}

[]{.c4}

[Why We Want to Solve It]{.c5}

[Below are some concrete settings where relying directly on humans for oversight seems unlikely to work. I sloppily anthropomorphize AIs in the below examples by thinking of them as a single system that "reasons", "explains", etc.; however, I think the problems I describe are harder rather than easier for less anthropomorphic AIs.]{.c4}

[]{.c4}

1.  A candidate reward function for an AI system is "take actions that lead to outcomes that humans would approve of". To instantiate this, we might ask the agent to sample possible outcomes from its predictive model ^[\[11\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt11){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref11}^   and show them to us (and perhaps even create an interface for us to look around and examine the world under this possible outcome). However, it is unclear how to then assign reward\-\--the world is big and it might take an infeasibly long time to find everything in it. Maybe everyone on Earth is happy but there is a bunch of bad stuff happening on Jupiter [. We could ask the AI to direct us to the "relevant" parts but that just kicks the can down the road. Finally, beyond being big, the world likely also looks weird as we go farther into the future, in a way that may also make it hard to evaluate.]{.c4}
2.  Alternatively, maybe we don't want to rely on evaluating outcomes, and instead  want AIs to " [ [reason in a way that humans would endorse / approve of](https://www.google.com/url?q=https://ai-alignment.com/model-free-decisions-6e6609f5d99e&sa=D&source=editors&ust=1646871528222629&usg=AOvVaw3jv5F_rFIM9XxvMBPQpO4w){.c0} ]{.c1} [". But we would need to understand the decision-making process of the AI to provide supervision along these lines. These decisions are not by default human-interpretable, and even if they were human-interpretable the basic decisions are probably low-level enough that human-relevant concepts like "happiness" can only be seen emergently through the combination of many low-level decisions (in the same way that biological functions like "eat" or "sleep" cannot be easily understood in terms of the behavior of individual cells).]{.c4}
3.  We could imagine a middle-ground where the AI " explains its reasoning to us " and we give feedback on whether the explanation makes sense and is oriented towards human goals. But if it is explaining its reasoning to a human, there is a fundamental bottleneck on how complex the explanation can become, while the decisions the AI is making become more and more complex. We might expect the explanations to then become less faithful as time goes on. ^[\[12\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt12){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref12}^
4.  Returning to the original idea of "create outcomes that humans approve of", maybe we only have the AI ask us to evaluate things that are somewhat more familiar to us (medium-term consequences, rather than long-term consequences). This has the problem that a situation that looks good in the medium-term might be bad in the long-term, although we could perhaps avoid this with a medium-term goal of "create a really well-aligned AGI". [ Another problem is that even evaluating the medium-term might require understanding subtle consequences. We can already see this today with e.g. the Facebook news feed (the most obvious metrics for tracking user engagement neglect systemic effects, leading to filter bubbles) or just the fact that governance is pretty hard (cosmopolitan technocratic liberalism seems pretty good but maybe has harmed a bunch of working-class people?). Or maybe humans appear to be happy and in control of the world, but an AI system is secretly pulling the strings behind all major decisions.]{.c4}

[]{.c4}

[A solution to scalable reward generation should give humans a credible way to provide feedback on decisions whose consequences are by default fairly foreign to humans (as is the case in \#1-\#3 above) and that could involve subtle consequences that are hard to unearth (as is the case in \#4 and \#1). Keep in mind that the AI system is likely optimizing against this reward, so the reward generation process has to be robust enough to not be exploitable (rather than just giving reasonable answers in some nominal world).]{.c4}

[]{.c4}

[The examples above are a bit more disjunctive than in the previous sections\-\--examples \#1-\#3 all give disjoint paths to success (evaluate outcomes, evaluate process, interact with explanations) so we have some freedom in how to generate reward.]{.c4}

[]{.c4}

While reward generation would most directly be used to train  the system, I would personally feel more comfortable if we also had mechanisms to oversee the system during its actual operation (and potentially issue course corrections). This raises some additional challenges\-\--for instance, providing feedback to the system might be [ [time-constrained](https://www.google.com/url?q=https://ordinaryideas.wordpress.com/2015/11/30/driving-fast-in-the-counterfactual-loop/&sa=D&source=editors&ust=1646871528227069&usg=AOvVaw1ciXx0J_iDLZrGC5t-eA0Z){.c0} ]{.c1}  (perhaps even severely time-constrained if the system is operating on a much faster time-scale than humans). In this case we either need a way to provide good feedback quickly, or for the system to avoid requiring time-constrained feedback in the first place .

[]{.c4}

[What A Solution Looks Like]{.c5}

Scalable reward generation is less fleshed out as a problem than Robustness or Reward Learning, partially because it doesn't fit neatly into a well-established machine learning paradigm; perhaps the closest is indirect supervision  ( [ [example reference](https://www.google.com/url?q=http://l2r.cs.uiuc.edu/~danr/Papers/CSGR10.pdf&sa=D&source=editors&ust=1646871528228768&usg=AOvVaw0syzs9bUmP35ZtI7wZ2Yod){.c0} ]{.c1} ), but this is itself less well-established than the preceding areas, and does not grapple with the extreme level of scalability needed here or with being robust to another system optimizing against it.

[]{.c4}

Scalable reward generation can be split into two goals: generating a [single point]{.c6}  of good reward in a feasible length of time, and decreasing the [number of points]{.c6} [ of reward required (thus making the system more economically scalable and making it more okay to spend lots of time generating each data point).]{.c4}

[ ]{.c4}

For [generating]{.c6} [ a single point]{.c6}  of reward, the most concrete research program is [ [capability amplification](https://www.google.com/url?q=https://ai-alignment.com/policy-amplification-6a70cbee4f34&sa=D&source=editors&ust=1646871528230689&usg=AOvVaw11fX6wr8a6-3IzYHqNSbge){.c0} ]{.c1}  (discussed above as well; see also [ [debate](https://www.google.com/url?q=https://arxiv.org/abs/1805.00899&sa=D&source=editors&ust=1646871528231124&usg=AOvVaw3PXRgdKCoHXkydGIagJ1BM){.c0} ]{.c1}  which is closely related) .  The basic idea is to train a system to do the supervision that the human would do (this is useful because you can run many copies of that system and also likely run it at a faster speed than humans). You can also iterate this process (train a second system to do the supervision that the "human + first trained system" would do, etc.). Getting the system to exactly imitate a human is probably too hard,  but the hope is that using the system in conjunction with the original human overseer can create a system that is noticeably better than the human while also being faithful to the human's intentions; this could perhaps be done by decomposing tasks into smaller tasks that the human could provide feedback on. This approach has the virtue of being concrete, though there are many open questions (Can we actually usefully leverage the trained systems? Can we get a good decomposition? How good of a supervisor can this asymptote to? [) that we should try to resolve.]{.c4}

[]{.c4}

There are some other less fleshed out ideas, which I would like to flesh out further and pursue, or abandon if they turn out to be unpromising . One idea is interactive learning\-\--the system and human interact in a back-and-forth that could potentially be much more informative than placing the primary burden on the human to provide all the input. How to have the AI choose good questions for the interaction is not currently clear (and might itself involve solving some form of alignment problem). In addition, it is similarly unclear how well this scales\-\--does it just make the human's job noticeably easier [ (if that), or could it get all the way to letting us oversee very powerful AI? If the former, how useful is this (could we stack this together with other improvements that add up to sufficient scalability, or do we need a fundamentally different approach)?]{.c4}

[]{.c4}

A  related idea is to generate reward via some form of dialog system \-\--maybe we can specify some [ [high-level mission](https://www.google.com/url?q=https://machinethoughts.wordpress.com/2014/08/10/friendly-ai-and-the-servant-mission/&sa=D&source=editors&ust=1646871528234280&usg=AOvVaw0BsKW9C029yqF4ELkp009L){.c0} ]{.c1}  using natural language concepts (which may not be fully precise), and thus obviate the need to examine particular situations in detail . I could imagine the dialog approach will end up anywhere between "extremely promising" and "fundamentally unsound" once examined in more detail. It's also possible that natural language dialog would arise as a specific consequence of [ a different, more abstract approach, which would make it less valuable to work on directly; however, it's likely enough this won't happen that I still think it's worth pursuing.]{.c4}

[]{.c4}

Some other proposals include interpretability (if we can understand a system better, maybe we can oversee it better) or better human-computer interfaces (the idea being similar to capability amplification\-\--give humans really good tools to make it easier to provide the reward). My personal but unconfident intuition is that these are less likely to scale to powerful AI systems, but it's at least plausible. Interpretability may be useful for other purposes such as stress-testing models (discussed in more detail in its [ [dedicated section](#AIAlignmentResearchOverviewOct2019.xhtml#h.z2ttvjp52vq2){.c0} ]{.c1}  below [).]{.c4}

[]{.c4}

We next consider [decreasing the number of points]{.c6}  needed by the system ( or in an online setting, the frequency of points needed). Here there are more fleshed out research directions, as using less training data is a consistent desideratum in ML. These include active learning (having the system decide what oversight to ask for) and semi-supervised learning (trying to use unlabeled data to figure out as much as possible about the world, which hopefully decreases the amount of labeled data that is required) .  For a deployed system, we could also imagine only asking for oversight when the system is not confident in the right action [, which could allow for cheaper / more targeted ongoing oversight (but doesn't clearly avoid issues with time-constrained feedback).]{.c4}

[]{.c4}

[Research on generating a single point feels more urgent than research on decreasing the number of points\-\--if we can't do the former, then the latter won't help us. Also, the latter has had more progress so far (although I think still not that much), and there are many incentives for people to solve it beyond AI alignment.]{.c4}

[]{.c4}

[Aside. ]{.c6} The difficulty of scalable reward generation depends on how well we solve reward learning and robustness. The more the system is trying to work with us to get information (rather than optimizing against us), the easier it is to create a "good enough" reward generation process. The more we can expect the system to faithfully extrapolate knowledge from human-familiar situations to human-unfamiliar situations, the less we need humans to provide rewards for unfamiliar situations. An interesting strategic question (that I haven't thought much about) is to what extent solutions to these different problems smoothly trade off  against each other, versus being all-or-nothing. E.g. is it the case [ that we basically need to either solve the hardest version of distributional robustness (with a little bit of scalable reward generation thrown on top) or the hardest version of scalable reward generation (with a little bit of robustness + reward learning thrown on top), or could we hope to solve medium-difficulty versions of each of the problems?]{.c4}

[]{.c4}

[Possible ]{.c6} [Directions]{.c6}

I don't expect it to be directly useful, but it's worth becoming familiar with some of the existing work on indirect and distant supervision. A non-comprehensive list of work includes [ [DeepDive / Snorkel](https://www.google.com/url?q=http://deepdive.stanford.edu/&sa=D&source=editors&ust=1646871528240197&usg=AOvVaw22X89073qiuqtT2lNnpkN0){.c0} ]{.c1} , [ [expectation regularization](https://www.google.com/url?q=http://www.jmlr.org/papers/volume11/mann10a/mann10a.pdf&sa=D&source=editors&ust=1646871528240747&usg=AOvVaw3tOnBfoGE3-T05mcmqrEJO){.c0} ]{.c1} , and distant/indirect supervision for [ [relation extraction](https://www.google.com/url?q=https://www.aclweb.org/anthology/P09-1113&sa=D&source=editors&ust=1646871528241190&usg=AOvVaw33UM17FLqPHpDHjOg05AzP){.c0} ]{.c1}  and [ [structured prediction](https://www.google.com/url?q=http://l2r.cs.uiuc.edu/~danr/Papers/CSGR10.pdf&sa=D&source=editors&ust=1646871528241615&usg=AOvVaw277UGa9wKk63AaGqvQgY3F){.c0} ]{.c1} . Rohin Shah et al.'s idea of [ [Preferences Implicit in the State of the World](https://www.google.com/url?q=https://arxiv.org/abs/1902.04198&sa=D&source=editors&ust=1646871528242027&usg=AOvVaw2jN1rnobv86HRioO8vSqG8){.c0} ]{.c1}  can be thought of as a form of supervision that gives large amounts of indirect information about what humans value based on the fact that our existing environment is well-optimized for humans (though it seems unlikely that this information is robust enough to be optimized against directly).

[]{.c4}

Paul Christiano [ [posted](https://www.google.com/url?q=https://www.lesswrong.com/posts/SqcPWvvJJwwgZb6aH/prize-for-probable-problems&sa=D&source=editors&ust=1646871528243004&usg=AOvVaw14c42DbUMrzUO_ws9V6rp_){.c0} ]{.c1}  a request for potential problems with capability amplification (which he calls "iterated distillation and amplification" or IDA in the linked post), as well as some potential issues he himself sees. Many of these could be flipped around to yield potential research problems (e.g. "here is a potential obstacle, what is the proof of concept that we can overcome it?"). I am personally most interested in grounding out amplification in some concrete existing system that allows us to oversee something which we wouldn't otherwise be able to. My  biggest concerns (with low confidence) are (1) that we won't be able to usefully leverage the trained overseer without making it unaligned; and (2) that there are some things that are difficult for a human to supervise even given a lot of sped-up copies (e.g. analyzing low-level motor commands and then having to decide whether the higher-level policy that generated them corresponds to an aligned system). A concrete system (or proposal of a system) would make it easier to instantiate these and other objections and determine whether they hold  for that particular system. [ [Ought](https://www.google.com/url?q=https://ought.org/research&sa=D&source=editors&ust=1646871528244273&usg=AOvVaw1YVK3_v3qYCfeGEoVHaZ3T){.c0} ]{.c1} [ is currently working on this.]{.c4}

[]{.c4}

[For interactive learning and dialog systems, I don't think there's currently any real sketch of what we would even do, or an analysis of how scalable we should expect either to be. So I would say that is the first step. Ideally the sketch of what to do would be thorough enough that any potential limitations are real limitations (as opposed to just not trying hard enough to flesh out the idea), and have enough detail to point to a concrete-ish program of research.]{.c4}

[]{.c4}

[I could also imagine being convinced that interpretability or HCI are more credible paths to scalable reward generation than I currently believe. (I'm actually pretty excited about both routes as potential avenues, just currently would guess they won't work.) So seeing good positive arguments for either of these would be exciting.]{.c4}

[]{.c4}

For semi-supervised learning and active learning, there is a large literature on both topics that I won't recapitulate here [.]{.c4}

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.e1mpfimje89t}

### Acting [ Conservatively]{.c22} {#AIAlignmentResearchOverviewOct2019.xhtml#h.wrd99evb3row .c16}

[The Problem]{.c6}

[Safety will likely be easier to achieve if systems can notice situations where it is unclear how to act, and either avoid encountering them, take actions that reduce the uncertainty, or take actions that are robustly good. This would, for instance, allow us to specify an ambiguous reward function that the system could clarify as needed, rather than having to think about every possible case up-front.]{.c4}

[]{.c4}

While out-of-distribution robustness focuses on [noticing]{.c12}  the uncertainty, acting conservatively focuses on what to do [given]{.c12}  the uncertainty. Solutions to one thus interface with solutions to the other. Unfortunately, current methods for constructing uncertainty estimates seem inadequate for acting conservatively, and even given a good uncertainty estimate little work has been done on how the system should use such estimates to shape its actions .

[]{.c4}

[Why We Want to Solve It]{.c5}

[Here are some examples of places where we would want a system to act conservatively:]{.c4}

1.  [We specified a reward function for the system, but some pretty similar reward function would lead to very different actions/outcomes. We would like the system to realize this, realize that it's not that unlikely that we screwed up the reward function, and act appropriately (get further clarification, or avoid ending up in worlds where the reward functions diverge).]{.c4}
2.  [Conversely, maybe the system is seeking feedback from humans but is not confident that the humans understood the situation it is asking for feedback on. If the feedback plays a crucial role in its decisions, it could continue to clarify until it is confident the feedback it is getting is informative.]{.c4}
3.  [We want the system to avoid irreversible decisions unless it has high confidence they are good (for instance, genetically engineering all humans to be really happy all the time might be good if happiness is the thing we want to optimize for, but this change could cause substantial value drift in humans, and so is irreversible in the sense that you can no longer ask the original humans what they want/endorse).]{.c4}
4.  Maybe the AI discovers some weird fact about sentience that might affect how humans think about moral patienthood, such that the initial values given to the AI seem likely to be flawed. It is probably better for the AI to collaborate with humans on how to fix this rather than making something up itself [.]{.c4}
5.  [We also don't want the AI to experiment with weird physics that might accidentally blow up the earth or cause some other major disaster (unless it is correctly really confident that nothing will go wrong).]{.c4}

[]{.c4}

[All of these involve, on some level, having a representation of some form of uncertainty and altering actions according to that uncertainty. The paradigmatic way of representing uncertainties is via probabilities (typically predictive probabilities under some statistical model). An important point (elaborated below) is that the existing ways of obtaining these probabilities are insufficient for addressing most of the examples above. In fact, it is possible that probabilities themselves are inadequate for this (but let's not get too metaphysical too quickly).]{.c4}

[]{.c4}

[What]{.c6} [ A Solution Looks Like]{.c5}

Here's a default (but in my opinion incorrect) solution that one might express to the Acting Conservatively problem: "Most ML models give outputs that can be interpreted as probabilities (e.g. the final softmax layer in neural nets). Those probabilities give us a measure of uncertainty, so why not just maximize expected value with respect to that?" There are a number of issues with this approach.  The most direct is that the probabilities output by most ML models are not well-calibrated, so we probably don't want to rely on them . This itself could plausibly be fixed, but there are also some deeper issues. Take scenario \#1 above\-\--an event like "the reward function was written down wrong" isn't a reified  event in most ML frameworks, and it's also not an event that you can directly gather data about, so it's not clear where you would get the probabilities from  (and once you have them, should you actually maximize with respect to the expected value of the reward function? That seems like it would lead to weird behavior, and is not an approach that most people I know would endorse for handling the related issue of moral uncertainty in humans). Additionally, part of the uncertainty we want to represent is uncertainty in the presence of out-of-distribution events (such as the nanobots in example \#5), which won't necessarily manifest  in the probabilities learned by ML models if the model family is mis-specified.

[]{.c4}

O ne way to make this issue more crisp (at the expense of oversimplifying a bit) is to divide uncertainty into a few different categories (borrowed from [ [this](https://www.google.com/url?q=https://arxiv.org/pdf/1802.10501.pdf&sa=D&source=editors&ust=1646871528253732&usg=AOvVaw2GNAmncvwCHG3R_s9w9aic){.c0} ]{.c1}  paper): [aleatoric uncertainty ]{.c12} (uncertainty in the predictive distribution of a fixed model, e.g. from process randomness such as a coin flip or label noise, or inability of the model to distinguish between inputs with different outcomes ), [ model uncertainty ]{.c12} (uncertainty about which model is correct, usually due to statistical estimation noise), and [distributional uncertainty ]{.c12} (uncertainty due to mismatch between the train and test distributions). I would also add in [specification uncertainty]{.c12}  (uncertainty due to errors in how the problem itself was specified, e.g. maybe the likelihood function or reward function is wrong in some way). When people say that neural nets and other ML models tend to be mis-calibrated, they mean that just using aleatoric uncertainty gives overly-confident probability estimates. Typically the assumption is that this is due to not incorporating model uncertainty, and if we did so we would get better-calibrated estimates. But this ignores the additional sources of distributional and specification uncertainty that remain even in the limit of infinite data (when model uncertainty should in theory disappear) . ^[\[13\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt13){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref13}^

[]{.c4}

In summary, I think a solution to Acting Conservatively likely needs to incorporate richer sources of uncertainty than just aleatoric uncertainty + estimation error in the model. It should have a way of reasoning about out-of-distribution inputs and fundamental mis-specifications in the problem. We also probably need a decision rule other than "maximize expected value" for some forms of uncertainty (such as specification uncertainty), which is part of why it is unclear to me if probabilities are even the correct framework to use .

[]{.c4}

Another  default solution to Acting Conservatively is to say: "RL gives a framework for decision-making under uncertainty. The system might not explicitly represent probabilities, but the policy is trained to maximize expected reward, which is an average over possible rollouts. So we should expect to get the behavior you want out of Acting Conservatively as a by-product of a reasonable training process." I t is possible that something like this could work, but I'm currently skeptical.  At the very least, we probably need to modify the training process\-\--current RL algorithms only optimize for aleatoric uncertainty, so the above issues regarding other forms of uncertainty would likely apply here (I think we would minimally need to incorporate model uncertainty).

[]{.c4}

[Possible]{.c6} [ Directions]{.c5}

[To some extent, solutions to Acting Conservatively depend on solutions to other parts of the technical alignment problem (e.g. distributional shift, reward learning). If you buy into the "aleatoric/model/distributional/specification" (A/M/D/S) framework, then distributional uncertainty would come from the solution to distributional shift, while "specification uncertainty" would interface with reward learning and perhaps reward generation. However, even given these, I don't think it's clear how to actually choose actions. We could build RL environments that necessitate handling each of these types of uncertainty, or think abstractly about what desiderata a given rule for choosing actions should have.]{.c4}

[]{.c4}

One idea for  Acting Conservatively is to have conservative fallback policies that can be employed when necessary to minimize the chance of a bad outcome , or to find constraints (pre-conditions) on a policy that ensure it will be safe. This idea appears in many guises\--relevant terms include safe exploration and reachability analysis. Here are  several example references: [ [Safe Exploration in Markov Decision Processes](https://www.google.com/url?q=https://arxiv.org/abs/1205.4810&sa=D&source=editors&ust=1646871528259234&usg=AOvVaw0vnd8C44VfeztAD-mB9Q10){.c0} ]{.c1} , [ [Safe Exploration in Finite Markov Decision Processes with Gaussian Processes](https://www.google.com/url?q=https://arxiv.org/abs/1606.04753&sa=D&source=editors&ust=1646871528259786&usg=AOvVaw1EObhu6VXSDgCERrw0dplb){.c0} ]{.c1} , [ [A General Safety Framework for Learning-BasedControl in Uncertain Robotic Systems](https://www.google.com/url?q=https://people.eecs.berkeley.edu/~jfisac/papers/General_Safe_Learning.pdf&sa=D&source=editors&ust=1646871528260281&usg=AOvVaw3ixJMp5ZkgnuZuAHsYPr3p){.c0} ]{.c1} , [ [A Reachability Method for Verifying Dynamical Systems with Deep Neural Network Controllers](https://www.google.com/url?q=https://arxiv.org/abs/1903.00520&sa=D&source=editors&ust=1646871528260723&usg=AOvVaw36oH6E0JsnwhYG6iLz2zJl){.c0} ]{.c1} , and [ [Reachability-Based Safe Learning with Gaussian Processes](https://www.google.com/url?q=https://people.eecs.berkeley.edu/~jfisac/papers/Safe_Learning.pdf&sa=D&source=editors&ust=1646871528261170&usg=AOvVaw1JmahX1ZiI2HBFEE5yz-U7){.c0} ]{.c1} . Here are [ [two](https://www.google.com/url?q=http://cmp.felk.cvut.cz/~peckama2/papers/safe_exploration_overview_lncs.pdf&sa=D&source=editors&ust=1646871528261599&usg=AOvVaw2AL7h1bucAUGjZT8Ccm54B){.c0} ]{.c1}   [ [surveys](https://www.google.com/url?q=http://jmlr.org/papers/volume16/garcia15a/garcia15a.pdf&sa=D&source=editors&ust=1646871528261983&usg=AOvVaw0qqAK8PGLDyDSHH0ANNOoR){.c0} ]{.c1} [ on safe exploration, although they are somewhat outdated and a lot of it is what I would call risk-sensitive reinforcement learning (trying to avoid rare high-cost events, rather than trying to avoid unrecoverable errors during training), which seems less relevant. ]{.c4}

[]{.c4}

In the context of classification, see [ [these](https://www.google.com/url?q=https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber%3D6248086%26tag%3D1&sa=D&source=editors&ust=1646871528262937&usg=AOvVaw1sVMC-skbofp_cmVMzp-hi){.c0} ]{.c1}   [ [references](https://www.google.com/url?q=https://arxiv.org/pdf/1804.00722.pdf&sa=D&source=editors&ust=1646871528263324&usg=AOvVaw1KigLw2lQwNFCEkDD1Ik2O){.c0} ]{.c1} [ that consider classification where the output classes form a hierarchy, and act conservatively by trading off specificity and accuracy  (by outputting higher-level classes) in cases of uncertainty.]{.c4}

[]{.c4}

If you don't want to buy into the A/M/D/S framework and want to go the "let's just do RL route", it seems to me that you'd want to modify the RL objective to implicitly capture the M/D/S aspects in the learned policy. A basic question  might be how to  modify the RL objective to account for something like model uncertainty, and how to make sure that it was actually accounting for it in a good way. Then one could move on to distributional and specification uncertainty. See [ [these slides](https://www.google.com/url?q=https://www.ed.ac.uk/files/atoms/files/dietterich-edinburgh-v1.pdf&sa=D&source=editors&ust=1646871528264601&usg=AOvVaw0wFhXDsCQhkF49-U0SsPrw){.c0} ]{.c1} [ by Tom Dietterich for some examples of modifying the objective, as well as a number of other interesting ideas.]{.c4}

[]{.c4}

[ [Risk-sensitive reinforcement learning](https://www.google.com/url?q=https://arxiv.org/abs/1311.2097&sa=D&source=editors&ust=1646871528265333&usg=AOvVaw1I6jLYI3UWpQgWwgfNQAGe){.c0} ]{.c1}  aims to avoid rare high-cost events, although it still generally experiences these events at least during training, so this would not suffice for events that are too catastrophic to permit at all. It could, however, perhaps be combined with exploration in simulation (where the bad events would be permissible).

[]{.c4}

There is also a recent line of work on calibrating neural networks and representing model uncertainty, see e.g. [ [these](https://www.google.com/url?q=https://arxiv.org/abs/1706.04599&sa=D&source=editors&ust=1646871528266299&usg=AOvVaw1YDWemnLzwQMDGaC82gGZr){.c0} ]{.c1}   [ [references](https://www.google.com/url?q=https://arxiv.org/abs/1506.02142&sa=D&source=editors&ust=1646871528266686&usg=AOvVaw1OoU4R2P4OjvtixjMHDTlU){.c0} ]{.c1} [.]{.c4}

[]{.c4}

In general I expect distributional and specification uncertainty to be much harder to handle than model uncertainty, and to likely require different ideas .

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.lyrrtg5y1d27}

### Counterfactual Reasoning {#AIAlignmentResearchOverviewOct2019.xhtml#h.oj6ckreyf3ff .c16}

[The Problem]{.c5}

A system possesses [counterfactual reasoning]{.c12} [ abilities if it can reason not just about the actual world / distribution of inputs it is faced with, but also other (potentially physically or logically impossible) worlds / distributions. There are many theories for how a system ought to / might go about this, although no clear approaches for how to apply them scalably to existing systems. At the same time, counterfactual reasoning touches upon many other aspects of AI alignment, and so a satisfying solution could broadly make alignment easier.]{.c4}

[]{.c5}

[Why We Want to Solve It]{.c5}

[In the most optimistic case, counterfactual reasoning would resolve each of the following problems:]{.c4}

1.  Robustness to distributional shift asks for a system that does well not just on its training distribution, but on other possible test distributions as well. If those test distributions are within the set of counterfactuals that the system can reason about, then we expect it to do well on the test distribution as a consequence.
2.  Asking a system to simulate the consequences of various plans it might execute (and then showing them to humans) involves a limited form of counterfactual reasoning. This would allow us to evaluate plans without actually executing them, which seems safer.
3.  We can ask a system to take actions whose outcomes a present-day human would counterfactually rate highly if you showed the outcome to them. This is related  to Paul Christiano's [ [human in the counterfactual loop proposal](https://www.google.com/url?q=https://arbital.com/p/human_counterfactual_loop/&sa=D&source=editors&ust=1646871528270784&usg=AOvVaw35ftDsrUnYAymqcEne56X8){.c0} ]{.c1} [, but has the advantage that the counterfactual human cannot be physically manipulated by the AI system (because the human doesn't exist). (Note however that the AI could try to fool the counterfactual human, e.g. by presenting outcomes in a misleading way.)]{.c4}
4.  Relatedly, you could ask a system to do "what a human would tell it to do" in a given situation; this avoids the problem of the system optimizing against the human ratings (since it just follows directions instead of optimizing), but has its own host of issues such as it being unclear how a human would scalably tell an AI system what to do in complex situations .
5.  [We could imagine an interactive process for reward learning, where the system probes the human's preferences regarding a large number of counterfactual situations. This might provide a more robust way of specifying a reward function (because it is pinned down in a wider set of worlds, including "impossible" worlds).]{.c4}

[]{.c4}

[In addition, there are some more abstract reasons to think counterfactuals might be important:]{.c4}

6.  [Much of human dialog relates to counterfactuals, so counterfactuals might be an important part (or perhaps consequence) of dialog-based approaches to alignment.]{.c4}
7.  Logical counterfactuals  (e.g., "What would happen if 619023 was prime?") seem related to [ [logical uncertainty](https://www.google.com/url?q=https://intelligence.org/files/QuestionsLogicalUncertainty.pdf&sa=D&source=editors&ust=1646871528273313&usg=AOvVaw1sW_Oyr0Z8b8tuzhXCz-0n){.c0} ]{.c1} , which may be helpful if we wish to make predictions that are robust to computational constraints. ^[\[14\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt14){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref14}^

[]{.c4}

[Different desiderata require stronger or weaker forms of counterfactual reasoning. Intuitively, \#2 and \#5 require weaker forms of counterfactuals, while \#7 requires a strong (albeit narrow) form of counterfactuals. From the examples above, we can see that counterfactual reasoning captures many underlying themes in AI alignment, although it is possible that the concept is too broad or fuzzy to make useful progress on\--perhaps it captures all of these themes because we don't really understand counterfactuals right now and it is tempting to hope they will solve a lot of problems for us. That caveat aside, I am pretty excited about work on counterfactual reasoning\--I think it's possible that there are unifying conceptual ideas that would lead to progress on all or most of the problems above.]{.c4}

[]{.c4}

[What A Solution Looks Like]{.c6}

The most common way of formalizing counterfactuals is through [causality]{.c6} [. I know of two overarching approaches to causality, most famously advocated by Judea Pearl and Donald Rubin, respectively (I will refer to them as Pearl-causality and Rubin-causality).]{.c4}

[]{.c4}

In Pearl-causality, we require models to specify a complete process that generates the data (as in a Bayes net or probabilistic program), and define the set of allowed counterfactuals to be questions of the form "What would happen if we ran the process forward but intervened to set variable X  to value x0 (as opposed to whatever it would have been)?" For instance, you could ask "If there was bad weather on the roads but hypothetically it did not cause any accidents, would we expect to hear sirens?" This corresponds to setting "weather" to "bad" and "accidents" to "false" in [ [this](https://www.google.com/url?q=https://jliszka.github.io/assets/img/pgm/trafficjam.png&sa=D&source=editors&ust=1646871528275998&usg=AOvVaw0pnKqTxjQwxPwr1HubQH6h){.c0} ]{.c1}   Bayes [ net.]{.c4}

[]{.c4}

The issue: By requiring a complete generative process for the data, Pearl-causality seems to assume that you have (or can learn) a complete model of everything that is causally interesting in the world. In practice this approach leans on using domain knowledge to write down the necessary causal structure, which doesn't seem scalable.

[]{.c4}

In contrast to Pearl-causality, Rubin-causality separates counterfactual questions from the specific model. It defines an [intervention]{.c12}  as any experiment that you could in principle perform in the world (it's okay if it's expensive or unethical, but if there's no in-principle experiment then your question is " [ [FUQ'd](https://www.google.com/url?q=http://econ.lse.ac.uk/staff/spischke/mhe/ch1.pdf&sa=D&source=editors&ust=1646871528277598&usg=AOvVaw0Wt22xtC2Su0nrprFeH_zy){.c0} ]{.c1} " ). The set of counterfactual questions corresponds to the set of interventions ("What would happen if we applied this intervention?"). We then typically try to leverage some minimal set of assumptions to identify the effect of a specific intervention. [ A famous example concerns the effect of class size on student achievement. There was a school system that mandated a maximum class size of 30, so if there were 31 students they would be split into groups of 15 and 16. If we make the believable assumption that having 30 vs. 31 students is basically random and independent of any other decisions, then we can use this to estimate the effect of class size.]{.c4}

[]{.c4}

[Rubin-causality doesn't seem scalable, but for a different reason than Pearl-causality. Pearl-causality needed a model of all the causal effects, whereas Rubin-causality can get by with just identifying a carefully-selected "essentially random" part of the data (the 30 vs. 31 assumption in the above example). But identifying this seems to itself require domain expertise. Moreover, Pearl-causality gives us answers to an entire combinatorial family of counterfactual questions (everything defined in the Bayes net), while Rubin-causality requires us to come up with a different source of randomness for every intervention we care about.]{.c4}

[]{.c4}

[I haven't done justice to either approach above. For instance, Rubin-causality has other cool ideas like propensity matching (re-weight your data to simulate the effect of whatever intervention you wanted to apply) and doubly-robust estimators (a way of getting an estimate that is correct as long as either one of two assumptions holds). Pearl-causality probably contains lots of additional ideas as well, but I'm less familiar with it. Nevertheless, the scalability issue does seem like a real problem for both approaches.]{.c4}

[]{.c4}

Can we salvage these frameworks to make them more scalable? There's something intuitively-appealing about Pearl's version of counterfactuals as "alternative settings of variables in a generative program", but we need an account of how we would learn such a generative program from data. It seems unlikely to me that maximum-likelihood estimation is the right approach\-\--Newtonian mechanics is a pretty good causal theory of the world, even though quantum mechanics is more accurate. If we trained a model to predict physics well, and weren't lucky enough to stumble on quantum mechanics, I suspect maximum likelihood would not give us Newtonian mechanics but rather some jumbled mess [ (even if it gave Newtonian mechanics as the first-order term, we should expect the learned model to contain less meaningful components that happen to fit the error terms well on the training distribution).]{.c4}

[]{.c4}

[Here is a more abstract restating of the above: Maximum likelihood is not very good in the presence of model mis-specification\-\--it tends to aggressively optimize against the specifics of whatever dataset it is presented with, and will therefore tend to take advantage of spurious correlations that don't hold up in even nearby possible worlds. For many reasons (such as simplicity) we often want to work with incorrect models (such as Newtonian mechanics), but we don't have a good learning paradigm for pointing to "good" incorrect models relative to bad ones.]{.c4}

[]{.c4}

[My preferred framing of counterfactual reasoning]{.c6}  is therefore\-\-- [What alternatives to maximum likelihood produce models that are less overfit to spurious correlations in the data, and which are therefore more likely to still produce good predictions in counterfactual worlds? Can we characterize the family of counterfactuals that such models produce good predictions for?]{.c12}  This framing separates counterfactual worlds from causal interventions (causal interventions yield one type of counterfactual world, but any world where our model makes "good" predictions comprises a valid counterfactual ). This framing is admittedly far less fleshed out than Pearl- or Rubin-causality. I feel particularly excited about it because, among other things, alternative learning procedures to maximum likelihood might also shed light on alternatives to reward maximization (a large part of the alignment problem is that maximization is hard to employ safely [).]{.c4}

[]{.c4}

There are other definitions of counterfactuals beyond the Pearl and Rubin definitions given above. They are less instantiatable than Pearl and Rubin, but have the advantage of being less tied to interventions, which I feel produce an overly narrow conception of counterfactuals. One example is Lewis's [ [possible worlds](https://www.google.com/url?q=https://philpapers.org/rec/LEWC-2&sa=D&source=editors&ust=1646871528281404&usg=AOvVaw3eb0T07sMwOjQkHe8GUD2L){.c0} ]{.c1}  interpretation of causality. I think some people have also defined counterfactuals in terms of robust predictive rules (kind of similar to my really fuzzy attempt above) but I don't have a reference off-hand .

[]{.c4}

A final issue is that many (but not all) of the counterfactuals in our examples \#1-\#7 above are about fuzzy high-level concepts, and so (a) might not be valid counterfactual events in e.g. a Pearl-style model, and (b) might be hard to specify formally in the language of the model, even if they do correspond to a valid counterfactual event [.]{.c4}

[]{.c5}

[]{.c5}

[Possible Directions]{.c6}

On alternatives to maximum likelihood: the "not really an alternative" alternative is to incorporate lots of counterfactual worlds into your training data. This is the idea behind [ [domain randomization](https://www.google.com/url?q=https://arxiv.org/abs/1703.06907&sa=D&source=editors&ust=1646871528283108&usg=AOvVaw26xf1ISHBnodSxJ6VeEGnG){.c0} ]{.c1}  and [ [sim-to-real](https://www.google.com/url?q=https://arxiv.org/abs/1610.04286&sa=D&source=editors&ust=1646871528283571&usg=AOvVaw1RYBFdcZCyBsydwmDtqdiT){.c0} ]{.c1} : if we modify the simulation in a lot of different ways, hopefully the family of simulated worlds is rich enough to contain the real world or something like it. We can re-state this as saying that hopefully the real world is within the family of counterfactuals defined by the different versions of the simulation. Some challenges are (1) it's less clear how to do this if we aren't training in simulation, and (2) we need to define the task [objective in the counterfactual worlds (in domain randomization, the objective is to reach some goal state that is the same in all the counterfactuals, and what changes are other aspects of the environment like the dynamics, lighting, etc.; but it's not clear we can always expect defining the objective to be so easy).]{.c4}

[]{.c4}

Some larger departures from maximum likelihood are based on [agnostic learning]{.c6}  and [partial specification]{.c6} . Agnostic learning seeks to design procedures that work even if the model is mis-specified. For instance, a classical approach  to clustering is via fitting a mixture model using maximum likelihood, and then defining cluster membership via each component of the mixture; this works well if the mixture model is well-specified and identifiable, but can fail otherwise. [Agnostic clustering]{.c12}  asks whether there are clustering algorithms that work without having to assume a specific model family; for instance, maybe we can just assume that clusters are "well-separated" in some appropriate sense\-\--see [ [this paper](https://www.google.com/url?q=https://arxiv.org/abs/1004.1823&sa=D&source=editors&ust=1646871528285745&usg=AOvVaw3C_H2sRMqmOsqjgeVMAxEI){.c0} ]{.c1}  by Kumar and Kannan as well as my more recent   [ [paper](https://www.google.com/url?q=https://arxiv.org/abs/1711.07465&sa=D&source=editors&ust=1646871528286233&usg=AOvVaw1MOjRtGkYHPv32QfB3BjqP){.c0} ]{.c1}  (among many others) for more info [.]{.c4}

[]{.c4}

A closely related concept is [partial specification]{.c12} , where rather than modeling a complete generative process of the data we only specify certain aspects of the generative process (this in particular precludes maximum likelihood because there is no likelihood to maximize). The simplest example is ordinary least squares, which works whenever the errors are uncorrelated from the signal (even if they are non-Gaussian, which is what we would need to model least squares as maximum likelihood). See my paper on [ [unsupervised risk estimation](https://www.google.com/url?q=https://cs.stanford.edu/~jsteinhardt/publications/risk-estimation/paper.pdf&sa=D&source=editors&ust=1646871528287604&usg=AOvVaw07r-KlApWH3mSiPP3rDjiz){.c0} ]{.c1}  for a more complex example as well as a brief literature review on partial specification, and my paper (with Banghua Zhu and Jiantao Jiao) on [ [generalized resilience](https://www.google.com/url?q=https://arxiv.org/abs/1909.08755&sa=D&source=editors&ust=1646871528288090&usg=AOvVaw056mDV13YI98wMI5TQLI26){.c0} ]{.c1} , which defines non-parametric assumptions under which robust estimation is possible . I also recommend Section III of [ [this](https://www.google.com/url?q=https://pdfs.semanticscholar.org/13c5/01825cd4388152cdad85d152d95690529dd4.pdf&sa=D&source=editors&ust=1646871528288619&usg=AOvVaw0EdFeSycTWbcT-S7x_G-A0){.c0} ]{.c1} [ excellent lecture by Lars Peter Hansen.]{.c4}

[]{.c5}

I also recommend Leon Bottou's [ [magnum opus](https://www.google.com/url?q=https://www.microsoft.com/en-us/research/wp-content/uploads/2013/11/bottou13a.pdf&sa=D&source=editors&ust=1646871528289440&usg=AOvVaw39z1GAiliZQUIMHnSFCrwB){.c0} ]{.c1}  on counterfactual reasoning (beyond being a great overview, it gives additional examples of how counterfactuals might help with mis-specified rewards; for instance, "maximize average user happiness" might incentivize getting all unhappy users to quit your website as quickly as possible, while "maximize average happiness that the current pool of users would have" removes this effect). Another relevant area is [ [contextual bandits](https://www.google.com/url?q=http://hunch.net/~exploration_learning/main.pdf&sa=D&source=editors&ust=1646871528290098&usg=AOvVaw1iS6qnKG9MooPPrbJrIQK1){.c0} ]{.c1} .

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.5f2efge7es77}

[Detecting failures in advance]{.c15} {#AIAlignmentResearchOverviewOct2019.xhtml#h.bcg9ehclyjjp .c20}
-------------------------------------

[The previous section lays out a list of obstacles to AI alignment and technical directions for working on them. This list may not be exhaustive, so we should also develop tools for discovering new potential alignment issues. Even for the existing issues, we would like ways of being more confident that we have solved them and what sub-problems remain.]{.c4}

[]{.c4}

[While machine learning often prefers to hew close to empirical data, much of the roadmap for AI alignment has instead followed from more abstract considerations and thought experiments, such as asking "What would happen if this reward function were optimized as far as possible? Would the outcome be good?" I actually think that ML undervalues this abstract approach and expect it to continue to be fruitful, both for pointing to useful high-level research questions and for analyzing concrete systems and approaches.]{.c4}

[]{.c4}

At the same time, I am uncomfortable relying solely on abstract arguments for detecting potential failures. Rigorous empirical testing can make us more confident that a problem is actually solved and expose issues we might have missed. Finding concrete instantiations of a problem can both more fruitfully direct work and convince a larger set of people to care about it (as in the case of adversarial examples for images). More broadly, empirical investigations have the potential  to reveal new issues that were missed under purely abstract considerations [.]{.c4}

[]{.c4}

T wo more empirically-focused ways of detecting failures are [ [model probing/visualization](#AIAlignmentResearchOverviewOct2019.xhtml#h.z2ttvjp52vq2){.c0} ]{.c1 .c6}  and [ [red-teaming](#AIAlignmentResearchOverviewOct2019.xhtml#h.xja47c1n2on){.c0} ]{.c1 .c6} [, ]{.c6} discussed below . Also valuable is [examining trends]{.c6}  in ML. For instance, it looks to me like reward hacking in real deployed systems is becoming a bigger issue over time; this provides concrete instances of the problem to examine for insight, gives us a way to measure how well we're doing at the problem, and helps rally a community around the problem. Examining trends is also a good way to take an abstract consideration and make it more concrete.

### Model Probing and Visualization  {#AIAlignmentResearchOverviewOct2019.xhtml#h.z2ttvjp52vq2 .c16}

[]{.c5}

[The Problem]{.c5}

Most ML models are "black boxes" in the sense that they are complex systems built up from low-level primitives (such as linear transformations and non-linearities). Even if they make good predictions, they do not necessarily do so by means of human-meaningful concepts . Our intuitions about how ML models work are likely misguided, and better tools for probing ML models could both help us build better intuition and help predict potential failures. This could include error analysis, evaluation on distributions of atypical inputs, or visualization tools.

[]{.c4}

[Why We Want to Solve It]{.c5}

[Currently, the most common way of evaluating a model is its test accuracy on a task or set of tasks. Richer ways of probing a model could help us predict failures in advance, because potential failure modes may manifest as e.g. unexpected internal states of a neural network, or unexpected extrapolation to atypical examples, before they lead to actual perverse outputs.]{.c4}

[]{.c4}

A good example here is [ [adversarial examples](https://www.google.com/url?q=https://arxiv.org/abs/1312.6199&sa=D&source=editors&ust=1646871528296458&usg=AOvVaw3KDroATRTLHcgDe2DXDxRK){.c0} ]{.c1} [, which seem to have been discovered as a result of an investigation that (a) probed the robustness properties of neural networks and (b) questioned the semantic meaningfulness of hidden node activations in neural networks.]{.c4}

[]{.c4}

[Some additional examples and arguments:]{.c4}

-   [It is tempting to conjecture that ML models achieving human-level performance on e.g. object recognition do so in the same way that humans do, but this need not be the case, and the presence of adversarial images provides evidence against this. Good model probing tools could help us determine to what extent neural nets or other ML models share concepts with humans. This could be particularly important for concepts that are used as part of a learned reward function.]{.c4}

```{=html}
<!-- -->
```
-   I've seen opinions ranging from "what neural nets do is similar to what humans do" to "what neural nets do is nothing like what humans do". From my limited experience visualizing object recognition models, the answer seems somewhere in between, where in some aspects model behavior is quite intuitive and in others it seems bizarre. Understanding this better seems important to predicting how models will behave in new situations.

```{=html}
<!-- -->
```
-   W [e would also like to make sure that the model is "doing the right things for the right reasons", which is a much higher bar than just getting high test accuracy (which is what ML as a field traditionally emphasizes).]{.c4}
-   The impact  o f training data on the learned model is not well-understood\-\--there seem to be some potentially weird effects in high-dimensional data where every individual training point has a "large influence" on the learned parameters, and the prediction of many individual test points are sensitive to small changes in the data or training procedure (moreso than the aggregate accuracy). It is not clear if this will lead to safety issues, but it is sufficiently counterintuitive that it may be worth understanding better.

[]{.c5}

[A separate, cross-cutting point is that having a high-bandwidth channel for interrogating a model (e.g. a vivid visualization vs. a table of numbers) can in my experience quickly change my mental map of what neural nets or other models are doing. As noted above, having accurate mental maps of how ML models work seems important for anticipating their behavior in new situations and for anticipating when they might fail.]{.c4}

[]{.c5}

[What A Solution Looks Like / Possible Directions]{.c5}

[In contrast to the technical alignment problem, detecting failures is an ongoing process rather than a concrete question that can be "solved". Below I describe some tools that could help with this process.]{.c4}

[]{.c4}

One way of probing a model is to simply evaluate its accuracy on additional datasets (or under additional metrics). Evaluating robustness to different types of perturbations (as in [ [ImageNet-C](https://www.google.com/url?q=https://github.com/hendrycks/robustness&sa=D&source=editors&ust=1646871528299205&usg=AOvVaw2SF0KBSAkq15ilZBoW8gTV){.c0} ]{.c1} , [ [Stylized-ImageNet](https://www.google.com/url?q=https://github.com/rgeirhos/Stylized-ImageNet&sa=D&source=editors&ust=1646871528299528&usg=AOvVaw1QsjSkswHVXujWwsjliPjb){.c0} ]{.c1} , or [ [adversarial examples](https://www.google.com/url?q=https://arxiv.org/abs/1905.01034&sa=D&source=editors&ust=1646871528299773&usg=AOvVaw3GOGwpWh-Wq8NowJv4LALL){.c0} ]{.c1} ) or different data distributions (as in [ [ImageNet-A](https://www.google.com/url?q=https://arxiv.org/abs/1907.07174&sa=D&source=editors&ust=1646871528300019&usg=AOvVaw0sEYr5lepy-lMPq85ARiUP){.c0} ]{.c1} [) falls into this category. Such evaluations seem particularly useful because they are about how a model's predictions extrapolate to a new situation. Neural nets are empirically robust to some types of stochastic perturbations but not others, and I (and probably most people) have limited ability to predict in advance whether a network will be robust to some new type of perturbation. Building better understanding here seems important.]{.c4}

[]{.c4}

Visualization tools aimed at providing researchers with a higher-bandwidth window into the behavior of ML models also seem broadly useful. Chris Olah  and collaborators have [ [written](https://www.google.com/url?q=https://distill.pub/2018/building-blocks/&sa=D&source=editors&ust=1646871528300767&usg=AOvVaw3_Xafz0MIKpN94HC5hcr8u){.c0} ]{.c1}  extensively about these sorts of tools and provided several examples through the [ [Lucid](https://www.google.com/url?q=https://github.com/tensorflow/lucid&sa=D&source=editors&ust=1646871528301033&usg=AOvVaw1pOpueXgLZtIxZE7TDVlft){.c0} ]{.c1}  library. Note however that visualization can be misleading\--it is easy to extract things from models that [seem]{.c12}  human-interpretable but are not tracking what NNs are doing in a meaningful way. For instance, [ [recent work suggests](https://www.google.com/url?q=https://arxiv.org/abs/1810.03292&sa=D&source=editors&ust=1646871528301392&usg=AOvVaw0fyhrV5e-sz82cXlsFaSbt){.c0} ]{.c1} [ that saliency maps have this problem, and I expect many visualizations to have this issue by default.]{.c4}

[]{.c4}

There have recently been some interesting case studies using a combination of out-of-distribution evaluation and visualization to reach some preliminary conclusions about how models tend to generalize. For instance, [ [Geirhos et al.](https://www.google.com/url?q=https://openreview.net/forum?id%3DBygh9j09KX&sa=D&source=editors&ust=1646871528301974&usg=AOvVaw37qB9Txz8IFjcL2BHbdX6T){.c0} ]{.c1}  argue that CNNs are biased towards texture over shape, while [ [Zhang and Zhu](https://www.google.com/url?q=https://arxiv.org/abs/1905.09797&sa=D&source=editors&ust=1646871528302351&usg=AOvVaw1qMNuWCa5FVwSmTmuV725i){.c0} ]{.c1}  argue that adversarial training removes some of this texture bias .

[]{.c4}

Other potentially useful tools include better evaluation procedures for model performance (e.g. work on [ [evaluating generative models](https://www.google.com/url?q=https://openreview.net/pdf?id%3DB1M8JF9xx&sa=D&source=editors&ust=1646871528303268&usg=AOvVaw3gL6yfaHFzdD-AdAasgX6D){.c0} ]{.c1} ), [ [influence functions](https://www.google.com/url?q=https://arxiv.org/abs/1703.04730&sa=D&source=editors&ust=1646871528303668&usg=AOvVaw0U3RGhrGAHX8EU5BS-vF4h){.c0} ]{.c1} , and debugging tools such as [ [model assertions](https://www.google.com/url?q=https://ddkang.github.io/papers/2018/omg-nips-ws.pdf&sa=D&source=editors&ust=1646871528304100&usg=AOvVaw06eShh2IDv4E4zvjQSfQhp){.c0} ]{.c1} .

[]{.c4}

Model visualization falls under the umbrella of interpretability. However, "interpretability" is poorly-defined in the ML literature and people mean a lot of different things by it (Zachary Lipton elaborates on this [ [here](https://www.google.com/url?q=https://arxiv.org/abs/1606.03490&sa=D&source=editors&ust=1646871528305000&usg=AOvVaw2TA4ghAqMU88IG-cG8FV6r){.c0} ]{.c1} ). I think most work under this broader umbrella has limited relevance to detecting failures and is instead pursuing some other goal, such as satisfying certain legal or ethical desiderata.

[]{.c4}

For instance, work focused on making models interpretable to naive (non-ML-savvy) human subjects does not seem that relevant to detecting failures, since the end users of interpretability tools for failure detection would be ML researchers. Relative to naive users, ML researchers will both be more familiar with the tools and place more demands on them. It is plausible that this style of work would still generate insights that are applicable to this use case, but it is driven by a different goal.

[]{.c4}

Some papers also make vague claims that Bayes nets, or linear models, or some other anointed model family is more interpretable than neural nets. I could imagine some version of this work being useful (e.g. maybe you could design a different neural net architecture for which it was apparent that adversarial examples wouldn't be an issue). But most of the time these claims are not substantiated, and do not address interpretability in the regime where there are a large number of input features, which is necessary for scalability.

[]{.c4}

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.jcyeckebdfdm}

### [Red-teaming]{.c22} {#AIAlignmentResearchOverviewOct2019.xhtml#h.xja47c1n2on .c16}

[T]{.c6} [he Problem]{.c5}

["]{.c6} [ [Red-teaming](https://www.google.com/url?q=https://en.wikipedia.org/wiki/Red_team&sa=D&source=editors&ust=1646871528307800&usg=AOvVaw00qF6Wz_fYLnSJUZf_lb_E){.c0} ]{.c1} [" is a concept in cybersecurity as well as military and business intelligence. The idea is to create a group that works separately from the rest of the organization, and whose goal is to find flaws in plans or defenses. In AI safety and alignment, red teams could expose flaws in plans or research agendas, or could anticipate or demonstrate failure modes of an individual system. Currently, there is little formal infrastructure (in terms of organizations, tools, and ideas) to support red teams.]{.c4}

[]{.c4}

Paul Christiano's [ [blog post](https://www.google.com/url?q=https://ai-alignment.com/red-teams-b5b6de33dc76&sa=D&source=editors&ust=1646871528308843&usg=AOvVaw0jJCJiQm_bCfDpGYlk_mOS){.c0} ]{.c1}  on red teams also provides a good overview. Red-teaming need not be done formally as part of an organization; it could e.g. involve an individual researcher trying to find flaws in an existing system or approach.

[]{.c5}

[Why We Want to Solve It]{.c5}

[As mentioned above, it is unlikely that we have already articulated all potential failures of AI systems. Red-teaming at the level of plans or agendas is a mechanism for finding additional such failures.]{.c4}

[]{.c4}

[Additionally, red-teaming individual systems can help instantiate failures modes that were previously abstract, as in the case of adversarial examples (or in traditional computer security, where one style of work is to show that a given security vulnerability actually has a corresponding exploit). Such instantiation can both make it more clear how to address the failure, and also create a greater sense of urgency among any skeptical parties. If the system is actually going to be deployed in the world, red-teaming can increase our confidence that the system is safe.]{.c4}

[]{.c4}

[My intuition is that red-teaming often improves the effectiveness of organizations and helps uncover blind spots. If we think of "the field of AI alignment" as a sort of organization, then we would want red teams for this reason as well.]{.c4}

[]{.c5}

[What A Solution Looks Like]{.c5}

[I would like to see more object-level work that looks like red-teaming, as well as more institutional infrastructure designed to support such teams (e.g. it would be good if DeepMind had such a team, assuming they don't already; and it would also be good for red-teaming to be seen as a genre of work within academia). ]{.c4}

[]{.c4}

An example of red-team thinking is [ [this post](https://www.google.com/url?q=https://ai-alignment.com/an-unaligned-benchmark-b49ad992940b&sa=D&source=editors&ust=1646871528312229&usg=AOvVaw1Wk2b3QE4JDo_bTZJ71eF5){.c0} ]{.c1}  where Paul Christiano shows that naive deep RL systems would do quite poorly on alignment. While not directly about safety, my [ [paper](https://www.google.com/url?q=https://arxiv.org/abs/1807.03341&sa=D&source=editors&ust=1646871528312769&usg=AOvVaw1CMlnaIL9eSbUr6eayEGBV){.c0} ]{.c1}  with Zachary Lipton can be thought of as red-teaming empirical standards in ML. Researchers in the FAT (Fairness-Accountability-Transparency) area often engage in red-teaming, as in the [ [Excavating AI project](https://www.google.com/url?q=https://www.excavating.ai/&sa=D&source=editors&ust=1646871528313280&usg=AOvVaw2r21aW3dbUVt4Z46deIHtQ){.c0} ]{.c1} .

[]{.c4}

One can also exhibit failures in specific systems. The [ [original adversarial examples paper](https://www.google.com/url?q=https://arxiv.org/abs/1312.6199&sa=D&source=editors&ust=1646871528314095&usg=AOvVaw1U614IpX3bRIUwK4-ZBPyd){.c0} ]{.c1}  is an instance of this that was very high-impact and exposed a way that ML systems might behave differently than intended. Papers questioning the reliability of various datasets have also sometimes had high impact ( [ [here](https://www.google.com/url?q=https://arxiv.org/abs/1808.04926&sa=D&source=editors&ust=1646871528314618&usg=AOvVaw2u-ROSAfelfsyRU_sOQR2e){.c0} ]{.c1} , [ [here](https://www.google.com/url?q=https://arxiv.org/abs/1606.02858&sa=D&source=editors&ust=1646871528314993&usg=AOvVaw0__Az4gHaEFIGL6zbT5hny){.c0} ]{.c1} , and [ [here](https://www.google.com/url?q=https://arxiv.org/abs/1711.06640&sa=D&source=editors&ust=1646871528315368&usg=AOvVaw2G7zhT7JnaEsWbFIJ7gv0x){.c0} ]{.c1} ). ^[\[15\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt15){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref15}^  My sense based on the success of the above papers is that there is appetite in the ML community for good empirical red-teaming work, which is a hopeful sign . [ On the other hand, there is often defensiveness on the part of the people whose work got red-teamed. It would probably be good to have better social conventions around how to constructively red-team, and how one ought to respond to red-teaming efforts.]{.c4}

[]{.c5}

[Possible Directions]{.c5}

Continuing work on adversarial examples seems useful, and I am particularly excited about work that uses more realistic and general threat models, such as the [ [unrestricted adversarial examples competition](https://www.google.com/url?q=https://github.com/google/unrestricted-adversarial-examples&sa=D&source=editors&ust=1646871528316733&usg=AOvVaw15d9guzQ6-W0fkyz2eQ8ZB){.c0} ]{.c1} [. I would like to see work in additional domains beyond images (including value learning) and on additional failure modes beyond prediction errors.]{.c4}

[]{.c4}

On domains beyond images: There has been some work exploring adversarial examples in [ [RL](https://www.google.com/url?q=https://arxiv.org/abs/1702.02284&sa=D&source=editors&ust=1646871528317576&usg=AOvVaw1Sqs-2xAtED8nulPS0MDYs){.c0} ]{.c1}  and in [ [malware detection](https://www.google.com/url?q=http://www.patrickmcdaniel.org/pubs/esorics17.pdf&sa=D&source=editors&ust=1646871528318004&usg=AOvVaw2DmljPyO0yF_4WlivVmwfd){.c0} ]{.c1} . Regarding value learning in particular, I would guess that the value functions learned by existing frameworks (such as [ [deep RL from human preferences](https://www.google.com/url?q=https://arxiv.org/abs/1706.03741&sa=D&source=editors&ust=1646871528318496&usg=AOvVaw12RXHMwaJo12QNvWMMdBbf){.c0} ]{.c1} ) would perform very poorly out-of-distribution; verifying this could be [ worthwhile.]{.c4}

[]{.c4}

For uncovering failure modes beyond prediction errors, here is one possible example. Supposing that learned representations (e.g. image vectors, word vectors) will be used in powerful AI systems, we may be interested in ways in which these representations behave counterintuitively or perversely. Is there some way of probing such representations and understanding in what ways we should/shouldn't expect them to perform well? For instance,   [ [some work suggests](https://www.google.com/url?q=https://arxiv.org/abs/1607.06520&sa=D&source=editors&ust=1646871528319694&usg=AOvVaw2usTeNBaNYsJTX-Lh5t4J6){.c0} ]{.c1}  that word embeddings can reflect gender stereotypes present in the data (though see a recent [ [partial qualification](https://www.google.com/url?q=https://arxiv.org/pdf/1905.09866.pdf&sa=D&source=editors&ust=1646871528320184&usg=AOvVaw1hdyY2u6y1AnDwLAP3CRHG){.c0} ]{.c1} [).]{.c4}

[]{.c4}

Paul Christiano also discusses some potential directions in the previously-mentioned [ [blog post](https://www.google.com/url?q=https://ai-alignment.com/red-teams-b5b6de33dc76&sa=D&source=editors&ust=1646871528320929&usg=AOvVaw0zMdZvMQ99ed83AqlaIKVu){.c0} ]{.c1} .

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.7peo885oamfj}

Methodological [ Understanding]{.c15} {#AIAlignmentResearchOverviewOct2019.xhtml#h.gelap16ik8p5 .c20}
-------------------------------------

[The Problem]{.c5}

Methodological understanding is the understanding of best practices regarding good design and development patterns, how to debug a system, common failure modes to check for, and so on. In addition to a good conceptual understanding of AI alignment and a will to put that in place for powerful AI systems, we will likely need a well-developed understanding of best practices around how to build safe and aligned systems.

[]{.c4}

[Why We Want to Solve It]{.c5}

[In machine learning, examples of methodological understanding include:]{.c4}

-   " [ [Gradient checking](https://www.google.com/url?q=https://stackoverflow.com/questions/47506521/what-exactly-is-gradient-checking&sa=D&source=editors&ust=1646871528323199&usg=AOvVaw3gtdNUKjWKMYKfI7BgA2jw){.c0} ]{.c1} [": A way of checking that backpropagation was implemented correctly, which guards against errors that would otherwise be difficult to even notice.]{.c4}
-   [ [A/A testing](https://www.google.com/url?q=https://www.optimizely.com/optimization-glossary/aa-testing/&sa=D&source=editors&ust=1646871528323889&usg=AOvVaw2sNu1aHPMhtKWHrP-1o4WL){.c0} ]{.c1} [: A way of checking that A/B testing software doesn't turn up spurious false positives.]{.c4}
-   Train/validation/test (and cross-validation): this underlies essentially all empirical work in ML, but one could imagine a worse version of ML where we didn't use validation sets and just did train and test\--or even worse, train and train [ as is actually the case in RL right now.]{.c4}

[]{.c4}

[I am worried that we are missing methodological ideas as basic as these in the context of AI alignment, and also that the methodological problems we need to solve are harder (because having any catastrophic errors is unacceptable, and because we will be more limited in our ability to safely experiment with full-scale systems before deploying them).]{.c4}

[]{.c4}

One might be inclined to think of methodological work as "easy" compared to technical work. I think this is wrong. Some illustrative recent anecdotes include  the continuing difficulty  of [ [establishing best practices](https://www.google.com/url?q=https://arxiv.org/abs/1709.06009&sa=D&source=editors&ust=1646871528326012&usg=AOvVaw1OmHNjMY2O-O9-7TJ916pW){.c0} ]{.c1}  for RL in deterministic environments , the difficulty of attributing improvements to architecture choice versus [ [hyperparameter](https://www.google.com/url?q=https://arxiv.org/abs/1707.05589&sa=D&source=editors&ust=1646871528326619&usg=AOvVaw1uz-uD4HADvHNmeJ2VrnVe){.c0} ]{.c1}   [ [tuning](https://www.google.com/url?q=https://arxiv.org/abs/1711.10337&sa=D&source=editors&ust=1646871528326995&usg=AOvVaw2C4Q7333IyRP4IkOzroEZL){.c0} ]{.c1}   (or even [ [different random seeds](https://www.google.com/url?q=https://arxiv.org/abs/1709.06560&sa=D&source=editors&ust=1646871528327435&usg=AOvVaw0YAST20v7OxmYoNUK8BBEV){.c0} ]{.c1} ), and the ease of constructing datasets that appear to measure an interesting capability (like understanding long-range dependencies) but can actually be [ [solved by simple heuristics](https://www.google.com/url?q=https://arxiv.org/abs/1606.02858&sa=D&source=editors&ust=1646871528327922&usg=AOvVaw1wA_8Ts4-RHSypl4VE_fA8){.c0} ]{.c1} [.]{.c4}

[]{.c4}

An example  of a recent good methodological paper is " [ [Obfuscated Gradients Give A False Sense of Security](https://www.google.com/url?q=https://arxiv.org/abs/1802.00420&sa=D&source=editors&ust=1646871528328801&usg=AOvVaw3DQJb364pXyeuOy_vp9Y7n){.c0} ]{.c1} "\-\--it highlights the problem it discusses by breaking a number of existing systems, and then offers a checklist of signs that a system suffers from obfuscated gradients. " [ [On Evaluating Adversarial Robustness](https://www.google.com/url?q=https://arxiv.org/abs/1902.06705&sa=D&source=editors&ust=1646871528329316&usg=AOvVaw2o8CtZhoAuclcRtGKG_PZS){.c0} ]{.c1} [" is another good methodological paper from the adversarial robustness community.]{.c4}

[]{.c4}

[What A Solution Looks Like]{.c6}

[I do not have a complete list of the sorts of methodological understanding we would like to have for AI alignment, and the goal here is fuzzier than for much of the technical work. Nevertheless, here are some example questions that seem unanswered to me:]{.c4}

[]{.c4}

-   [Best practices for eliciting human preferences/feedback. Relatedly, understanding the ways in which humans tend to mess up when providing preferences/feedback.]{.c4}
-   [There is a danger of overfitting to only the failure modes we have observed so far (and then succumbing to some new not-yet-observed failure mode at deployment time). For instance, adversarial training solves known issues with a system by adding examples exhibiting that issue to the training process. How do we avoid overfitting to only the failure modes we've found so far, and when can we be confident that a procedure such as adversarial training has actually conferred sufficient robustness?]{.c4}

```{=html}
<!-- -->
```
-   Our paper on [ [transfer of adversarial robustness across perturbation types](https://www.google.com/url?q=https://arxiv.org/abs/1905.01034&sa=D&source=editors&ust=1646871528331664&usg=AOvVaw3lv_8Gh4AkocYN7QEssnlc){.c0} ]{.c1}  attempts to start grappling with this issue, as does the [ [ImageNet-C paper](https://www.google.com/url?q=https://arxiv.org/abs/1903.12261&sa=D&source=editors&ust=1646871528332208&usg=AOvVaw3kHrji6o82-OtYP2ENyTWD){.c0} ]{.c1} [.]{.c4}

```{=html}
<!-- -->
```
-   [What amount and types of testing tend to be enough to ensure that a system will behave as intended in new situations (or at least not catastrophically fail)?]{.c4}
-   [What are good ways of testing whether a given method is likely to scale to very powerful systems?]{.c4}
-   What are good ways of assessing the degree of progress in each of the technical alignment categories discussed above?

[]{.c4}

[In addition, each of the specific technical problems discussed in the earlier sections would benefit from a corresponding set of best practices around their proposed solutions.]{.c4}

[]{.c4}

Near-term safety-critical systems could help provide inspiration for best practices. For instance, in robotic surgery, even a single catastrophic error is likely considered unacceptable and I would be interested in understanding the vetting process there. Other areas that might provide inspiration are [ [air collision avoidance systems](https://www.google.com/url?q=https://ieeexplore.ieee.org/abstract/document/7311450/&sa=D&source=editors&ust=1646871528334370&usg=AOvVaw1wX0DOG5L8qMuE9Rw75F1e){.c0} ]{.c1} , autonomous driving, cybersecurity, and certain large-scale systems such as those deployed by Google, Facebook, Microsoft, etc. While the best practices in these areas would not apply directly to very powerful AI systems, there may still be important insights and some borrowable methodologies . See [ [this](https://www.google.com/url?q=http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf&sa=D&source=editors&ust=1646871528335154&usg=AOvVaw1qphmJ9GsEIn5QHhWkI65g){.c0} ]{.c1}  best practices paper by Google and [ [this](https://www.google.com/url?q=https://arxiv.org/abs/1606.03966v2&sa=D&source=editors&ust=1646871528335584&usg=AOvVaw3KP2Y-fgzy6r_DZqSvg2ER){.c0} ]{.c1} [ MSR paper on avoiding technical debt. My understanding is also that Google has to regularly deal with reward hacking in the form of their systems overfitting to easily-collected metrics at the expense of actual user satisfaction, and also with adversarial attacks on their ML systems such as people getting past Youtube's content filters.]{.c4}

[]{.c4}

[Possible Directions]{.c6}

As mentioned above, understanding when training against some failure modes generalizes to novel failures seems important. One could build a collection of failure modes, and test when training on some subset of the failure modes confers robustness to the remaining subset (and try to understand how to predict when this will happen) .

[]{.c4}

Constructing and critiquing benchmarks for various AI alignment tasks could also be worthwhile. However, constructing benchmarks is a large amount of work, and it is easy for benchmarks to end up not being useful, so I would advise anyone doing this to solicit feedback (especially from people with experience building datasets, and people with experience in the relevant domain) before doing so. An example of constructing a benchmark is the various [ [adversarial](https://www.google.com/url?q=https://arxiv.org/abs/1804.00097&sa=D&source=editors&ust=1646871528337752&usg=AOvVaw34PsDGud28to1jysXlH6Te){.c0} ]{.c1}   [ [example](https://www.google.com/url?q=https://github.com/google/unrestricted-adversarial-examples&sa=D&source=editors&ust=1646871528338185&usg=AOvVaw2ppAtU6hcuVtt2wvMth5uA){.c0} ]{.c1} [ competitions.]{.c4}

[]{.c4}

I expect some methodological work to be done in conjunction with fleshing out a given technical proposal (e.g. best practices around elicitation might be done in conjunction with work on reward learning or scalable reward generation). Attending to the methodological aspects of the work (i.e. understanding in detail what practices are necessary for things to work well and reproducibly, as opposed to just building a proof-of-concept system) will likely lead to better methodological understanding.

[]{#AIAlignmentResearchOverviewOct2019.xhtml#id.od7iufi6mafd}

System-Building {#AIAlignmentResearchOverviewOct2019.xhtml#h.aazn4b1tanc .c20}
---------------

[The Problem]{.c5}

[When building large systems, many challenges arise that would not arise in a proof-of-concept implementation, or even in a small-scale system. For example, we often need to split large systems into smaller self-contained components for the design process to be feasible. We also likely need to have many engineers working together, not all of whom communicate on a daily basis, which requires more effective separation of responsibilities than for a smaller system. Proposals for building powerful AI systems safely should account for the fact that such systems might be the result of such a large-scale project.]{.c4}

[]{.c4}

[Note: ]{.c6} [This is one of the more speculative sections, as it makes assumptions about how powerful AI systems might be built (i.e. that it will be a large-scale engineering project). There are alternative possibilities, such as "small research team with access to large amounts of compute resources and data". In this case System-Building might be an unnecessary problem to solve, or we might need to instead solve a different problem related to the alternative design process. On the other hand, system-building issues could still arise in other aspects of AI alignment. For instance, the sub-system for scalable reward generation might involve many humans interacting with the system in a complex way.]{.c4}

[]{.c4}

[Why We Want to Solve It]{.c5}

Here is an example of something that might go wrong in the absence of good system-building. It is based on a [ [talk](https://www.google.com/url?q=https://icml.cc/2015/invited/LeonBottouICML2015.pdf&sa=D&source=editors&ust=1646871528341903&usg=AOvVaw3uAc2rki6yR9YSzNufVZOj){.c0} ]{.c1}  by [Leon Bottou:]{.c4}

[]{.c4}

-   [There are two teams. Team A is building a recommender system, and team B builds a system to decide what font size to display the recommendations in.]{.c4}
-   [Team A decides to perform epsilon-greedy to explore the space of recommendations.]{.c4}
-   [Team B, if optimizing independently of A, will always display the exploratory recommendations in small font (since it learns that those recommendations are less likely to be clicked on). Therefore, the exploration effectively never occurs since in practice users will not notice the small font.]{.c4}

[]{.c4}

The issue in the above example is that components that in isolation appear to achieve their goals can fail to do so in the context of the larger system. We might worry about something similar for alignment: perhaps individual components appear reliable and aligned but the system as a whole is not. Or maybe some other type of problem occurs that we haven't articulated yet. My main worry is that we don't really know much about what might go wrong here, and our current evidence about large-scale systems is that things generically tend to go wrong without good engineering paradigms.

[]{.c4}

Thinking about large systems has other benefits: many failure modes of systems get exposed at scale\--we can think of this as a brute-force version of red-teaming ("anything that can go wrong will go wrong given a sufficiently large number of users"). Large-scale systems also probably bring a lot more optimization power to bear than smaller systems, making them behave "more like" future powerful AI in some ways. For instance, my sense is that (some) people at Google care more about reward hacking than the average ML researcher because their systems are actually powerful enough to reward-hack their proxy signals. Finally, we want proposed methods for alignment to be robust to scaling issues so that they work for powerful future AI systems, and some of these issues likely already manifest for existing large-scale systems.

[]{.c4}

[What A Solution Looks Like]{.c5}

I don't have a clean idea of what it would mean to "solve" the System-Building problem, and maybe there is no concrete goal state and it is more about accruing increasingly good intuition and best practices. So figuring out what exactly we want here is part of the question. It might look a lot like getting better [ methodological understanding, but in the specific context of a large system. Perhaps it would also incorporate ideas from systems engineering in other disciplines (e.g. civil engineering).]{.c4}

[]{.c4}

[Relative to the other areas above, I would potentially expect "regular" system-building work to transfer to "alignment" system-building work more than normal, because large systems already need high reliability, which is a key component of alignment.]{.c4}

[]{.c4}

[Possible Directions]{.c5}

I would start by trying to better understand current best practices and challenges in large systems. There have been [ [some](https://www.google.com/url?q=https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf&sa=D&source=editors&ust=1646871528346561&usg=AOvVaw15AGe1jf7gbGOiqKdenFhO){.c0} ]{.c1}   [ [white](https://www.google.com/url?q=http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf&sa=D&source=editors&ust=1646871528346884&usg=AOvVaw2kRtdmyK6oeWol6uKRvz22){.c0} ]{.c1}  papers and [ [talks](https://www.google.com/url?q=https://icml.cc/2015/invited/LeonBottouICML2015.pdf&sa=D&source=editors&ust=1646871528347154&usg=AOvVaw0Xi0Z6oU-jwK9sXHgDXe3G){.c0} ]{.c1} [ about this.]{.c4}

[]{.c4}

I expect a lot of work on System-Building to involve accumulating best practices rather than doing traditional technical research, but there are also potential technical challenges. For instance, ML systems break many traditional software abstractions, and we might want to fix this by causing ML systems to have better contracts (see [ [this talk](https://www.google.com/url?q=https://cs.stanford.edu/~jsteinhardt/talk/&sa=D&source=editors&ust=1646871528347729&usg=AOvVaw3cJ-yCQmBu9sM9jPCxjtp1){.c0} ]{.c1}  by me; Leon Bottou also talks about ML with contracts in the [ [talk](https://www.google.com/url?q=https://icml.cc/2015/invited/LeonBottouICML2015.pdf&sa=D&source=editors&ust=1646871528348034&usg=AOvVaw1o9c5RyeO8iAJvcOQ4h6YB){.c0} ]{.c1}  linked in the preceding paragraph) .  As one example of technical work, [ [this paper](https://www.google.com/url?q=https://arxiv.org/abs/1606.03966&sa=D&source=editors&ust=1646871528348415&usg=AOvVaw0cVa0F_lesTQd1bEQUBelm){.c0} ]{.c1} [ is an effort by MSR to deploy contextual bandits algorithms without incurring technical debt.]{.c4}

[]{.c4}

Another possible direction might be trying to design better tools for debugging stochastic systems. For example, [ [Certigrad](https://www.google.com/url?q=https://github.com/dselsam/certigrad&sa=D&source=editors&ust=1646871528348929&usg=AOvVaw3fnNovujbCaPD5sFgCN9XT){.c0} ]{.c1}   formally verifies certain stochastic properties  of ML software.

------------------------------------------------------------------------

<div>

[\[1\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref1){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt1} [ E.g. from Stuart Russell's recent ]{.c17} [ [book](https://www.google.com/url?q=https://www.amazon.com/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS&sa=D&source=editors&ust=1646871528349500&usg=AOvVaw30y-q3RKJIwwUjR4FdZ8HK){.c0} ]{.c1 .c17} [ on the topic]{.c17} [.]{.c2}

</div>

<div>

[\[2\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref2){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt2} [ ]{.c17} Conversely, maybe it will only involve 10 engineers at DeepMind or OpenAI doing some end-to-end training and almost all of the engineering effort is in backend scaling of simulators and distributed GPU usage, rather than on the AI system itself. This seems more likely under "short timelines" ( powerful  AI arriving in the next \~5 years), so system-building may be less (but still somewhat) useful in these scenarios.

</div>

<div>

[\[3\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref3){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt3} [ There are much more prosaic examples already affecting current systems. Here I intentionally try to imagine what issues might befall a much more sophisticated AI, at the cost of my examples being more speculative.]{.c2}

</div>

<div>

[\[4\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref4){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt4} [ The "act" part of this would fall under acting conservatively rather than out-of-distribution robustness.]{.c2}

</div>

<div>

[\[5\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref5){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt5} [ We may wish to specify something other than a reward function, in which case a more appropriate term might be "goal learning" or "specification learning". I will use reward learning for concreteness, but the reader should understand that rewards are one of several ways of specifying the desired behavior of a system.]{.c2}

</div>

<div>

[\[6\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref6){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt6} [ ]{.c17} [So for instance, solutions that rely on the fact that gradient descent updates don't lead to sudden changes are not scalable, because the meta-learned agent likely makes use of new data in a more sophisticated way.]{.c13}

</div>

<div>

[\[7\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref7){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt7} [ When using the term statistical uncertainty, I assume that we have some likelihood function \$p(x \\mid \\theta)\$ for the observed data \$x\$ conditional on the parameters \$\\theta\$ of the reward function. In a Bayesian framework the statistical uncertainty is \$p(\\theta \\mid x)\$, while in a frequentist framework it is whatever confidence region we construct for \$\\theta\$.]{.c2}

</div>

<div>

[\[8\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref8){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt8} [ Specifically, ]{.c17} [suppose the true reward function is R\*, and we have a family of reward functions {R\_{theta}}. If R\* = R\_{theta} for some theta, then the model family is correct. But it could be that our method for defining R\* is incorrect, and we accidentally define the reward R\' (which might even also be in the model family). An example of this would be if we assume that people\'s desires are given by their revealed preferences (or their preferences if you ask them questions and give them 5 minutes to think about it) rather than the desires they would endorse upon reflection. It could be that the model family contains the reward functions corresponding to both cases, but you still have to correctly specify how to back out the reward function from the ]{.c11} [data]{.c11} [.]{.c11}

</div>

<div>

[\[9\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref9){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt9} [ ]{.c17} This reward may be expensive to evaluate; the amplification approach is willing to accept this as long as "expensive" is closer to \$10\^3 / example than \$10\^12 / example.

</div>

<div>

[\[10\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref10){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt10} [ For instance, asking people to reason about systemic filter bubble effects caused by a given newsfeed recommendation might be infeasible, and hence "do people approve of this newsfeed recommendation" would in that case be a misspecified proxy of "is this recommendation good".]{.c2}

</div>

<div>

[\[11\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref11){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt11} [ There is a separate issue of incentivizing the predictive model to be accurate and honest, which I'm ignoring here.]{.c2}

</div>

<div>

[\[12\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref12){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt12} [ I'm again ignoring the issue that the AI might be incentivized to give explanations that sound good but are not accurate reflections of their reasoning.]{.c2}

</div>

<div>

[\[13\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref13){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt13} [ Additionally, the ways in which people measure mis-calibration don\'t tend to pick up on distributional or specification uncertainty. So solving the problem of mis-calibration as it is currently understood would not mean we had good uncertainty estimates for the purpose of Acting Conservatively.]{.c2}

</div>

<div>

[\[14\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref14){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt14} [ MIRI, the organization that wrote the linked document, make a different case for logical uncertainty than the one I give, and also care about counterfactuals for ]{.c17} [ [decision theory](https://www.google.com/url?q=https://intelligence.org/files/TechnicalAgenda.pdf&sa=D&source=editors&ust=1646871528353115&usg=AOvVaw1NMVd32RiGEcLWjR7uzKNe){.c0} ]{.c1 .c17} [; I don't fully understand their arguments and suspect I would disagree with them, so have not tried to reproduce them here.]{.c17}

</div>

<div>

[\[15\]](#AIAlignmentResearchOverviewOct2019.xhtml#ftnt_ref15){#AIAlignmentResearchOverviewOct2019.xhtml#ftnt15} [ ]{.c17}  Dataset reliability isn't directly relevant to alignment, although better knowledge of how to construct informative datasets may aid in evaluating the safety of ML systems.

</div>

AI Alignment Research Overview-by Jacob Steinhardt-date 2019-10-06.md