AI Alignment Podcast: An Overview of Technical AI Alignment: [Part 1](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) and [Part 2](https://futureoflife.org/2019/04/25/an-overview-of-technical-ai-alignment-with-rohin-shah-part-2/)

In this podcast, I go through a large swath of research agendas around technical AI alignment. The first part is more of a description of what research agendas exist, who works on them, and what they are trying to do, while the second part delves more into the details of each approach. I'd strongly recommend listening to them if you're trying to orient yourself in the technical AI safety landscape.

Topics covered include [embedded agency](https://www.alignmentforum.org/s/Rm6oQRJJmhGCcLvxh), [value learning](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc), <@impact regularization methods@>(@Designing agent incentives to avoid side effects@), [iterated amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd), <@debate@>(@AI safety via debate@), <@factored cognition@>(@Factored Cognition@), <@robustness@>(@Techniques for optimizing worst-case performance@), interpretability (no canonical link, but <@activation atlases@>(@Exploring Neural Networks with Activation Atlases@) is an example), <@comprehensive AI services@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@), <@norm following@>(@Incomplete Contracting and AI Alignment@), and [boxing](https://www.alignmentforum.org/posts/pZhDWxDmwzuSwLjou/asymptotically-benign-agi) (this newsletter).


Summary of the Technical Safety Workshop

David identifies two broad types of AI safety work: human in the loop approaches, and theory approaches. A notable subset of the former category is methods which improve our ability to give advanced systems meaningful feedback - this includes debate, IDA, and recursive reward modeling. CIRL and CAIS are also human-in-the-loop. Meanwhile the theory category includes MIRI's work on agent foundations; side effect metrics; and verified boxing.


Comments on CAIS

This post is another summary of and response to [Comprehensive AI Services](https://www.fhi.ox.ac.uk/wp-content/uploads/Reframing_Superintelligence_FHI-TR-2019-1.1-1.pdf), after [mine](https://www.alignmentforum.org/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services-as) last week ([AN #40](https://mailchi.mp/b649f32b07da/alignment-newsletter-40)). I recommend it to get a different take on an important set of ideas. It delves much more into the arguments between the CAIS perspective and the single-AGI-agent perspective than my summary did.


People sometimes expect me to be against this sort of work, because I wrote <@Coherence arguments do not imply goal-directed behavior@>. This is not true. My point in that post is that coherence arguments _alone_ are not enough, you need to combine them with some other assumption (for example, that there exists some “resource” over which the agent has no terminal preferences). I do think it is plausible that this research agenda gives us a better picture of agency that tells us something about how AI systems will behave, or something about how to better infer human values. While I am personally more excited about studying particular development paths to AGI rather than more abstract agent models, I do think this research would be more useful than other types of alignment research I have seen proposed.

Selection Theorems: A Program For Understanding Agents

This post proposes a research area for understanding agents: **selection theorems**. A selection theorem is a theorem that tells us something about agents that will be selected for in a broad class of environments. Selection theorems are helpful because (1) they can provide additional assumptions that can help with learning human values, and (2) they can tell us likely properties of the agents we build by accident (think inner alignment concerns).

As an example, [coherence arguments](https://www.alignmentforum.org/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities) demonstrate that when an environment presents an agent with "bets" or "lotteries", where the agent cares only about the outcomes of the bets, then any "good" agent can be represented as maximizing expected utility. (What does it mean to be "good"? This can vary, but one example would be that the agent is not subject to Dutch books, i.e. situations in which it is guaranteed to lose resources.) This can then be turned into a selection argument by combining it with something that selects for "good" agents. For example, evolution will select for agents that don't lose resources for no gain, so humans are likely to be represented as maximizing expected utility. Unfortunately, many coherence arguments implicitly assume that the agent has no internal state, which is not true for humans, so this argument does not clearly work. As another example, our ML training procedures will likely also select for agents that don't waste resources, which could allow us to conclude that the resulting agents can be represented as maximizing expected utility, if the agents don't have internal states.

Coherence arguments aren't the only kind of selection theorem. The <@good(er) regulator theorem@>(@Fixing The Good Regulator Theorem@) provides a set of scenarios under which agents learn an internal "world model". The [Kelly criterion](http://www.eecs.harvard.edu/cs286r/courses/fall10/papers/Chapter6.pdf) tells us about scenarios in which the best (most selected) agents will make bets as though they are maximizing expected log money. These and other examples are described in [this followup post](https://www.alignmentforum.org/posts/N2NebPD78ioyWHhNm/some-existing-selection-theorems).

The rest of this post elaborates on the various parts of a selection theorem and provides advice on how to make original research contributions in the area of selection theorems. Another [followup post](https://www.alignmentforum.org/posts/RuDD3aQWLDSb4eTXP/what-selection-theorems-do-we-expect-want) describes some useful properties for which the author expects there are useful selections theorems to prove.


Early on, this post reminded me of [Twenty Billion Questions](https://arxiv.org/pdf/1705.10720.pdf); even though they are concretely different, these two pieces share a conceptual thread. They both consider the measurement of multiple quantities essential for solving their problems: 20BQ for encouraging AIs to be low-impact, and this post for productive framings of ill-defined concepts and as a heads-up about potential catastrophes.

Measurement is important, and this article poignantly argues why and illustrates how. It volunteers potential ideas that can be worked on today by mainstream ML researchers, and offers up a powerful toolkit to improve one's own quality of analysis. It would be great to see more examples of this technique applied to other contentious, fuzzy concepts in ML and beyond. I'll quickly note that while there seems to be minimal interest in this from academia, measurement of optimization power has been discussed earlier in several ways, e.g. [Measuring Optimization Power](https://www.lesswrong.com/posts/Q4hLMDrFd8fbteeZ8/measuring-optimization-power), or <@the ground of optimization@>(@The ground of optimization@).

**Rohin's opinion:** I broadly agree with the perspective in this post. I feel especially optimistic about the prospects of measurement for (a) checking whether our theoretical arguments hold in practice and (b) convincing others of our positions (assuming that the arguments do hold in practice).

Measurement, Optimization, and Take-off Speed

In this blogpost, the author argues that "trying to measure pretty much anything you can think of is a good mental move that is heavily underutilized in machine learning". He motivates the value of measurement and additional metrics by (i) citing evidence from the history of science, policy-making, and engineering (e.g. x-ray crystallography contributed to rapid progress in molecular biology), (ii) describing how, conceptually, "measurement has several valuable properties" (one of which is to act as interlocking constraints that help to error-check theories), and (iii) providing anecdotes from his own research endeavours where such approaches have been productive and useful (see, e.g. <@Rethinking Bias-Variance Trade-off@>(@Rethinking Bias-Variance Trade-off for Generalization of Neural Networks@)).

He demonstrates his proposal by applying it to the notion of _optimization power_ -- an important idea that has not been measured or even framed in terms of metrics. Two metrics are offered: (a) the change (typically deterioration) of performance when trained with a perturbed objective function with respect to the original objective function, named _Outer Optimization_, and (b) the change in performance of agents during their own lifetime (but without any further parameter updates), such as the log-loss on the next sentence for a language model after it sees X number of sequences at test time, or _Inner Adaptation_. Inspired by these, the article includes research questions and possible challenges.

He concludes with the insight that take-off would depend on these two continuous processes, Outer Optimization and Inner Adaptation, that work on very different time-scales, with the former being, at this time, much quicker than the latter. However, drawing an analogy from evolution, where it took billions of years of optimization to generate creatures like humans that were exceptional at rapid adaptation, we might yet see a fast take-off were Inner Adaptation turns out to be an exponential process that dominates capabilities progress. He advocates for early, sensitive measurement of this quantity as it might be an early warning sign of imminent risks.


Transparency and AGI safety

This post identifies four different motivations for working on transparency:

1. By learning more about how current neural networks work, we can improve our forecasts for AI timelines.
2. It seems _necessary_ for inner alignment. In particular, whatever AI development model you take, it seems likely that there will be some possibility of emergent misbehavior, and there doesn't yet seem to be a way to rule that out except via transparency.
3. A good solution to transparency would be _sufficient_ for safety, since we could at least notice when AI systems were misaligned, and then choose not to deploy them.
4. Even if AI will "go well by default", there are still instrumental reasons for transparency, such as improving cause prioritization in EA (via point 1), and for making systems more capable and robust.

After reviewing work on <@circuits@>(@Thread: Circuits@), the post suggests a few directions for future research:

1. Investigating how modular neural networks tend to be,
2. Figuring out how to make transparency outputs more precise and less subjective,
3. Looking for circuits in other networks (i.e. not image classifiers), see e.g. <@RL vision@>(@Understanding RL Vision@),
4. Figuring out how transparency fits into an end-to-end story for AI safety.


Certainly to the extent that AI alignment requires us to “lock in” philosophical approaches, I think it is important that we consider a plurality of views for this purpose (see also <@The Argument from Philosophical Difficulty@>). I especially think this is true if our approach to alignment is to figure out “human values” and then tell an AI to maximize them. However, I’m more optimistic about other approaches to alignment; and I think they require fewer philosophical commitments, so it becomes less of an issue that the alignment community has a specific philosophical bubble. See [this comment](https://www.alignmentforum.org/posts/jS2iiDPqMvZ2tnik2/ai-alignment-philosophical-pluralism-and-the-relevance-of?commentId=zaAYniACRc29CM6sJ) for more details.

AI Alignment, Philosophical Pluralism, and the Relevance of Non-Western Philosophy

This post argues that AI alignment has specific philosophical tendencies: 1) connectionism, where knowledge is encoded in neural net weights rather than through symbols, 2) behaviorism, where we learn from data rather than using reasoning or planning, 3) Humean motivations for humans (i.e. modeling humans as reward maximizers), 4) viewing rationality as decision theoretic, that is, about maximizing expected utility, rather than also considering e.g. logic, argumentation, and dialectic, and 5) consequentialism. This could be a "philosophical bubble" caused by founder effects from the EA and rationality communities, as well as from the recent success and popularity of deep learning.

Instead, we should be aiming for philosophical plurality, where we explore other philosophical traditions as well. This would be useful because 1) we would likely find insights not available in Western philosophy, 2) we would be more robust to moral uncertainty, 3) it helps us get buy in from more actors, and 4) it is the "right" thing to do, to allow others to choose the values and ethical frameworks that matter to them.

For example, certain interpretations of Confucian philosophy hold that norms have intrinsic value, as opposed to the dominant approach in Western philosophy in which individual preferences have intrinsic value, while norms only have instrumental value. This may be very relevant for learning what an AI system should optimize. Similarly, Buddhist thought often talks about problems of ontological shifts.


This agenda, like others, is motivated by the scenario where we need to get alignment right the first time, without empirical feedback loops, both because we might be facing one-shot success or failure, and because the stakes are so high that we should aim for high reliability subject to time constraints. I put low probability on the first reason (alignment being one-shot), and it seems much less tractable, so I mostly ignore those scenarios. I agree with the second reason, but aiming for this level of rigor seems like it will take much longer than the time we actually have. Given this high level disagreement, it's hard for me to evaluate the research agenda itself.

The Learning-Theoretic AI Alignment Research Agenda

This agenda aims to create a general abstract theory of intelligence (in a manner similar to [AIXI](https://en.wikipedia.org/wiki/AIXI), but with some deficiencies removed). In particular, once we use the framework of reinforcement learning, regret bounds are a particular way of provably quantifying an agent's intelligence (though there may be other ways as well). Once we have this theory, we can ground all other AI alignment problems within it. Specifically, alignment would be formalized as a value learning protocol that achieves some regret bound. With this formalization, we can solve hard metaphilosophy problems such as "What is imperfect rationality?" through the intuitions gained from looking at the problem through the lens of value learning protocols and universal reinforcement learning.


I focus almost exclusively on mid-term safety (while thinking about long-term safety), not because I disagree with this argument, but in spite of it. I think there is a good chance that any work I do will be useless for aligning superintelligent AI because of a paradigm shift, but I do it anyway because it seems very important on short timelines, which are easier to affect; and I don't know of other approaches to take that would have a significantly higher probability of being useful for aligning superintelligent AI.

A possible stance for AI control research

Conceptual issues in AI safety: the paradigmatic gap

Lots of current work on AI safety focuses on what we can call "mid-term safety" -- the safety of AI systems that are more powerful and more broadly deployed than the ones we have today, but work using relatively similar techniques as the ones we use today. However, it seems plausible that there will be a paradigm shift in how we build AI systems, and if so it's likely that we will have a new, completely different set of mid-term concerns, rendering the previous mid-term work useless. For example, at the end of the 19th century, horse excrement was a huge public health hazard, and "mid-term safety" would likely have been about how to remove the excrement. Instead, the automobile was developed and started replacing horses, leading to new set of mid-term concerns (eg. pollution, traffic accidents), and any previous work on removing horse excrement became near-useless.


I like this post -- it seems like an accessible summary of the state of the debate so far. My opinions are already in the post, so I don’t have much to add.

Plausible cases for HRAD work, and locating the crux in the "realism about rationality" debate

This post tries to identify the possible cases for highly reliable agent design (HRAD) work to be the main priority of AI alignment. HRAD is a category of work at MIRI that aims to build a theory of intelligence and agency that can explain things like logical uncertainty and counterfactual reasoning.

The first case for HRAD work is that by becoming less confused about these phenomena, we will be able to help AGI builders predict, explain, avoid, detect, and fix safety issues and help to conceptually clarify the AI alignment problem. For this purpose, we just need _conceptual_ deconfusion -- it isn't necessary that there must be precise equations defining what an AI system does.

The second case is that if we get a precise, mathematical theory, we can use it to build an agent that we understand "from the ground up", rather than throwing the black box of deep learning at the problem.

The last case is that understanding how intelligence works will give us a theory that allows us to predict how _arbitrary_ agents will behave, which will be useful for AI alignment in all the ways described in the first case and <@more@>(@Theory of Ideal Agents, or of Existing Agents?@).

Looking through past discussions on the topic, the author believes that people at MIRI primarily believe in the first two cases. Meanwhile, critics (particularly me) say that it seems pretty unlikely that we can build a precise, mathematical theory, and a more conceptual but imprecise theory may help us understand reasoning better but is less likely to generalize sufficiently well to say important and non-trivial things about AI alignment for the systems we are actually building.


One of the counterarguments the podcast talks about is <@my position@>(@Conversation with Rohin Shah@) that we’ll probably learn from smaller catastrophes in order to avoid actual extinction. I just want to note that while it might sound like I disagree with Stuart on this point, I don’t think we actually do. I was arguing against the position that extinction is the default outcome (> 50% probability) while Stuart is arguing against the position that extinction is near-impossible (~0% probability). I ended up around 10%; I’d guess that if Stuart were forced to, he’d give a number similar to mine, for similar reasons as me.

The flaws that make today’s AI architecture unsafe and a new approach that could fix it

This podcast delves into many of the ideas in Stuart's book <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@). Rob especially pushes on some aspects that are less talked about in the AI safety community, like the enfeeblement problem and whether we'd be locking in suboptimal values. They also discuss Stuart's response to some counterarguments.


I am broadly sympathetic to the focus on preventing the worst outcomes and it seems plausible that extortion could play an important role in these, even though I worry more about distributional shift plus incorrigibility. Still, I am excited about the focus on cooperation, as this seems robustly useful for a wide range of scenarios and most value systems.

**Rohin's opinion:** Under a suffering-focused ethics under which s-risks far overwhelm x-risks, I think it makes sense to focus on this agenda. There don't seem to be many plausible paths to s-risks: by default, we shouldn't expect them, because it would be quite surprising for an amoral AI system to think it was particularly useful or good for humans to _suffer_, as opposed to not exist at all, and there doesn't seem to be much reason to expect an immoral AI system. Conflict and the possibility of carrying out threats are the most plausible ways by which I could see this happening, and the agenda here focuses on neglected problems in this space.

However, under other ethical systems (under which s-risks are worse than x-risks, but do not completely dwarf x-risks), I expect other technical safety research to be more impactful, because other approaches can more directly target the failure mode of an amoral AI system that doesn't care about you, which seems both more likely and more amenable to technical safety approaches (to me at least). I could imagine work on this agenda being quite important for _strategy_ research, though I am far from an expert here.

Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda

This agenda by the [Effective Altruism Foundation](https://ea-foundation.org/) focuses on risks of astronomical suffering (s-risks) posed by <@Transformative AI@>(@Defining and Unpacking Transformative AI@) (TAI) and especially those related to conflicts between powerful AI agents. This is because there is a very clear path from extortion and executed threats against altruistic values to s-risks.  While especially important in the context of s-risks, cooperation between AI systems is also relevant from a range of different viewpoints. The agenda covers four clusters of topics: strategy, credibility and bargaining, current AI frameworks, as well as decision theory.

The extent of cooperation failures is likely influenced by how power is distributed after the transition to TAI. At first glance, it seems like widely distributed scenarios (as <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@)) are more problematic, but related literature from international relations paints a more complicated picture. The agenda seeks a better understanding of how the distribution of power affects catastrophic risk, as well as potential levers to influence this distribution. Other topics in the strategy/governance cluster include the identification and analysis of realistic scenarios for misalignment, as well as case studies on cooperation failures in humans and how they can be affected by policy.

TAI might enable unprecedented credibility, for example by being very transparent, which is crucial for both contracts and threats. The agenda aims at better models of the effects of credibility on cooperation failures. One approach to this is open-source game theory, where agents can see other agents' source codes. Promising approaches to prevent catastrophic cooperation failures include the identification of peaceful bargaining mechanisms, as well as surrogate goals. The idea of surrogate goals is for an agent to commit to act as if it had a different goal, whenever it is threatened, in order to protect its actual goal from threats.

As some aspects of contemporary AI architectures might still be present in TAI, it can be useful to study cooperation failure in current systems. One concrete approach to enabling cooperation in social dilemmas that could be tested with contemporary systems is based on bargaining over policies combined with punishments for deviations.  Relatedly, it is worth investigating whether or not multi-agent training leads to human-like bargaining by default. This has implications on the suitability of behavioural vs classical game theory to study TAI. The behavioural game theory of human-machine interactions might also be important, especially in human-in-the-loop scenarios of TAI.

The last cluster discusses the implications of bounded computation on decision theory as well as the decision theories (implicitly) used by current agent architectures. Another focus lies on acausal reasoning and in particular the possibility of [acausal trade](https://wiki.lesswrong.com/wiki/Acausal_trade), where different correlated AI systems cooperate without any causal links between them.


Just Imitate Humans?

This post asks whether it is safe to build AI systems that just imitate humans. The comments have a lot of interesting debate.


Technical AGI safety research outside AI

This post lists 30 questions relevant to technical AI safety that could benefit from expertise outside of AI, divided into four categories: studying and understanding safety problems, solving safety problems, forecasting AI, and meta.


This post does not exhaust the possible success stories, but it gets us a lot closer to being able to look at a particular approach and ask, "Where exactly does this help us?" My guess is that most research ends up being only minimally helpful for the long run, and so I consider inquiry like this to be very useful for cause prioritization.

AI Safety "Success Stories"

It is difficult to measure the usefulness of various alignment approaches without clearly understanding what type of future they end up being useful for. This post collects "Success Stories" for AI -- disjunctive scenarios in which alignment approaches are leveraged to ensure a positive future. Whether these scenarios come to pass will depend critically on background assumptions, such as whether we can achieve global coordination, or solve the most ambitious safety issues. Mapping these success stories can help us prioritize research.


I am most excited about impact as a **regularizer** and impact as a **safety protocol**. I feel like AIs that are impact-limited at runtime (the **influence-limiter** case) are unlikely to be competitive with other AIs that have no impact penalty (this is discussed in the post). I found the argument that impact could be particularly useful for **deconfusion** uncompelling.

**Rohin's opinion**: It seems to me like the safety protocol argument is for limited actions at training time, while the influence limiter argument is for limited actions at test time. I don't really get how the regularizer is supposed to be different from these two cases -- perhaps the idea is that it is a regularizer specifically on the distribution over utility functions that the AI is optimizing? This is still confusing, I would have expected the influence limiter case to also be a change to the utility function. Like Asya, I am worried about competitiveness: see the post about [reversible changes](https://www.alignmentforum.org/posts/zrunBA8B5bmm2XZ59/reversible-changes-consider-a-bucket-of-water) below.

Four Ways An Impact Measure Could Help Alignment

Much <@recent@>(@Towards a New Impact Measure@) <@work@>(@Designing agent incentives to avoid side effects@) has focused on quantifying the effect an AI has on the world, aka measuring impact, though some are [skeptical](https://www.alignmentforum.org/posts/kCY9dYGLoThC3aG7w/best-reasons-for-pessimism-about-impact-of-impact-measures). This post presents four potential ways impact measures could help with AI alignment. _First_, impact could act as a **regularizer**: an untrained AI attempting to do value learning could have an impact penalty that prevents it from taking dangerous actions before it is confident it has learned the right utility function. _Second_, impact could act as a **safety protocol**: if our training process is dangerous, e.g. due to <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@), we can penalize impact during training to safely test models that may be misaligned. _Third_, impact could act as an **influence-limiter**: impact measures could help us construct AIs with intentionally limited scope that won't heavily optimize the world as a side effect. _Fourth_, impact could help us with **deconfusion**: even if impact measures themselves aren't used, conceptual clarity about impact could help us gain conceptual clarity about other important concepts such as corrigibility, mild optimization, etc.


On the purposes of decision theory research

In this post, Wei Dai clarifies that he thinks decision theory research is important because it can help us learn about the nature of rationality, philosophy, and metaphilosophy; it allows us to understand potential AI failure modes; we can better understand puzzles about intelligence such as free will, logical uncertainty, counterfactuals and more; and it could improve human rationality. It is _not_ meant to find the "correct" decision theory to program into an AI, nor to create safety arguments that show that an AI system is free of "decision-theoretic" flaws.


I strongly agree with the points made in this post. I don't know to what extent researchers themselves agree with this point -- it seems like there is _a lot_ of adversarial examples research that is looking at the imperceptible perturbation case and many papers that talk about new types of adversarial examples, without really explaining why they are doing this or giving a motivation that is about unsolved research problems rather than real world settings. It's possible that researchers do think of it as a research problem and not a real world problem, but present their papers differently because they think that's necessary in order to be accepted.

The distinction between research problems and real world threat models seem to parallel the distinction between theoretical or conceptual research and engineering in AI safety. The former typically asks questions of the form "how could we do this in principle, making simplifying assumptions X, Y and Z", even though X, Y and Z are known not to hold in the real world, for the sake of having greater conceptual clarity that can later be leveraged as a solution to a real world problem. Engineering work on the other hand is typically trying to scale an approach to a more complex environment (with the eventual goal of getting to a real world problem).

Unsolved research problems vs. real-world threat models

Papers on adversarial examples often suggest that adversarial examples can lead to real world problems as their motivation. As we've <@seen@>(@Motivating the Rules of the Game for Adversarial Example Research@) <@previously@>(@Introducing the Unrestricted Adversarial Examples Challenge@), many adversarial example settings are not very realistic _threat models_ for any real world problem. For example, adversarial "stickers" that cause vision models to fail to recognize stop signs could cause an autonomous vehicle to crash... but an adversary could also just knock over the stop sign if that was their goal.

There are more compelling reasons that we might care about imperceptible perturbation adversarial examples. First, they are a proof of concept, demonstrating that our ML models are not robust and make "obvious" mistakes and so cannot be relied on. Second, they form an unsolved research problem, in which progress can be made more easily than in real settings, because it can be formalized straightforwardly (unlike realistic settings). As progress is made in this toy domain, it can be used to inform new paradigms that are closer to realistic settings. But it is _not_ meant to mimic real world settings -- in the real world, you need a threat model of what problems can arise from the outside world, which will likely suggest much more basic concerns than the "research problems", requiring solutions involving sweeping design changes rather than small fixes.


I broadly agree with the areas of AI progress identified here, though I would probably also throw in NLP, e.g. [BERT](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html). I disagree on the details -- for example, I think that [OpenAI Five](https://blog.openai.com/openai-five/) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13)) was much better than I would have expected at the time and the same would have been true of AlphaStar if I hadn't already seen OpenAI Five, and the fact that they did a few things that humans can't do barely diminishes the achievement at all. (My take is pretty similar to Alex Irpan's take in his [post on AlphaStar](https://www.alexirpan.com/2019/02/22/alphastar.html).)

FLI Podcast: AI Breakthroughs and Challenges in 2018 with David Krueger and Roman Yampolskiy

Ariel Conn, David Krueger and Roman Yampolskiy

David and Roman review AI progress in 2018 and speculate about its implications. Roman identified a pattern where we see breakthroughs like [AlphaZero](https://deepmind.com/blog/alphazero-shedding-new-light-grand-games-chess-shogi-and-go/) ([AN #36](https://mailchi.mp/6751e45fbb48/alignment-newsletter-36)), [AlphaStar](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/) ([AN #43](https://mailchi.mp/768a8130013f/alignment-newsletter-43)) and [AlphaFold](https://deepmind.com/blog/alphafold/) ([AN #36](https://mailchi.mp/6751e45fbb48/alignment-newsletter-36)) so frequently now that it no longer seems as impressive when a new one comes out. David on the other hand sounded less impressed by progress on Dota and StarCraft, since both AI systems were capable of executing actions that humans could never do (fast reaction times for Dota and high actions-per-minute for StarCraft). He also thought that these projects didn't result in any clear general algorithmic insights the way AlphaZero did.

On the deep RL + robotics side, David identified major progress in [Dactyl](https://blog.openai.com/learning-dexterity/) ([AN #18](https://mailchi.mp/51717855ea27/alignment-newsletter-18)) and [QT-Opt](https://ai.googleblog.com/2018/06/scalable-deep-reinforcement-learning.html) (which I remember reading and liking but apparently I failed to put in the newsletter). He also cited GANs as having improved significantly, and talked about feature-wise transformations in particular. Roman noted the improving performance of evolutionary algorithms.

David also noted how a lot of results were obtained by creating algorithms that could scale, and then using a huge amount of compute for them, quoting [AI and Compute](https://blog.openai.com/ai-and-compute/) ([AN #7](https://mailchi.mp/3e550712419a/alignment-newsletter-7)), [Interpreting AI Compute Trends](https://aiimpacts.org/interpreting-ai-compute-trends/) ([AN #15](https://mailchi.mp/4920e52dd61b/alignment-newsletter-15)) and [Reinterpreting AI and Compute](https://aiimpacts.org/reinterpreting-ai-and-compute/) ([AN #38](https://mailchi.mp/588354e4b91d/alignment-newsletter-38)).

On the policy side, they talked about deep fakes and the general trend that AI may be progressing to fast for us to keep up with its security implications. They do find it promising that researchers are beginning to accept that their research does have safety and security implications.

On the safety side, David noted that the main advance seemed to be with approaches using [superhuman feedback](https://www.lesswrong.com/posts/naccwaCQEEBXK7hiJ/my-use-of-the-phrase-super-human-feedback), including [debate](https://blog.openai.com/debate/) ([AN #5](https://mailchi.mp/0ae5d69de63b/alignment-newsletter-5)), [iterated amplification](https://blog.openai.com/amplifying-ai-training/) (discussed frequently in this newsletter, but that paper was in [AN #30](https://mailchi.mp/c1f376f3a12e/alignment-newsletter-30)) and [recursive reward modeling](https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)). He also identified [unrestricted adversarial examples](https://ai.googleblog.com/2018/09/introducing-unrestricted-adversarial.html) ([AN #24](https://mailchi.mp/d7b5059d64ed/alignment-newsletter-24)) as an area to watch in the future.


This is probably most useful for understanding the motivations behind many of Dylan's papers and how they all tie into each other, which can be hard to glean just from reading the papers. There were also a lot of framings of problems that felt useful to me that I haven't seen elsewhere.

Inverse Reinforcement Learning and Inferring Human Preference with Dylan Hadfield-Menell

A few weeks ago, Lucas Perry interviewed Dylan Hadfield-Menell on the FLI podcast about his research (which includes papers like [Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137),  [The Off-Switch Game](https://arxiv.org/abs/1611.08219), and [Inverse Reward Design](https://arxiv.org/abs/1711.02827)). They discussed a variety of topics including the motivations behind Dylan's research, future directions, thoughts on hard problems such as corrigibility and preference aggregation, etc.


I generally agree with these arguments and the general viewpoint that we will probably solve alignment in this narrow sense. The most compelling argument to me is the second one, that we will eventually devote significant resources to the problem. This does depend on the crux that we see examples of these problems and how they could be dangerous before it is too late.

I also agree that it's much less clear whether we will solve other related problems, such as how to deal with malicious uses of AI, issues that arise when multiple superintelligent AI systems aligned with different humans start to compete, and how to ensure that humans have "good" values. I don't know if this implies that _on the margin_ it is more useful to work on the related problems. It could be that these problems are so hard that there is not much that we can do. (I'm neglecting [importance of the problem](https://concepts.effectivealtruism.org/concepts/importance-neglectedness-tractability/) here.)

Why I expect successful alignment

This post gives three arguments that we will likely solve the narrow alignment problem of having an AI system do what its operators intend it to do. First, advanced AI systems may be developed in such a way that the alignment problem doesn't even happen, at least as we currently conceive of it. For example, under the comprehensive AI services model, there are many different AI services that are superintelligent at particular tasks that can work together to accomplish complex goals, but there isn't a single unified agent to "align". Second, if it becomes obvious that alignment will be a serious problem, then we will devote a lot of resources to tackling the problem. We already see reward hacking in current systems, but it isn't sufficiently dangerous yet to merit the application of a lot of resources. Third, we have already come up with some decent approaches that seem like they could work.


This post concisely conveys a number of Vika and Jan's views, albeit without explanations for most of them. I'd encourage other safety researchers to do the same exercise, with a view to fleshing out the cruxes behind whatever disagreements come up.

Discussion on the machine learning approach to AI safety

This blog post (based on a talk at EA Global London) discusses whether current work on the machine learning approach to AI safety will remain relevant in the face of potential paradigmatic changes in ML systems. Vika and Jan rate how much they rely on each assumptions in a list drawn from [this blog post by Jon Gauthier] (http://www.foldl.me/2018/conceptual-issues-ai-safety-paradigmatic-gap/) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13)), and how likely each assumptions is to hold up over time. They also evaluate arguments for human-in-the-loop approaches versus problem-specific approaches.


I think it’s important to take this post as an explication of MIRI’s mindset, not as an argument for that mindset. In the former role, it’s excellent: the analogy is a fitting one in many ways. It’s worth noting, though, that the idea of only having one shot at success seems like an important component, but isn’t made explicit. Also, it’d be nice to have more clarity about the “approximately 0% chance of success” without advances in agent foundations - maybe that credence is justified under a specific model of what’s needed for AI alignment, but does it take into account model uncertainty?

The Rocket Alignment Problem

Eliezer explains the motivations behind MIRI's work using an analogy between aligning AI and designing a rocket that can get to the moon. He portrays our current theoretical understanding of intelligence as having massive conceptual holes; MIRI is trying to clarify these fundamental confusions. Although there's not yet any clear path from these sorts of advances to building an aligned AI, Eliezer estimates our chances of success without them as basically 0%: it's like somebody who doesn't understand calculus building a rocket with the intention of manually steering it on the way up.


My basic opinion is that this makes sense and agrees with my model. On the other hand, I'm not planning to switch to working on decision theory now, so perhaps I should say why. Partly it's that I have a comparative advantage at ML work, but it's also an impression that Agent Foundations will not help much with the first powerful AI systems we build. On one axis, I wouldn't be surprised if the first powerful AI systems don't look like the good reasoners that MIRI studies, and so Agent Foundations research won't apply. On another axis, Agent Foundations seems like a hard problem that we may not solve before powerful AI systems are created. I do find it plausible that to build _aligned_ AI systems that are _much_ more powerful than humans, we must understand it at the level of Agent Foundations understanding. (Though I also find the opposite statement plausible.) However, I think we will first build powerful AI systems that are not that much more powerful than humans, and that direct alignment of ML techniques will be sufficient to make that safe (even though they do pose an x-risk). (I suspect this is where my main disagreement with people at MIRI is.) We can then use those systems to help us solve Agent Foundations before we scale up.

Comment on decision theory

MIRI works on Agent Foundations because AGIs should be good reasoners, and we're currently confused about how to have good reasoning, and work on logical uncertainty and decision theory should help us resolve this confusion. If we don't resolve the confusion, it will be significantly harder to build AGI in a way that is clean, understandable and interpretable, and as a result it will be harder to understand what is happening and to fix it if anything goes wrong. This is analogous to how it seems really useful to understand Newton's law of gravitation before you try to build rockets, even though the work of figuring out Newton's law is very different from the rocket-building work.


This seems like a good way to generate good concrete empirical projects to work on. It does differ from the original post in placing less of an emphasis on “fuzzy” tasks, where aligned metrics are hard to come by, though it isn’t incompatible with it (in a “fuzzy” task, you probably still want as aligned a metric as you can get in order to measure progress).

Some criteria for sandwiching projects

This post outlines the pieces needed in order to execute a "sandwiching" project on <@aligning narrowly superhuman models@>(@The case for aligning narrowly superhuman models@), with the example of answering questions about a text when humans have limited access to that text. (Imagine answering questions about a paper, where the model can read the full paper but human labelers can only read the abstract.) The required pieces are:

1. **Aligned metric:** There needs to be some way of telling whether the project succeeded, i.e. the technique made the narrowly superhuman model more aligned. In the Q&A case, we get the aligned metric by seeing how humans answer when they can read the entire text.
2. **A narrowly superhuman model:** The model must have the capability to outperform the labelers on the task. In the Q&A case, we get this by artificially restricting the input that the labelers get (relative to what the model gets). In other cases we could use labelers who lack the relevant domain expertise that the model instead knows.
3. **Headroom on the aligned metric:** Baseline methods (such as training from labeler feedback) should not perform very well, so that there is room for a better technique to improve performance. It would be especially nice if making the model larger led to no improvement in the aligned metric; this would mean that we are working in a situation that is primarily an alignment failure.
4. **A natural plan of attack:** We have some approach for doing better than the baseline. For the Q&A example, we could train one model that selects the most relevant piece of text (by training on labelers' ratings of relevance) and another model that answers the question given that relevant piece.


Adaptive Mechanism Design: Learning to Promote Cooperation

Mechanism design for AI

One cause of outcomes worse than extinction could be escalating conflicts between very capable AI systems (that could eg. threaten to simulate suffering beings). It is worth studying how we could have AI systems implement mechanism design in order to guide such systems into more cooperative behavior.


High Impact Careers in Formal Verification: Artificial Intelligence

This post considers the applicability of formal verification techniques to AI alignment. Now in order to "verify" a property, you need a specification of that property against which to verify. The author considers three possibilities:

1. **Formally specifiable safety:** we can write down a specification for safe AI, _and_ we'll be able to find a computational description or implementation
2. **Informally specifiable safety:** we can write down a specification for safe AI mathematically or philosophically, but we will not be able to produce a computational version
3. **Nonspecifiable safety:** we will never write down a specification for safe AI.

Formal verification techniques are applicable only to the first case. Unfortunately, it seems that no one expects the first case to hold in practice: even CHAI, with its mission of building provably beneficial AI systems, is talking about proofs in the informal specification case (which still includes math), on the basis of comments like [these](https://www.alignmentforum.org/posts/nd692YfFGfZDh9Mwz/an-69-stuart-russell-s-new-book-on-why-we-need-to-replace?commentId=4LhBaSuYPyFvTnDrQ) in Human Compatible. In addition, it currently seems particularly hard for experts in formal verification to impact actual practice, and there doesn't seem to be much reason to expect that to change. As a result, the author is relatively pessimistic about formal verification as a route to reducing existential risk from failures of AI alignment.


I like it -- I think I will send this to newer researchers as a precursor to the full paper.

A Summary of Concrete Problems in AI Safety

A nice summary of [Concrete Problems in AI Safety](https://blog.openai.com/concrete-ai-safety-problems/) that's a lot quicker to read than the original paper.


I would be really excited to see good work on this agenda, it would be a big step forward on how good our design process for neural nets is.

Mechanistic Transparency for Machine Learning

One useful thread of alignment research would be to figure out how to take a neural net, and distill parts or all of it into pseudocode or actual code that describes how the neural net actually works. This could then be read and analyzed by developers to make sure the neural net is doing the right thing. Key quote: "I'm excited about this agenda because I see it as giving the developers of AI systems tools to detect and correct properties of their AI systems that they see as undesirable, without having to deploy the system in a test environment that they must laboriously ensure is adequately sandboxed."


Alignment of Language Agents

Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, Geoffrey Irving

This paper analyzes the various problems we consider in AI alignment from the perspective of language agents. Problems covered include <@specification gaming@>(@Specification gaming examples in AI@), <@whom and what to align to@>(@Artificial Intelligence, Values and Alignment@), <@intent alignment@>(@Clarifying "AI Alignment"@), <@removing tampering incentives@>(@Avoiding Tampering Incentives in Deep RL via Decoupled Approval@), and <@inner alignment@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@). These can be categorized as different kinds of misspecification, namely misspecification in the _training data_, the _training process_, and the _behavior under distributional shift_.

While the conceptual problems are similar to the ones already considered for embodied RL agents, the ways they manifest are different. In particular, the authors highlight the possibility that language agents will _deceive us_, _manipulate us_, or _produce harmful content_. The authors review some existing definitions of deception and manipulation that are purely behavioral (that is, the definitions do not require an _intent_ to deceive or manipulate). A signaller **deceives** a receiver if the signaller transmits (or suggestively doesn't transmit) a signal that causes the receiver to believe some false claim that benefits the signaller. **Manipulation** is similar, except rather than causing the receiver to believe a false claim, it causes the receiver to take some action that benefits the signaller, that in some sense the receiver "shouldn't" have taken. We could cash out "the receiver 'shouldn't' have taken the action" just as "the action is harmful to the receiver", but from a safety / security mindset, the authors prefer a broader definition that aims to identify bad _means_ of influencing the receiver, instead of only focusing on whether the _ends_ were bad.

Some other miscellaneous points:
- Since the "action space" is just language, it seems like it should be easier (though still requires work) to prevent language agents from causing physical harm.
- It will hopefully be easier to train language agents to be explainable, since they have native fluency in natural language with which they can explain their behavior.


If you have strongly suffering-focused ethics (unlike me), this seems mostly right. The post claims that suffering-focused AI safety should be more tractable than AI alignment, because it focuses on a subset of risks and only tries to minimize them. However, it's not necessarily the case that focusing on a simpler problem makes it easier to solve. It feels easier to me to figure out how to align an AI system to humans, or how to enable human control of an AI system, than to figure out all the ways in which vast suffering could happen, and solve each one individually. You can make an analogy to mathematical proofs and algorithms -- often, you want to try to prove a _stronger_ statement than the one you are looking at, because when you use induction or recursion, you can rely on a stronger inductive hypothesis.

An introduction to worst-case AI safety

Argues that people with suffering-focused ethics should focus on "worst-case AI safety", which aims to find technical solutions to risks of AIs creating vast amounts of suffering (which would be much worse than extinction).


This assumes an ontology where there exists a utility function that an AI is optimizing, and changes to the AI seem especially likely to change the utility function in a random direction. In such a scenario, yes, you probably should be worried. However, in practice, I expect that powerful AI systems will not look like they are explicitly maximizing some utility function. If you change some component of the system for the worse, you are likely to degrade its performance, but not likely to drastically change its behavior to cause human extinction. For example, even in RL (which is the closest thing to expected utility maximization), you can have serious bugs and still do relatively well on the objective. A public example of this is in OpenAI Five (https://blog.openai.com/openai-five/), but I also hear this expressed when talking to RL researchers (and see this myself). While you still want to be very careful with self-modification, it seems generally fine not to have a formal proof before making the change, and evaluating the change after it has taken place. (This would fail dramatically if the change drastically changed behavior, but if it only degrades performance, I expect the AI would still be competent enough to notice and undo the change.) It may be the case that adversarial subprocesses could take advantage of these sorts of bugs, but I expect that we need adversarial-subprocess-specific research to address this, not zero-shot reasoning.

Another take on agent foundations: formalizing zero-shot reasoning

There are strong incentives to build a recursively self-improving AI, and in order to do this without value drift, the AI needs to be able to reason effectively about the nature of changes it makes to itself. In such scenarios, it is insufficient to "reason with extreme caution", where you think really hard about the proposed change, and implement it if you can't find reasons not to do it. Instead, you need to do something like "zero-shot reasoning", where you prove under some reasonable assumptions that the proposed change is good. This sort of reasoning must be very powerful, enabling the AI to eg. build a spacecraft that lands on Mars, after observing Earth for one day. This motivates many of the problems in MIRI's agenda, such as Vingean reflection (self-trust), logical uncertainty (how to handle being a bounded reasoner), counterfactuals, etc., which all help to formalize zero-shot reasoning.


I think I agree with this post? Certainly for a superintelligence that is vastly smarter than humans, I buy this argument (and in general am not optimistic about solving alignment). However, humans seem to be fairly good at keeping each other in check, without a deep understanding of what makes humans tick, even though humans often do optimize against each other. Perhaps we can maintain this situation inductively as our AI systems get more powerful, without requiring a deep understanding of what's going on? Overall I'm pretty confused on this point.

Optimization Amplifies

One model of the difference between mathematicians and scientists is that a scientist is good at distinguishing between 0.01%, 50% and 99.99%, whereas a mathematician is good at distinguishing between 99.99% and 100%. Certainly it seems like if we can get 99.99% confidence that an AI system is aligned, we should count that as a huge win, and not hope for more (since the remaining 0.01% is extremely hard to get), so why do we need mathematicians? Scott argues that optimization is particularly special, in that the point of very strong optimization is to hit a very narrow target, which severely affects extreme probabilities, moving them from 0.01% to near-100%. For example, if you draw a million samples from a normal distribution and optimize for the largest one, it is almost certain to be 4 standard deviations above the mean (which is incredibly unlikely for a randomly chosen sample). In this sort of setting, the deep understanding of a problem that you get from a mathematician is still important. Note that Scott is _not_ saying that we don't need scientists, nor that we should aim for 100% certainty that an AI is aligned.


One possible conclusion is that safety research should be more diversified across different paradigms and artefacts, in order to properly maximize expected safety. However, this isn’t obvious: it seems likely that if the dominant paradigm has 50% of the research, it will also have, say, 80% of future real-world deployments, and so it could make sense to have 80% of the safety research focused on it. Rather than try to predict which paradigm will become dominant (a very difficult task), it may be more efficient to simply observe which paradigm becomes dominant and then redirect resources at that time (even though that process takes 5 years to happen).

AI Paradigms and AI Safety: Mapping Artefacts and Techniques to Safety Issues

Jose Hernandez-Orallo, Fernando Martinez-Plumed, Shahar Avin, Jess Whittlestone, Seán Ó hÉigeartaigh

What should prioritization within the field of AI safety look like? Ideally, we would proactively look for potential issues that could arise with many potential AI technologies, making sure to cover the full space of possibilities rather than focusing on a single area. What does prioritization look like in practice? This paper investigates, and finds that it is pretty different from this ideal.

In particular, they define a set of 14 categories of AI _techniques_ (examples include neural nets, planning and scheduling, and combinatorial optimization), and a set of 10 kinds of AI _artefacts_ (examples include agents, providers, dialoguers, and swarms). They then analyze trends in the amount of attention paid to each technique or artefact, both for AI safety and AI in general. Note that they construe AI safety very broadly by including anything that addresses potential real-world problems with AI systems.

While there are a lot of interesting trends, the main conclusion is that there is an approximately 5-year delay between the emergence of an AI paradigm and safety research into that paradigm. In addition, safety research tends to neglect non-dominant paradigms.


This is eminently sensible to me and I agree with the takeaways. See also [Methodology of Unbounded Analysis](https://arbital.com/p/unbounded_analysis/).

Infinite Data/Compute Arguments in Alignment

This reference post makes a short argument for why we might consider hypotheticals in which we have infinite data and compute. The core idea is that this allows us to focus on _hard subproblems_. Compute and data capacity have been growing substantially, and so it makes sense to treat them as "cheap"; the hard subproblems are then the ones that remain when we assume unlimited compute and data.

In particular, in this case we can get perfect predictive power, using Bayesian updates on low-level physics models, or Solomonoff induction. Indeed, most of ML tends to be about figuring out how to turn the problem of interest into a prediction or optimization problem, after which we use off-the-shelf algorithms. So the hard subproblems are the ones that arise even when you can use Bayesian updates on low-level physics models.


I was expecting a list of questions that were specific to AI somehow, but the actual list is very broad and feels to me like a list of subquestions of "how do you design a good society in the face of technological development". For example, it includes unemployment, system maintenance, the potential of blackmail, side-channel attacks, etc.

AI Services as a Research Paradigm

The <@CAIS report@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@) suggests that future technological development will be driven by _systems of AI services_, rather than a single monolithic AGI agent. However, there has not been much followup research since the publication of the report. This document posits that this is because the concepts of tasks and services introduced in the report are not amenable to formalization, and so it is hard to do research with them. So, it provides a simple abstract model of a system of services, in which each service takes some inputs, produces some outputs, and is trained with some loss function, and these services are interconnected with each other. (Some of the services could be provided by humans instead of AI systems.) It then lists several research problems that could be tackled under this generic view.


I fall more on the side of preferring indirect approaches, though by that I mean that we should delegate to future humans, as opposed to defining some particular value-finding mechanism into an AI system that eventually produces a definition of values.

The Value Definition Problem

This post considers the Value Definition Problem: what should we make our AI system <@try to do@>(@Clarifying "AI Alignment"@) to have the best chance of a positive outcome? It argues that an answer to the problem should be judged based on how much easier it makes alignment, how competent the AI system has to be to optimize it, and how good the outcome would be if it was optimized. Solutions also differ on how "direct" they are -- on one end, explicitly writing down a utility function would be very direct, while on the other, something like [Coherent Extrapolated Volition](https://intelligence.org/files/CEV.pdf) would be very indirect: it delegates the task of figuring out what is good to the AI system itself.


Drexler on AI Risk

This is another analysis of [Comprehensive AI Services](https://www.fhi.ox.ac.uk/reframing/). You can read [my summary of CAIS](https://www.alignmentforum.org/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services) ([AN #40](https://mailchi.mp/b649f32b07da/alignment-newsletter-40)) to get my views.


I don't know how hard it would be to create integrative biological simulations, but it does strike me as very useful if we did have them. If we had a complete mechanistic understanding of how intelligence happens in biological brains (in the sense that we can simulate them), the obvious next step would be to understand _how_ the mechanistic procedures lead to intelligence (in the same way that we currently try to understand why neural nets work). If we succeed at this, I would expect to get several insights into intelligence that would translate into significant progress in AI. However, I know very little about biological neurons and brains so take this with many grains of salt.

On the value learning side, it would be a good test of inverse reinforcement learning to see how well it could work on simple organisms, though it's not obvious what the ground truth is. I do want to note that this is specific to inverse reinforcement learning -- other techniques depend on uniquely human characteristics, like the ability to answer questions posed by the AI system.

Integrative Biological Simulation, Neuropsychology, and AI Safety

Gopal P. Sarma, Adam Safron, Nick J. Hay

This paper argues that we can make progress on AI capabilities and AI safety through integrative biological simulation, that is, a composite simulation of all of the processes involved in neurons that allow us to simulate brains. In the near future, such simulations would be limited to simple organisms like Drosophila, but even these organisms exhibit behavior that we find hard to replicate today using our AI techniques, especially at the sample efficiency that the organisms show. On the safety side, even such small brains share many architectural features with human brains, and so we might hope that we could discover neuroscience-based methods for value learning that generalize well to humans. Another possibility would be to create test suites (as in [AI Safety Gridworlds](https://arxiv.org/abs/1711.09883)) for simulated organisms.


I agree with the general point that figuring out a human utility function and then optimizing it is unlikely to work, but for different reasons (see the first chapter of the [Value Learning sequence](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc)). I also agree that humans are complex and you can’t get away with modeling them as Boltzmann rational and optimizing some fixed utility function. I wouldn’t try to make the model more accurate (eg. a model of a bunch of interacting subagents, each with their own utility function), I would try to make the model less precise (eg. a single giant neural net), because that reduces the chance of model misspecification. However, given the [impossibility result](https://arxiv.org/abs/1712.05812) saying that you must make assumptions to make this work, we probably have to give up on having some nice formally specified meaning of “values”. I think this is probably fine -- for example, iterated amplification doesn’t have any explicit formal value function.

Multi-agent minds and AI alignment

This post argues against the model of humans as optimizing some particular utility function, instead favoring a model based on predictive processing. This leads to several issues with the way standard value learning approaches like inverse reinforcement learning work. There are a few suggested areas for future research. First, we could understand how hierarchical models of the world work (presumably for better value learning). Second, we could try to invert game theory to learn objectives in multiagent settings. Third, we could learn preferences in multiagent settings, which might allow us to better infer norms that humans follow. Fourth, we could see what happens if we take a system of agents, infer a utility function, and then optimize it -- perhaps one of the agents' utility functions dominates? Finally, we can see what happens when we take a system of agents and give it more computation, to see how different parts scale. On the non-technical side, we can try to figure out how to get humans to be more self-aligned (i.e. there aren't "different parts pulling in different directions").


ICML Uncertainty and Robustness Workshop Accepted Papers

The Uncertainty and Robustness Workshop accepted papers are available. Topics include out-of-distribution detection, generalization to stochastic corruptions, label corruption robustness, and so on.


Previous calibration work in machine learning conferences would often to focus on calibrating regression models, but this work has renewed interest in calibrating classifiers. For that reason I view this paper highly. That said, this paper’s evaluation measure, the “Expected Calibration Error” is not a proper scoring rule, so optimizing this does not necessarily lead to calibration. In their approximation of the ECE, they use equally-wide bins when there is reason to use adaptively sized bins. Consequently I think [Nguyen and O’Connor](https://arxiv.org/pdf/1508.05154.pdf) Sections 2 and 3 provide a better calibration explanation, better calibration measure, and better estimation procedure. They also suggest using a convex optimization library to find the softmax temperature, but at least libraries such as CVXPY require far more time and memory than a simple softmax temperature grid search. Finally, an understandable limitation of this work is that it assumes test-time inputs are in-distribution, but when inputs are out-of-distribution this method hardly improves calibration.

On Calibration of Modern Neural Networks

Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger

Models should not be unduly confident, especially when said confidence is used for decision making or downstream tasks. This work provides a simple method to make models more calibrated so that the confidence estimates are closer to the true correctness likelihood. (For example, if a calibrated model predicts "toucan" with 60% confidence, then 60% of the time the input was actually a toucan.) Before presenting their method, they observe that batch normalization can make models less calibrated, while unusually large weight decay regularization can increase calibration. However, their proposed approach to increase calibration does not impact accuracy or require substantive model changes. They simply adjust the temperature of the softmax to make the model's "confidence" (here the maximum softmax probability) more calibrated. Specifically, after training they tune the softmax temperature to minimize the cross entropy (negative average log-likelihood) on validation data. They then measure model calibration with a measure which is related to the Brier score, but with absolute values rather than squares.


I like this paper: the intuition makes sense, the math is straightforward, and the empirical results are strong, and have continued to be strong when looking at later work that builds on it.

Representation Learning with Contrastive Predictive Coding

Aaron van den Oord, Yazhe Li, Oriol Vinyals

This paper from 2018 proposed Contrastive Predictive Coding (CPC): a method of unsupervised learning that has been quite successful. At its core it is quite simple: it simply combines the ideas of predictive coding and contrastive losses, both of which have been significantly studied in the past.

The simplest form of unsupervised learning would be data compression via generative models (as in e.g. VAEs), in which, to model the data **p(x)**, you attempt to encode **x** into a latent (hidden) state **z** in such a way that you can then recover the original data point **x** from **z**. Intuitively, we want **z** to have high mutual information with **x**.

For sequential data in a partially observed setting, you need to deal with the full sequence. Consider natural language: in this setting, each x would be a single word. Consider the sentence "I sat on the chair". If the **z** corresponding to the word "the" only has to reconstruct the word "the", it's not going to "remember" that the past context involved sitting, and so that **z** would be terrible at predicting that the next word will be chair. To fix this, we can use predictive coding, where we instead require that we can _predict_ future words using **z**. This now incentivizes **z_t** to have high mutual information with **x_{t+k}**.

There is still a problem: reconstructing the entire input **x** would require a lot of irrelevant information, such as e.g. the background color of the environment in RL, even if that never changes. How can we get rid of these irrelevant features? Contrastive losses allow us to do this: intuitively, since the irrelevant features are the ones that are common across all the **x**s (and so are fully captured by **p(x)** ), if we train the neural net to _distinguish_ between various **x**s, we can incentivize only the relevant features. In particular, given a latent state **z_t**, we take the true **x_{t+k}**, and throw in a bunch of other **x**s sampled from **p(x)** (known as _negative samples_), and train the network to correctly classify **x_{t+k}**. The authors show that the optimum of this loss function is indeed for the neural net to compute **p(x | z) / p(x)**, which implies that it is maximizing a lower bound on the mutual information between X and Z.

This gives us a pretty simple overall algorithm. Take a sequence **x_1 ... x_T**, compute **z_t** using a recurrent model on **x_1 ... x_t**, put **x_{t+k}** and some negative samples into a set, and train a classifier to correctly predict which of the samples is the true **x_{t+k}**. In practice, we do batches of these at the same time, and for every data point in the batch we use all of the other data points as our negative examples. The features you learn are then the ones that help _distinguish_ between **x_{t+k}** and the negative samples, and you'll ignore any features that are common across all the samples. This means that the results depend quite a lot on how you choose your samples (this effectively determines what **p(x)** you are using).

The authors evaluate their algorithm on several domains and show that it achieves or surpasses state of the art on them.


On Variational Bounds of Mutual Information

Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, George Tucker

This paper is a pretty dense and technical explanation of various ways in which we can estimate and/or optimize the mutual information between two variables. I specifically want to highlight that it provides a proof that the Contrastive Predictive Coding objective (summarized above) is a lower bound on the mutual information between the input and the representation, and compares it to other lower bounds on mutual information.


I really like this paper; I think it shines a light in an interesting new direction, and I look forward to seeing future work that builds on this in theoretical, mechanistic, and applied manners. On the other hand, I felt they did gloss over how exactly they do curriculum learning, and their reinforcement learning experiment was a little unclear to me.

I think the implications of this work are enormous. In a future where we might be limited by the maturity of available simulation platforms or inundated by deluges of data with little marginal information, this approach can circumvent such problems for the selection and (pre)training of suitable student networks.

Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, Jeff Clune

The Generative Teaching Networks (GTN) paper breaks new ground by training generators that produce synthetic data that can enable learner neural networks to learn faster than when training on real data. The process is as follows: The generator produces synthetic training data by transforming some sampled noise vector and label; a newly-initialized learner is trained on this synthetic data and evaluated on real data; the error signal from this evaluation is backpropagated to the generator via meta-gradients, to enable it to produce synthetic samples that will train the learner networks better. They also demonstrate that their curriculum learning variant, where the input vectors and their order are learned along with generator parameters, is especially powerful at teaching learners with few samples and few steps of gradient descent.

They apply their system to neural architecture search, and show an empirical correlation between performance of a learner on synthetic data and its eventual performance when trained on real data. In this manner, they make the argument that data from a trained GTN can be used to cheaply assess the likelihood of a given network succeeding to learn on the real task, and hence GTN data can tremendously speed up architecture search.


I really liked this paper; it was cleanly written, conceptually straightforward, and did a generally useful scientific service of checking whether an advance in one area might change our beliefs about a previous result. I particularly enjoyed looking at the "reconstructed" images they got by running their encoder and then generator: more so than anything I recall seeing from a VAE pixel-based reconstructor, this model seems to be treating images as valid reconstructions of one another if they're of the same class (i.e. two pizzas) even if the colors and low level detail are different. This makes reasonable sense if you think that those two pizzas are probably nearby in latent space, and so each is a plausible reconstruction of each other's latent space encoding, but it's still cool to see concretely borne out.

Large Scale Adversarial Representation Learning

The BigGAN paper, published last September, used a much larger model (and a handful of optimization tricks to facilitate training it) to achieve a huge leap forward in the quality of generated images. However, it was unclear from the earlier paper whether this improvement in generation quality would also be tied to an increase in the model's usefulness as a source of unsupervised semantic representations of images.  This paper set out to answer that question by taking an existing technique for learning representations with GANs - called BiGAN - and combining it with the BigGan architecture, which hadn't been available when BiGAN was originally published. BiGAN, short for Bidirectional GAN, works by learning both a latent space to image transformation, and also an image to latent space encoder, and then enforcing that pairs of (latent, image) from these two distributions be indistinguishable from one another. They evaluated the quality of learned representations by measuring the performance of a linear model trained using the  encoder's learned latent vectors as input, and did find it to be the case that a BiGAN trained with a BigGAN architecture performs better than one trained with a smaller architecture.


This is a clean, clear summary, but one without any real technical depth or detail; this would be a good writeup to hand someone without any machine learning background who wanted to get an intuitive grasp for unsupervised learning as a field.

Unsupervised learning: the curious pupil

A high-level but well-written explanation of why many believe unsupervised learning will be key to achieving general intelligence, touching on the approaches of GANs and autoregressive models as examples.


This strikes me as a really valuable injection of empirical realism, of the kind that tends to be good for research fields to have periodically, even if it can be a bit painful or frustrating. I appreciate in particular the effort and clarity that this paper puts into articulating the implicit assumptions of how disentanglement can be used or evaluated, and trying to test those assumptions under more real-world settings, such as the one where you don't have any ground truth factors of variation, since the real world doesn't tend to just hand out the Correct factorized model of itself.

Evaluating the Unsupervised Learning of Disentangled Representations

This blog post and paper describe a Google-scale comparative study of different representation learning methods designed to learn "disentangled" representations, where the axes of the representation are aligned with the true underlying factors generating the data. The paper's claims are a sobering result for the field, both theoretically and empirically. Theoretically, they show that in an unsupervised context, it's not possible to find a disentangled representation without embedding some form of inductive bias into your model. Empirically, they present evidence suggesting that variation between random seeds for a given hyperparameter setting (in particular, regularization strength) matters as much or more than variation between that hyperparameter's values. Finally, they run experiments that call into question whether disentangled representations actually support transfer learning, or can be identified as in fact being disentangled without using a metric that relies on having ground truth factors of variation to begin with, making it difficult to evaluate on the many realistic contexts where these aren't available.


This is a cool technique; I like the combination of two approaches (meta-learning and unsupervised learning) aimed at making deep learning applicable to many more real-world datasets. I can imagine promising follow-ups - e.g. randomly scaling embedding dimensions to get different clusters seems a bit hacky to me, so I wonder if there's a better approach (maybe learning many different embeddings?). It's interesting to note that their test-time performance is sometimes better than their training performance, presumably because some of the unsupervised training clusterings are "nonsensical", so there is room to improve here.

Unsupervised Learning via Meta-Learning

This paper trains a meta-learner on tasks which were generated using unsupervised learning. This is done by first learning an (unsupervised) embedding for a dataset, then clustering in that embedding space using k-means. Clustering is done many times with random scaling on each dimension; each meta-learning task is then based on one set of clusters. The resulting meta-learner is then evaluated on the actual task for that dataset, performing better than approaches based just on embeddings, and sometimes getting fairly close to the supervised-learning equivalent.


While the Infomin principle seems powerful and their results look impressive, I am not really convinced that the principle actually played an important role in finding the image augmentations they ended up using, as there is little description of how that happened and the augmentations rather look like the result of combining previously used approaches and doing some hyperparameter optimization.

What makes for good views for contrastive learning

Understanding View Selection for Contrastive Learning

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, Phillip Isola

<@Contrastive multiview learning@>(@Representation Learning with Contrastive Predictive Coding@) is a self-supervised approach to pretraining classifiers in which different views of data points are created and an encoder is trained to minimize the distance between encodings of views corresponding to data points with the same label while maximizing the distance between encodings of views with different labels.

The efficacy of this approach depends on the choice of views as well as the downstream task the neural network is going to be trained for. To find the most promising views, the authors propose the Infomin principle: all views should keep task-relevant information while the mutual information between views is minimized. The principle is supported by various observations: Firstly, earlier approaches to contrastive learning in the image domain that use data augmentation to preserve object identity while creating diverse views can be seen as an implicit application of the Infomin principle. Secondly, varying the mutual information between views (for example by changing the distance between two cropped views of the same image) creates an inverted U-curve for downstream performance corresponding to poor performance if there is too much or too little mutual information between the views. Lastly, the authors also find an inverted U-curve in performance for different colour spaces when using channels as views and the Lab colour space which was built to mimic human colour perception is close to the optimum, meaning that human colour perception might be near-optimal for self-supervised representation learning.

The authors then use the Infomin principle to select image augmentations for contrastive pretraining and improve the state of the art in linear readout on ImageNet from 69.3% to 73% for Top-1 accuracy and from 89% to 91.1% for Top-5 accuracy.


It continues to baffle me why model distillation is so helpful -- you’d think that if you train a student model to mimic a teacher model, it would do at most as well as the teacher, but in fact it seems to do better. It's remarkable that just "training a bigger model and then distilling it down" leads to an increase of _16 percentage points_ (when we just have 1% of the labels). Another thing to add to the list of weird empirical facts about deep learning that we don’t understand.

Big Self-Supervised Models are Strong Semi-Supervised Learners

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton

Previously, <@SimCLR@>(@A Simple Framework for Contrastive Learning of Visual Representations@) showed that you can get good results on semi-supervised learning on ImageNet, by first using self-supervised learning with a contrastive loss to learn good representations for images, and then finetuning a classifier on top of the representations with very few labels. This paper reports a significantly improved score, using three main improvements:

1. Making all of the models larger (in particular, deeper).
2. Incorporating momentum contrast, as done <@previously@>(@Improved Baselines with Momentum Contrastive Learning@).
3. Using model distillation to train a student network to mimic the original finetuned classifier.

On linear classification on top of learned features with a ResNet-50 architecture, they get a top-1 accuracy of 71.7%, so lower than the previous paper. Their main contribution is to show what can be done with larger models. According to top-1 accuracy on ImageNet, the resulting system gets 74.9% with 1% of labels, and 80.1% with 10% of labels. In comparison, standard supervised learning with a ResNet-50 (which is about 33x smaller) achieves 76.6% with all labels, and just 57.9% with 1% of labels and 68.4% with 10% of labels. When they distill down their biggest model into a ResNet-50, it gets 73.9% with 1% of labels and 77.5% with 10% of labels.


As I (Flo) <@previously@>(@The Quiet Semi-Supervised Revolution@) argued, forcing representations to respect invariances in the task seems likely to yield further improvements when labelled data is sparse. However, my intuition is that invariances will be a lot harder to identify for RL or advanced NLP tasks like coming up with consistent novel arguments, so I am unsure how much these improvements will directly matter for our trajectory to transformative AI.

Advancing Self-Supervised and Semi-Supervised Learning with SimCLR

A Simple Framework for Contrastive Learning of Visual Representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton

Contrastive learning is a major recent development, in which we train a neural net to learn representations by giving it the task of maximizing "agreement" between similar images, while minimizing it across dissimilar images. It has been used to achieve excellent results with semi-supervised learning on ImageNet.

The authors performed a large empirical study of contrastive learning. Their framework consists of three components. First, the _data augmentation method_ specifies how to get examples of "similar images": we simply take an (unlabeled) training image, and apply data augmentations to it to create two images that both represent the same underlying image. They consider random crops, color distortion, and Gaussian blur. Second is the _neural network architecture_, which is split into the first several layers f() which compute the representation from the input, and the last few layers g() which compute the similarity from the representation. Finally, the _contrastive loss function_ defines the problem of maximizing agreement between similar images, while minimizing agreement between dissimilar images. They primarily use the same InfoNCE loss used in <@CPC@>(@Representation Learning with Contrastive Predictive Coding@).

They then show many empirical results, including:
1. Having a simple linear layer in g() is not as good as introducing one hidden layer, or in other words, the representations in the penultimate layer are more useful than those in the final layer.
2. Larger batch sizes, longer training, and larger networks matter even more for unsupervised contrastive learning than they do for supervised learning.


Momentum Contrast for Unsupervised Visual Representation Learning

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick

In most deep learning settings, the batch size primarily controls the variance of the gradient, with higher batch sizes decreasing variance. However, with typical contrastive learning, batch size also determines the _task_: typically, the task is to maximize agreement between two examples in the batch, and minimize agreement with all the other examples in the batch. Put another way, given one input, you have to correctly classify which of the remaining examples in the minibatch is a differently transformed version of that input. So, the batch size determines the number of negative examples.

So, besides decreasing variance, large batch sizes also increase the difficulty of the task to be solved. However, such large batch sizes are hard to fit into memory and are computationally expensive. This paper proposes _momentum contrast_ (MoCo), in which we get large numbers of negative examples for contrastive learning, while allowing for small batch sizes.

Think of contrastive learning as a dictionary lookup task -- given one transformed image (the query), you want to find the same image transformed in a different way out of a large list of images (the keys). The key idea of this paper is to have the minibatch contain queries, while using all of the previous N minibatches as the keys (for some N > 1), allowing for many negative examples with a relatively small minibatch.

Of course, this wouldn't help us if we had to encode the keys again each time we trained on a new minibatch. So, instead of storing the images directly as keys, we store their _encoded representations_ in the dictionary, ensuring that we don't have to rerun the encoder every iteration on all of the keys. This is where the computational savings come from.

However, the encoder is being updated over time, which means that different keys are being encoded differently, and there isn't a consistent kind of representation against which similarity can be computed. To solve this, the authors use a momentum-based version of the encoder to encode keys, which ensures that the key encodings change slowly and smoothly, while allowing the query encoder to change rapidly. This means that the query representation and the key representations will be different, but the layers on top of the representations can learn to deal with that. What's important is that _within_ the key representations, the representations are approximately consistent.


Improved Baselines with Momentum Contrastive Learning

Xinlei Chen, Haoqi Fan, Ross Girshick, Kaiming He

This paper applies the insights from the SimCLR paper to the MoCo framework: it adds an extra hidden layer on top of the representations while training on the contrastive loss, and adds the blur data augmentation. This results in a new SOTA on self-supervised representation learning for images.


I’m not sure why we should expect this method to work, but empirically it does. Presumably I’d understand the motivation better if I read through all the related work it’s building on.

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin

There has been a lot of work in self-supervised representation learning for image classification (previously summarized in [AN #92](https://mailchi.mp/d7e950bc8dbd/an-92learning-good-representations-with-contrastive-predictive-coding) and [AN #99](https://mailchi.mp/4f7ffc5cbe53/an-99-doubling-times-for-the-efficiency-of-ai-algorithms)). This paper sets a new SOTA of 75.3% top-1 ImageNet accuracy, when allowed to first do self-supervised representation learning on ImageNet, and then to train a linear classifier on top of the learned features using all of ImageNet.

Previous methods use a contrastive loss across the learned representations (possibly after being processed by a few MLP layers), which can be thought of as using the learned representation to predict the representation of augmented versions of the same input. In contrast, this paper uses the representation to predict "codes" of augmented versions, where the codes are computed using clustering.


Conclusion to the sequence on value learning

This post summarizes the value learning sequence, putting emphasis on particular parts. I recommend reading it in full -- the sequence did have an overarching story, which was likely hard to keep track of over the three months that it was being published.


Future directions for narrow value learning

This post summarizes some future directions for narrow value learning that I'm particularly interested in from a long-term perspective.


The human side of interaction

The lens of [human-AI interaction](https://www.alignmentforum.org/posts/4783ufKpx8xvLMPc6/human-ai-interaction) ([AN #41](https://mailchi.mp/8c3f02cabccd/alignment-newsletter-41)) also suggests that we should focus on what the _human_ should do in AI alignment.

Any feedback that the AI system gets must be interpreted using some assumption. For example, when a human provides an AI system a reward function, it shouldn't be interpreted as a description of optimal behavior in every possible situation (which is what we currently do implicitly). [Inverse Reward Design](https://arxiv.org/abs/1711.02827) (IRD) suggests an alternative, more realistic assumption: the reward function is likely to the extent that it leads to high true utility in the training environment. Similarly, in inverse reinforcement learning (IRL) human demonstrations are often interpreted under the assumption of Boltzmann rationality.

Analogously, we may also want to train humans to give feedback to AI systems in the manner that they are expecting. With IRD, the reward designer should make sure to test the reward function extensively in the training environment. If we want our AI system to help us with long-term goals, we may want the overseers to be much more cautious and uncertain in their feedback (depending on how such feedback is interpreted). Techniques that learn to reason like humans, such as iterated amplification and debate, would by default learn to interpret feedback the way humans do. Nevertheless it will probably be useful to train humans to provide useful feedback: for example, in debate, we want humans to judge which side provided more true and useful information.


Following human norms

One approach to preventing catastrophe is to constrain the AI system to never take catastrophic actions, and not focus as much on what to do (which will be solved by progress in AI more generally). In this setting, we hope that our AI systems accelerate our rate of progress, but we remain in control and use AI systems as tools that allow us make better decisions and better technologies. Impact measures / side effect penalties aim to _define_ what not to do. What if we instead _learn_ what not to do? This could look like inferring and following human norms, along the lines of [ad hoc teamwork](http://www.cs.utexas.edu/users/ai-lab/?AdHocTeam).

This is different from narrow value learning for a few reasons. First, narrow value learning also learns what _to_ do. Second, it seems likely that norm inference only gives good results in the context of groups of agents, while narrow value learning could be applied in singe agent settings.

The main advantages of learning norms is that this is something that humans do quite well, so it may be significantly easier than learning "values". In addition, this approach is very similar to our ways of preventing humans from doing catastrophic things: there is a shared, external system of norms that everyone is expected to follow. However, norm following is a weaker standard than [ambitious value learning](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/5eX8ko7GCxwR5N9mN) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31)), and there are more problems as a result. Most notably, powerful AI systems will lead to rapidly evolving technologies, that cause big changes in the environment that might require new norms; norm-following AI systems may not be able to create or adapt to these new norms.


Reward uncertainty

Given that we need human feedback for the AI system to stay "on track" as the environment changes, we might design a system that keeps an estimate of the reward, chooses actions that optimize that reward, but also updates the reward over time based on feedback. This has a few issues: it typically assumes that the human Alice knows the true reward function, it makes a possibly-incorrect assumption about the meaning of Alice's feedback, and the AI system still looks like a long-term goal-directed agent where the goal is the current reward estimate.

This post takes the above AI system and considers what happens if you have a distribution over reward functions instead of a point estimate, and during action selection you take into account future updates to the distribution. (This is the setup of [Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137).) While we still assume that Alice knows the true reward function, and we still require an assumption about the meaning of Alice's feedback, the resulting system looks less like a goal-directed agent.

In particular, the system no longer has an incentive to disable the system that learns values from feedback: while previously it changed the AI system's goal (a negative effect from the goal's perspective), now it provides more information about the goal (a positive effect). In addition, the system has more of an incentive to let itself be shut down. If a human is about to shut it down, it should update strongly that whatever it was doing was very bad, causing a drastic update on reward functions. It may still prevent us from shutting it down, but it will at least stop doing the bad thing. Eventually, after gathering enough information, it would converge on the true reward and do the right thing. Of course, this is assuming that the space of rewards is well-specified, which will probably not be true in practice.


Ambitious vs. narrow value learning

While previously I defined narrow value learning as obtaining good behavior in a particular domain, this post defines it as learning the narrow subgoals and instrumental values that the human is pursuing. I believe that these are pointing at the same underlying concept. There are a few reasons to be optimistic about using only narrow value learning to build very powerful AI systems. First, it should be relatively easy to infer many instrumental goals that humans have, such as "acquiring more resources under my control", "better understanding the world and what I want", "remaining in control of deployed AI systems", etc. which an AI system could then pursue with all of its ingenuity. Second, we could infer enough of human preferences to keep humans in a safe environment where they can deliberate in order to figure out what they want to do with the future. Third, humans could use these narrow AI systems as tools in order to implement sophisticated plans, allowing them to perform tasks that we would currently consider to be beyond human ability.