I do agree that in the framework outlined in this post (the "ideal" being just a search over "feasible" strategies) the ideal solution doesn't give you much insight, but I don't think this is fully true of eg. Bayes rule. I do think that understanding Bayes rule can help you [make better decisions](https://80000hours.org/podcast/episodes/spencer-greenberg-bayesian-updating/), because it gives you a quantitative framework of how to work with hypotheses and evidence, which even simple feasible strategies can use. (Although I do think that logically-omniscient Bayes does not add much over regular Bayes rule from the perspective of suggesting a feasible strategy to use -- but in the world where logically-omniscient Bayes came first, it would have been helpful to derive the heuristic.) In the framework of the post, this corresponds to the choice of "weight" assigned to each hypothesis, and this is useful because feasible strategies do still look like search (but instead of searching over all hypotheses, you search over a very restricted subset of them). So overall I think I agree with the general thrust of the post, but don't agree with the original strong claim that 'grappling with embeddedness properly will inevitably make theories of this general type irrelevant or useless, so that "a theory like this, except for embedded agents" is not a thing that we can reasonably want'.

When does rationality-as-search have nontrivial implications?

Many theories of idealized intelligence, such as Solomonoff induction, logical inductors and Bayesian reasoning, involve a large search over a space of strategies and using the best-performing one, or a weighted combination where the weights depend on past performance. However, the procedure that involves the large search is not itself part of the space of strategies -- for example, Solomonoff induction searches over the space of computable programs to achieve near-optimality at prediction tasks relative to any computable program, but is itself uncomputable. When we want to actually implement a strategy, we have to choose one of the options from our set, rather than the infeasible idealized version, and the idealized version doesn't help us do this. It would be like saying that a chess expert is approximating the rule "consult all possible chess players weighted by past performance" -- it's true that these will look similar _behaviorally_, but they look very different _algorithmically_, which is what we actually care about for building systems.


This is an interesting theoretical analysis, and I'm surprised that the traditional ML approach seems to do so well in a context it wasn't designed for. I'm not sure about the part where it would converge to a fixed point of the function g, I've written a rambly comment on the post trying to explain more.

Standard ML Oracles vs Counterfactual ones

_(Note: This summary has more of my interpretation than usual.)_ Consider the setting where an AI system is predicting some variable y_t = f(x_t), but we will use the AI's output to make decisions that could affect the true value of y_t. Let's call the AI's prediction z_t, and have y_t = g(x_t, z_t), where g captures how humans use z_t to affect the value of y_t. The traditional ML approach would be to find the function f that minimizes the distance between y and f(x) on past examples, but this does not typically account for y_t depending on z_t. We would expect that it would converge to outputting a fixed point of g (so that y_t = z_t = g(x_t, z_t)), since that would minimize its loss. This would generally perform well; while manipulative predictions z_t are possible, they are unlikely. The main issue is that since the system does not get to observe z_t (since that is what it is predicting), it cannot model the true causal formulation, and has to resort to complex hypotheses that approximate it. This can lead to overfitting that can't be simply solved by regularization or simplicity priors. Instead, we could use a counterfactual oracle, which reifies the prediction z_t and then outputs the z_t that minimizes the distance between z_t and y_t, which allows it to model the causal connection between y_t = g(x_t, z_t).


Rohin Shah on the State of AGI Safety Research in 2021

As in [previous](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) [years](https://futureoflife.org/2019/04/25/an-overview-of-technical-ai-alignment-with-rohin-shah-part-2/) ([AN #54](https://mailchi.mp/3e2f43012b07/an-54-boxing-a-finite-horizon-ai-system-to-keep-it-unambitious)), on this FLI podcast I talk about the state of the field. Relative to previous years, this podcast is a bit more introductory, and focuses a bit more on what I find interesting rather than what the field as a whole would consider interesting.


I enjoyed this article quite a lot; it is _really_ thorough. I do see a lot of my own work as pushing on some of these more incremental methods for increasing caution, though I think of it more as a combination of generating more or better evidence and communicating arguments in a manner more suited to a particular audience. Perhaps I will think of new strategies that aim to reduce fear shame instead.

Beyond fire alarms: freeing the groupstruck

It has been claimed that [there’s no fire alarm for AGI](https://intelligence.org/2017/10/13/fire-alarm/), that is, there will be no specific moment or event at which AGI risk becomes sufficiently obvious and agreed upon, so that freaking out about AGI becomes socially acceptable rather than embarrassing. People often implicitly argue for waiting for an (unspecified) future event that tells us AGI is near, after which everyone will know that it’s okay to work on AGI alignment. This seems particularly bad if no such future event (i.e. fire alarm) exists.

This post argues that this is not in fact the implicit strategy that people typically use to evaluate and respond to risks. In particular, it is too discrete. Instead, people perform “the normal dance of accumulating evidence and escalating discussion and brave people calling the problem early and eating the potential embarrassment”. As a result, the existence of a “fire alarm” is not particularly important.

Note that the author does agree that there is some important bias at play here. The original fire alarm post is implicitly considering a _fear shame hypothesis_: people tend to be less cautious in public because they expect to be negatively judged for looking scared. The author ends up concluding that there is something broader going on and proposes a few possibilities, many of which still suggest that people will tend to be less cautious around risks when they are observed.

Some points made in the very detailed, 15,000-word article:

1. Literal fire alarms don’t work by creating common knowledge, or by providing evidence of a fire. People frequently ignore fire alarms. In [one experiment](https://www.youtube.com/watch?v=MtX-10c3fT0), participants continued to fill out questionnaires while a fire alarm rang, often assuming that someone will lead them outside if it is important.

2. They probably instead work by a variety of mechanisms, some of which are related to the fear shame hypothesis. Sometimes they provide objective evidence that is easier to use as a justification for caution than a personal guess. Sometimes they act as an excuse for cautious or fearful people to leave, without the implication that those people are afraid. Sometimes they act as a source of authority for a course of action (leaving the building).

3. Most of these mechanisms are amenable to partial or incremental effects, and in particular can happen with AGI risk. There are many people who have already boldly claimed that AGI risk is a problem. There exists person-independent evidence; for example, surveys of AI researchers suggest a 5% chance of extinction.

4. For other risks, there does not seem to have been a single discrete moment at which it became acceptable to worry about them (i.e. no “fire alarm”). This includes risks where there has been a lot of caution, such as climate change, the ozone hole, recombinant DNA, COVID, and nuclear weapons.

5. We could think about _building_ fire alarms; many of the mechanisms above are social ones rather than empirical facts about the world. This could be one out of many strategies that we employ against the general bias towards incaution (the post suggests 16).


I know I already harped on this in the summary, but these numbers are ridiculously non-robust and involve tons of selection biases. You probably shouldn’t conclude much from them about how much risk from AI there really is. Don’t be the person who links to this survey with the quote “experts predict 30% chance of doom from AI”.

"Existential risk from AI" survey results

This post reports on the results of a survey sent to about 117 people working on long-term AI risk (of which 44 responded), asking about the magnitude of the risk from AI systems. I’d recommend reading the exact questions asked, since the results could be quite sensitive to the exact wording, and as an added bonus you can see the visualization of the responses. In addition, respondents expressed _a lot_ of uncertainty in their qualitative comments. And of course, there are all sorts of selection effects that make the results hard to interpret.

Keeping those caveats in mind, the headline numbers are that respondents assigned a median probability of 20% to x-risk caused due to a lack of enough technical research, and 30% to x-risk caused due to a failure of AI systems to do what the people deploying them intended, with huge variation (for example, there are data points at both ~1% and ~99%).


The notion of “natural abstractions” seems quite important to me. There are at least some weak versions of the hypothesis that seem obviously true: for example, if you ask GPT-3 some new type of question it has never seen before, you can predict pretty confidently that it is still going to respond with real words rather than a string of random characters. This is effectively because you expect that GPT-3 has learned the “natural abstraction” of the words used in English and that it uses this natural abstraction to drive its output (leaving aside the cases where it must produce output in some other language).

The version of the natural abstraction hypothesis investigated here seems a lot stronger and I’m excited to see how the project turns out. I expect the author will post several short updates over time; I probably won’t cover each of these individually and so if you want to follow it in real time I recommend following it on the Alignment Forum.

Section 1.3 of [this version of the paper](https://mltheory.org/dg_short.pdf)

Testing The Natural Abstraction Hypothesis: Project Intro

We’ve previously seen some discussion about <@abstraction@>(@Public Static: What is Abstraction?@), and some [claims](https://www.lesswrong.com/posts/wopE4nT28ausKGKwt/classification-of-ai-alignment-research-deconfusion-good?commentId=cKNrWxfxRgENS2EKX) that there are “natural” abstractions, or that AI systems will <@tend@>(@Chris Olah’s views on AGI safety@) to <@learn@>(@Conversation with Rohin Shah@) increasingly human-like abstractions (at least up to a point). To make this more crisp, given a system, let’s consider the information (abstraction) of the system that is relevant for predicting parts of the world that are “far away”. Then, the **natural abstraction hypothesis** states that:

1. This information is much lower-dimensional than the system itself.
2. These low-dimensional summaries are exactly the high-level abstract objects/concepts typically used by humans.
3. These abstractions are “natural”, that is, a wide variety of cognitive architectures will learn to use approximately the same concepts to reason about the world.

For example, to predict the effect of a gas in a larger system, you typically just need to know its temperature, pressure, and volume, rather than the exact positions and velocities of each molecule of the gas. The natural abstraction hypothesis predicts that many cognitive architectures would all converge to using these concepts to reason about gases.

If the natural abstraction hypothesis were true, it could make AI alignment dramatically simpler, as our AI systems would learn to use approximately the same concepts as us, which can help us both to “aim” our AI systems at the right goal, and to peer into our AI systems to figure out what exactly they are doing. So, this new project aims to test whether the natural abstraction hypothesis is true.

The first two claims will likely be tested empirically. We can build low-level simulations of interesting systems, and then compute what summary is useful for predicting its effects on “far away” things. We can then ask how low-dimensional that summary is (to test (1)), and whether it corresponds to human concepts (to test (2)).

A [followup post](https://www.alignmentforum.org/posts/f6oWbqxEwktfPrKJw/computing-natural-abstractions-linear-approximation) illustrates this in the case of a linear-Gaussian Bayesian network with randomly chosen graph structure. In this case, we take two regions of 110 nodes that are far apart from each other, and operationalize the relevant information between the two as the covariance matrix between the two regions. It turns out that this covariance matrix has about 3-10 “dimensions” (depending on exactly how you count), supporting claim (1). (And in fact, if you now compare to another neighborhood, two of the three “dimensions” remain the same!) Unfortunately, this doesn’t give much evidence about (2) since humans don’t have good concepts for parts of linear-Gaussian Bayesian networks with randomly chosen graph structure.

While (3) can also be tested empirically through simulation, we would hope that we can also prove theorems that state that nearly all cognitive architectures from some class of models would learn the same concepts in some appropriate types of environments.

To quote the author, “the holy grail of the project would be a system which provably learns all learnable abstractions in a fairly general class of environments, and represents those abstractions in a legible way. In other words: it would be a standardized tool for measuring abstractions. Stick it in some environment, and it finds the abstractions in that environment and presents a standard representation of them.”


I'm really like having a candidate algorithm in mind when reasoning about alignment. It is a lot more concrete, which makes it easier to make progress and not get lost, relative to generic reasoning from just the assumption that the AI system is superintelligent.

I'm less clear on how exactly you move between the two steps -- from my perspective, there is a core reason for worry, which is something like "you can't fully control what patterns of thought your algorithms learn, and how they'll behave in new circumstances", and it feels like you could always apply that as your step 2. Our algorithms are instead meant to chip away at the problem, by continually increasing our control over these patterns of thought. It seems like the author has a better-defined sense of what does and doesn't count as a valid step 2, and that makes this methodology more fruitful for him than it would be for me. More discussion [here](https://www.alignmentforum.org/posts/EF5M6CmKRd6qZk27Z/my-research-methodology?commentId=8Hq4GJtnPzpoALNtk).

My research methodology

This post outlines a simple methodology for making progress on AI alignment. The core idea is to alternate between two steps:

1. Come up with some alignment algorithm that solves the issues identified so far
2. Try to find some plausible situation in which either a) the resulting AI system is misaligned or b) the AI system is not competitive.

This is all done conceptually, so step 2 can involve fairly exotic scenarios that probably won't happen. Given such a scenario, we need to argue why no failure in the same class as that scenario will happen, or we need to go back to step 1 and come up with a new algorithm.
This methodology could play out as follows:

Step 1: RL with a handcoded reward function.
Step 2: This is vulnerable to <@specification gaming@>(@Specification gaming examples in AI@).
Step 1: RL from human preferences over behavior, or other forms of human feedback.
Step 2: The system might still pursue actions that are bad that humans can't recognize as bad. For example, it might write a well researched report on whether fetuses are moral patients, which intuitively seems good (assuming the research is good). However, this would be quite bad if the AI wrote the report because it calculated that it would increase partisanship leading to civil war.
Step 1: Use iterated amplification to construct a feedback signal that is "smarter" than the AI system it is training.
Step 2: The system might pick up on <@inaccessible information@>(@Inaccessible information@) that the amplified overseer cannot find. For example, it might be able to learn a language just by staring at a large pile of data in that language, and then seek power whenever working in that language, and the amplified overseer may not be able to detect this.
Step 1: Use <@imitative generalization@>(@Imitative Generalisation (AKA 'Learning the Prior')@) so that the human overseer can leverage facts that can be learned by induction / pattern matching, which neural nets are great at.
Step 2: Since imitative generalization ends up learning a description of facts for some dataset, it may learn low-level facts useful for prediction on the dataset, while not including the high-level facts that tell us how the low-level facts connect to things we care about.

The post also talks about various possible objections you might have, which I’m not going to summarize here.


On the alignment problem

This 80,000 Hours podcast goes over many of the examples from Brian’s book, <@The Alignment Problem@>. I recommend listening to it if you aren’t going to read the book itself; the examples and stories are fascinating. (Though note I only skimmed through the podcast.)


Note that the explanation of corrigibility given here is in accord with the usage in [this MIRI paper](https://intelligence.org/2014/10/18/new-report-corrigibility/), but not to the usage in the <@iterated amplification sequence@>(@Corrigibility@), where it refers to a broader concept. The broader concept might roughly be defined as “an AI is corrigible when it leaves its user ‘in control’”; see the linked post for examples of what ‘in control’ involves. (Here also you can have both an impact- and intent-based version of the definition.)

On the model that AI risk is caused by utility maximizers pursuing the wrong reward function, I agree that non-obstruction is a useful goal to aim for, and the resulting approaches (mild optimization, low impact, corrigibility as defined here) make sense to pursue. I <@do not like this model much@>(@Conclusion to the sequence on value learning@), but that’s (probably?) a minority view.

Non-Obstruction: A Simple Concept Motivating Corrigibility

The <@Reframing Impact sequence@>(@Reframing Impact - Part 1@) suggests that it is useful to think about how well we could pursue a _range_ of possible goals; this is called the _attainable utility (AU) landscape_. We might think of a superintelligent AI maximizing utility function U as causing this landscape to become “spiky” -- the value for U will go up, but the value for all other goals will go down. If we get this sort of spikiness for an incorrect U, then the true objective will have a very low value.

Thus, a natural objective for AI alignment research is to reduce spikiness. Specifically, we can aim for _non-obstruction_: turning the AI on does not decrease the attainable utility for _any_ goal in our range of possible goals. Mild optimization (such as [quantilization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) ([AN #48](https://mailchi.mp/3091c6e9405c/alignment-newsletter-48))) reduces spikiness by reducing the amount of optimization that an AI performs. Impact regularization aims to find an objective that, when maximized, does not lead to too much spikiness.

One particular strategy for non-obstruction would be to build an AI system that does not manipulate us, and allows us to correct it (i.e. modify its policy). Then, no matter what our goal is, if the AI system starts to do things we don’t like, we would be able to correct it. As a result, such an AI system would be highly non-obstructive. This property where we can correct the AI system is [corrigibility](https://intelligence.org/2014/10/18/new-report-corrigibility/). Thus, corrigibility can be thought of as a particular strategy for achieving non-obstruction.

It should be noted that all of the discussion so far is based on _actual outcomes in the world_, rather than what the agent was trying to do. That is, all of the concepts so far are based on _impact_ rather than _intent_.


If we cannot even measure AI progress and the impacts of AI systems right now, how are we supposed to accurately forecast them? As better forecasts are crucial for prioritizing the right problems and solutions, I am glad that the measurement of AI progress and impacts is getting broader attention. While the focus on developing countries might be less important from the perspective of AI existential safety, increased attention on measuring AI from a diverse set of communities is likely very useful for bringing the topic to the mainstream.

Measurement in AI Policy: Opportunities and Challenges

Saurabh Mishra, Jack Clark, C. Raymond Perrault

This paper is itself a summary of a 2019 Stanford workshop on the measurement of AI systems and contains summaries of all of the 33 talks given at the workshop. The workshop featured three in-depth breakout sessions, one on R&D and performance, one on economic impact and policy, and one on AI for sustainable development and human rights. Based on the discussions,  the authors identify six central problems in measuring AI progress and the impacts of AI systems:

First, the exact definition of AI is hard to get down given the ongoing evolution of the field. The lack of clear definitions makes it tricky to combine results on different aspects like investments into "AI" and the effects of "AI" on productivity both with each other and across different countries or sectors.

Second, measuring progress in AI is hard for a variety of reasons: We don't just care about narrow benchmark performance but about many factors like robustness, transferability and compute-efficiency, and it is not clear how the tradeoff between performance and these factors should look like. Apart from that, progress on popular benchmarks might be faster than overall progress as methods overfit to the benchmark, and the rise and fall of benchmark popularity make it hard to track progress over longer time intervals. Still, focusing on specific benchmarks and subdomains seems like an important first step.

Third, bibliometric data is an important tool for better understanding the role of different actors in a scientific field. More precise definitions of AI could help with getting better bibliometric data and such data could shine some light on aspects like the lifecycle of AI techniques and the demographics of AI researchers.

Fourth, we would like to measure the impact of AI on the economy, especially on inequality and the labour market. This requires a better understanding of the relationship between inputs like skilled workers and data, and outputs, which is difficult to obtain because many of the involved factors are quite intangible and effects on outputs can be quite delayed. Short-term indicators that are strong predictors of longer-term effects would be very useful in this context. Lastly, even figuring out which businesses are deploying AI can be hard, especially if the applications are inward-focused.

The fifth problem is concerned with the measurement of societal impacts of AI with a special focus on developing countries: While a large number of metrics for impacts of AI systems on human rights and the UN's sustainable development goals have been proposed, there is little data on the deployment of AI systems for social good and in developing countries, so far.

Sixth, there is a need for better assessment of risks posed by and other negative impacts of AI systems, both before and after deployment. To that extent, a better understanding of risks posed by general classes of applications like autonomous weapons, surveillance and fake videos would be helpful. One barrier here is that many of the riskier applications are in the domain of governmental action such that detailed information is often classified.


I broadly agree with the perspective in this post: in particular, I think we really should have more optimism because of the tendency of neural nets to learn “natural abstractions”. There is structure and regularity in the world and neural nets often capture it (despite being able to memorize random noise); if we train neural nets on a bunch of human-relevant data it really should learn a lot about humans, including what we care about.

However, I am less optimistic than the author about the specific path presented here (and he only assigns 10% chance to it). In particular, while I do think human values are a “real” thing that a neural net will pick up on, I don’t think that they are well-defined enough to align an AI system arbitrarily far into the future: our values do not say what to do in all possible situations; to see this we need only to look at the vast disagreements among moral philosophers (who often focus on esoteric situations). If an AI system were to internalize and optimize our current system of values, as the world changed the AI system would probably become less and less aligned with humans. We could instead talk about an AI system that has internalized both current human values and the process by which they are constructed, but that feels much less like a natural abstraction to me.

I _am_ optimistic about a very similar path, in which instead of training the system to pursue (a proxy for) human values, we train the system to pursue some “meta” specification like “be helpful to the user / humanity” or “do what we want on reflection”. It seems to me that “being helpful” is also a natural abstraction, and it seems more likely that an AI system pursuing this specification would continue to be beneficial as the world (and human values) changed drastically.

Alignment By Default

I liked the author’s summary, so I’ve reproduced it with minor stylistic changes:

A low-level model of some humans has everything there is to know about human values embedded within it, in exactly the same way that human values are embedded in physical humans. The embedding, however, is nontrivial. Thus, predictive power alone is not sufficient to define human values. The missing part is the embedding of values within the model.

However, this also applies if we replace the phrase “human values” with “trees”. Yet we have a whole class of neural networks in which a simple embedding lights up in response to trees. This is because trees are a natural abstraction, and we should expect to see real systems trained for predictive power use natural abstractions internally.

Human values are a little different from trees: they’re a property of an abstract object (humans) rather than an abstract object themselves. Nonetheless, the author still expects that a broad class of systems trained for predictive power will end up with simple embeddings of human values (~70% chance).

Since an unsupervised learner has a simple embedding of human values, a supervised/reinforcement learner can easily score well on values-proxy-tasks by directly using that model of human values. In other words, the system uses an actual model of human values as a proxy for our proxy of human values (~10-20% chance). This is what is meant by _alignment by default_.

When this works, it’s basically a best-case scenario, so we can safely use the system to design a successor without worrying about amplification of alignment errors (among other things).


I agree with what I see as the key point of this post: myopic training does not mean that the resulting agent will have myopic cognition. However, I don’t think this means myopic training is useless. According to me, the main benefit of myopic training is that small errors in reward specification for regular RL can incentivize catastrophic outcomes, while small errors in approval feedback for myopic RL are unlikely to incentivize catastrophic outcomes. (This is because “simple” rewards that we specify often lead to <@convergent instrumental subgoals@>(@The Basic AI Drives@), which need not be the case for approval feedback.) More details in [this comment](https://www.alignmentforum.org/posts/GqxuDtZvfgL2bEQ5v/arguments-against-myopic-training?commentId=TYS4QPTLnY38GeEhg).

Arguments against myopic training

[Several](https://www.alignmentforum.org/posts/7Hr8t6xwuuxBTqADK/approval-directed-agents-overview) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)) <@proposals@>(@An overview of 11 proposals for building safe advanced AI@) in AI alignment involve some form of myopic training, in which an AI system is trained to take actions that only maximize the feedback signal in the **next timestep** (rather than e.g. across an episode, or across all time, as with typical reward signals). In order for this to work, the feedback signal needs to take into account the future consequences of the AI system’s action, in order to incentivize good behavior, and so providing feedback becomes more challenging.

This post argues that there don’t seem to be any major benefits of myopic training, and so it is not worth the cost we pay in having to provide more challenging feedback. In particular, myopic training does not necessarily lead to “myopic cognition”, in which the agent doesn’t think about long-term consequences when choosing an action. To see this, consider the case where we know the ideal reward function R*. In that case, the best feedback to give for myopic training is the optimal Q-function Q*. However, regardless of whether we do regular training with R* or myopic training with Q*, the agent would do well if it estimates Q* in order to select the right action to take, which in turn will likely require reasoning about long-term consequences of its actions. So there doesn’t seem to be a strong reason to expect myopic training to lead to myopic cognition, if we give feedback that depends on (our predictions of) long-term consequences. In fact, for any approval feedback we may give, there is an equivalent reward feedback that would incentivize the same optimal policy.

Another argument for myopic training is that it prevents reward tampering and manipulation of the supervisor. The author doesn’t find this compelling. In the case of reward tampering, it seems that agents would not catastrophically tamper with their reward “by accident”, as tampering is difficult to do, and so they would only do so intentionally, in which case it is important for us to prevent those intentions from arising, for which we shouldn’t expect myopic training to help very much. In the case of manipulating the supervisor, he argues that in the case of myopic training, the supervisor will have to think about the future outputs of the agent in order to be competitive, which could lead to manipulation anyway.


It’s interesting how much of the alignment problem manifests itself even when you assume that the human utility function is known, but the feature sets used by the human and agent are different. The only piece of the argument missing from this paper is that with sufficiently capable agents, the agent will actually be _adversarial_ towards the human because of [convergent instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf), and that argument can be made in this framework.

Unfortunately, both of their methods for producing alignment don’t scale well, as they admit in the paper. (The second method in particular is kind of like hardcoding the policy, similarly to the construction <@here@>(@Coherence arguments do not imply goal-directed behavior@).)

User-Agent Value Alignment

This paper **from 2002** investigates what it would take to align an artificial agent with a human principal, under the assumption that the human utility function is known, but that the agent reward and human utility might be computed from different feature sets fA and fH. In this case, it is possible that the agent reward cannot capture all of the effects that the human cares about, leading to misalignment.

They introduce the concept of _graphical value alignment_, in which the only way that the agent’s actions can affect fH is through fA. In this case, we can establish _functional value alignment_ (in which the agent’s optimal policy also maximizes human utility), by setting the agent's reward for any specific fA to be the expectation (over fH) of the utility of fH, given fA. Note that the graphical criterion is _very strong_: it requires that _none_ of the agent’s unobserved effects matter at all to the human.

They suggest two methods for establishing alignment. First, we can define additional agent features (perhaps requiring additional sensors), until all of the effects on fH are captured by fA. However, this would be very difficult, if not impossible. Second, we can include all agent actions and observations as agent features, since any effect of the agent’s choice of policy on fH depends only on the observations made and actions taken. Of course, to achieve functional value alignment we would then have to have a good understanding of the expected human utility for every action given any observation, which is also hard.

They also briefly discuss the relationship between aligned agents and capable agents: a stone is aligned with you (per their definition), but also entirely useless. An interesting quote: _“Note that it might be harder to establish alignment with more competent agents because their skills afford many more pathways for adverse effects. This is a somewhat troubling thought.”_


One particularly interesting finding is that so far, in all of the cases they've looked at, there was a lot of feedback available to develop a solution. The post notes that this could be interpreted in two ways. First, since feedback is abundant in real-world problems, we should expect feedback for AI alignment as well ([the optimistic take](https://mailchi.mp/b3dc916ac7e2/an-80-why-ai-risk-might-be-solved-without-additional-intervention-from-longtermists)). Second, since AI alignment has no opportunity for feedback, it is unlike other problems (<@the pessimistic take@>(@The Rocket Alignment Problem@)). I would add a third option: that any real-world problem without feedback is extremely hard to solve, and so we wouldn't generate any hypotheses of actions that were prescient for such problems (in which case AI risk is not special amongst problems, it is just an instance of a very difficult problem).

Survey of prescient actions

AI Impacts is looking into other examples in history where people took actions in order to address a complex, novel, severe future problem, and in hindsight we recognize those actions as prescient. Ideally we could learn lessons for AI alignment from such cases. The survey is so far very preliminary, so I'll summarize it later when it has been further developed, but I thought I'd send it along if you wanted to follow along (I found the six cases they've identified quite interesting).


Bayesian Evolving-to-Extinction

Consider a Bayesian learner, that updates the weights of various hypotheses using Bayes Rule. If the hypotheses can influence future events and predictions (for example, maybe it can write out logs, which influence what questions are asked in the future), then hypotheses that affect the future in a way that only they can predict will be selected for by Bayes Rule, rather than hypotheses that straightforwardly predict the future without trying to influence it. In some sense, this is "myopic" behavior on the part of Bayesian updating: Bayes Rule only optimizes per-hypothesis, without taking into account the effect on overall future accuracy. This phenomenon could also apply to neural nets if the <@lottery ticket hypothesis@>(@The Lottery Ticket Hypothesis: Training Pruned Neural Networks@) holds: in this case each "ticket" can be thought of as a competing hypothesis.


I liked this sequence. Often when presented with a potential problem in AI safety, I ask myself why the problem doesn't also apply to humans, and how humans have managed to solve the problem. This sequence was primarily this sort of reasoning, and I think it did a good job of highlighting how with sufficient conservatism it seems plausible that many problems are not that bad if the AI is well-intentioned, even if it has very little information, or finds it hard to communicate with humans, or has the wrong abstractions.

If I were a well-intentioned AI

This sequence takes on the perspective of an AI system that is well-intentioned, but lacking information about what humans want. The hope is to find what good AI reasoning might look like, and hopefully use this to derive insights for safety. The sequence considers Goodhart problems, adversarial examples, distribution shift, subagent problems, etc.


In a [comment](https://www.alignmentforum.org/posts/ivpKSjM4D6FbqF4pZ/cortes-pizarro-and-afonso-as-precedents-for-takeover?commentId=kNFNjBJjuzTd3irrR), I dispute the claimed reasons for why Europeans conquered American civilizations. I think that a large body of historical literature supports the conclusion that American civilizations fell primarily because of their exposure to diseases which they lacked immunity to, rather than because of European military power. I also think that this helps explain why Portugal was "only" able to capture Indian Ocean trading ports during this time period, rather than whole civilizations. I think the primary insight here should instead be that pandemics can kill large groups of humans, and therefore it would be worth exploring the possibility that AI systems use pandemics as a mechanism to kill large numbers of biological humans.

Cortés, Pizarro, and Afonso as Precedents for Takeover

This post lists three historical examples of how small human groups conquered large parts of the world, and shows how they are arguably precedents for AI takeover scenarios. The first two historical examples are the conquests of American civilizations by Hernán Cortés and Francisco Pizarro in the early 16th century. The third example is the Portugese capture of key Indian Ocean trading ports, which happened at roughly the same time as the other conquests. Daniel argues that technological and strategic advantages were the likely causes of these European victories. However, since the European technological advantage was small in this period, we might expect that an AI coalition could similarly take over a large portion of the world, even without a large technological advantage.


How To Solve Moral Conundrums with Computability Theory


I really like this formalization of incentives, which come up frequently in AI safety work. It seems like some people are <@already@>(@Asymptotically Benign AGI@) <@using@>(@Designing agent incentives to avoid reward tampering@) this framework, and this seems low-cost enough that it's easy to imagine a world where this features in the safety analysis of algorithm designers.

Paper: The Incentives that Shape Behaviour

The Incentives that Shape Behaviour

Ryan Carey*, Eric Langlois*, Tom Everitt, Shane Legg

This post and [paper](https://arxiv.org/abs/2001.07118) introduce a method for analyzing the safety properties of a system using a  _causal theory of incentives_ (<@past@>(@Understanding Agent Incentives with Causal Influence Diagrams@) <@papers@>(@Modeling AGI Safety Frameworks with Causal Influence Diagrams@)).  An _incentive_ is something an agent must do to best achieve its goals. A _control incentive_ exists when an agent must control some component of its environment in order to maximize its utility, while a _response incentive_ is present when the agent's decision must be causally responsive to some component of its environment. These incentives can be analyzed formally by drawing a _causal influence diagram_, which represents a decision problem as a graph where each variable depends on the values of its parents.

For example, consider the case where a recommender algorithm decides what posts to show to maximize clicks. In the causal influnce diagram representing this system, we can include that we have control over the node 'posts to show', which has a direct effect on the node we want to maximize, 'clicks'. However, 'posts to show' may also have a direct effect on the node 'influenced user opinions', which itself affects 'clicks'. In the system as it stands, in addition to there being a desirable control incentive on 'clicks', there is also an undesirable control incentive on 'influenced user opinions', since they themselves influence 'clicks'. To get rid of the undesirable incentive, we could reward the system for _predicted clicks_ based on a model of the original user opinions, rather than for actual clicks.


The degree of overlap between utility functions seems to be pretty crucial (also see <@here@>(@When Goodharting is optimal: linear vs diminishing returns, unlikely vs likely, and other factors@)). It does seem plausible for the Bayesian approach to work well without the correct utility in the prior if  there was a lot of overlap between the utilities in the prior and the true utility. However, I am somewhat sceptical of our ability to get reliable estimates for that overlap.

Does Bayes Beat Goodhart?

It has been <@claimed@>(@Using expected utility for Good(hart)@) that  Goodhart's law might not be a problem for expected utility maximization, as long as we correctly account for our uncertainty about the correct utility function.

This post argues that Bayesian approaches are insufficient to get around Goodhart. One problem is that with insufficient overlap between possible utility functions, some utility functions might essentially be ignored when optimizing the expectation, even if our prior assigns positive probability to them. However, in reality, there is likely considerable overlap between the utility functions in our prior, as they are selected to fit our intuitions.

More severely, bad priors can lead to systematic biases in a bayesian's expectations, especially given embeddedness. As an extreme example, the prior might assign zero probability to the correct utility function. Calibrated instead of Bayesian learning can help with this, but only for <@regressional Goodhart@>(@Non-Adversarial Goodhart and AI Risks@). Adversarial Goodhart, where another agent tries to exploit the difference between your utility and your proxy seems to also require randomization like [quantilization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) ([AN #48](https://mailchi.mp/3091c6e9405c/alignment-newsletter-48)).


When is unaligned AI morally valuable?

When might it be a good idea to hand the keys to the universe to an unaligned AI? This post looks more deeply at this question, which could be important as a backup plan if we don't think we can build an aligned AI. I can't easily summarize this, so you'll have to read the post.


I'm unsure, though leaning skeptical, whether incorporating a copy of the reward function into a deep learning model would help it learn. My guess is that if someone did that with a current model it would make the model harder to train, rather than making anything easier. I will be excited if someone can demonstrate at least one feasible approach to addressing proxy alignment that does more than sample the reward function.

**Rohin's opinion:** I'm skeptical of this approach. Mostly this is because I'm generally skeptical that an intelligent agent will consist of a separate "planning" part and "reward" part. However, if that were true, then I'd think that this approach could plausibly give us some additional alignment, but can't solve the entire problem of inner alignment. Specifically, the reward function encodes a _huge_ amount of information: it specifies the optimal behavior in all possible situations you could be in. The "intelligent" part of the net is only ever going to get a subset of this information from the reward function, and so its plans can never be perfectly optimized for that reward function, but instead could be compatible with any reward function that would provide the same information on the "queries" that the intelligent part has produced.

For a slightly-more-concrete example, for any "normal" utility function U, there is a utility function U' that is "like U, but also the best outcomes are ones in which you hack the memory so that the 'reward' variable is set to infinity". To me, wireheading is possible because the "intelligent" part doesn't get enough information about U to distinguish U from U', and so its plans could very well be optimized for U' instead of U.

What are some non-purely-sampling ways to do deep RL?

A deep reinforcement learning agent trained by reward samples alone may predictably lead to a [proxy alignment issue](https://www.lesswrong.com/posts/pL56xPoniLvtMDQ4J/the-inner-alignment-problem): the learner could fail to develop a full understanding of what behavior it is being rewarded for, and thus behave unacceptably when it is taken off its training distribution. Since we often use explicit specifications to define our reward functions, Evan Hubinger asks how we can incorporate this information into our deep learning models so that they remain aligned off the training distribution. He names several possibilities for doing so, such as giving the deep learning model access to a differentiable copy of the reward function during training, and fine-tuning a language model so that it can map natural language descriptions of a reward function into optimal actions.


My usual starting point for future forecasting is to assume that AI won't alter any long term trends, and then update from there on the evidence. Most technologies haven't disrupted centuries-long trends in conflict resolution, which makes me hesitant to accept the first implication. Here, I think the biggest weakness in the argument is the assumption that powerful AIs should be described as having explicit utility functions. I still think that cooperation will be easier in the future, but it probably won't follow a radical departure from past trends.

Strategic implications of AIs' ability to coordinate at low cost, for example by merging

There are a number of differences between how humans cooperate and how hypothetical AI agents could cooperate, and these differences have important strategic implications for AI forecasting and safety. The first big implication is that AIs with explicit utility functions will be able to merge their values. This merging may have the effect of rendering laws and norms obsolete, since large conflicts would no longer occur. The second big implication is that our approaches to AI safety should preserve the ability for AIs to cooperate. This is because if AIs *don't* have the ability to cooperate, they might not be as effective, as they will be outcompeted by factions who can cooperate better.


Certainly the ability to think through hypothetical scenarios helps a lot, as recently demonstrated by <@MuZero@>(@Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model@), and that alone is sufficient reason to expect advanced agents to use logic, or something like it. Another such intuition for me is that logic enables much better generalization, e.g. our grade-school algorithm for adding numbers is way better than algorithms learned by neural nets for adding numbers (which often fail to generalize to very long numbers).

Of course, the "logic" that advanced agents use could be learned rather than pre-specified, just as we humans use learned logic to reason about the world.

Do Sufficiently Advanced Agents Use Logic?

Current progress in ML suggests that it's quite important for agents to learn how to predict what's going to happen, even though ultimately we primarily care about the final performance. Similarly, it seems likely that the ability to use logic will be an important component of intelligence, even though it doesn't obviously directly contribute to final performance.

The main source of intuition is that in environments where data is scarce, agents should still be able to learn from the results of (logical) computations. For example, while it may take some data to learn the rules of chess, once you have learned them, it should take nothing but more thinking time to figure out how to play chess well. In game theory, the ability to think about similar games and learning from what "would" happen in those games seems quite powerful. When modeling both agents in a game this way, <@a single-shot game effectively becomes an iterated game@>(@In Logical Time, All Games are Iterated Games@).


I believe that it is quite important to be aware of the implications that different forms of takeoff should have on our prioritization and am glad that the article highlights this. However, I am a bit worried that this very broad definition of continuous progress limits the usefulness of the concept. For example, it seems plausible that a recursively self-improving agent which is very hard to deal with once deployed still improves its capabilities slow enough to fit the definition, especially if its developer has a significant lead over others.

Misconceptions about continuous takeoff

This post attempts to clarify the author's notion of continuous AI takeoff, defined as the growth of future AI capabilities being in line with extrapolation from current trends. In particular, that means that no AI project is going to bring sudden large gains in capabilities compared to its predecessors.

Such a continuous takeoff does not necessarily have to be slow. For example, generative adversarial networks have become better quite rapidly during the last five years, but progress has still been piecemeal. Furthermore, exponential gains, for example due to recursive self-improvement, can be consistent with a continuous takeoff, as long as the gains from one iteration of the improvement process are modest. However, this means that a continuous takeoff does not preclude large power differentials from arising: slight advantages can compound over time and actors might use their lead in AI development to their strategic advantage even absent discontinuous progress, much like western Europe used its technological advantage to conquer most of the world.

Knowing whether or not AI takeoff happens continuously is important for alignment research: A continuous takeoff would allow for more of an attitude of "dealing with things as they come up" and we should shift our focus on specific aspects that are hard to deal with as they come up. If the takeoff is not continuous, an agent might rapidly gain capabilities relative to the rest of civilization and it becomes important to rule out problems, long before they come up.


Vox interview with Stuart Russell

Kelsey talked with Stuart Russell about his new book, <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@).


I certainly agree that we want to avoid reward functions defined on observations, and this is one reason why. It seems like a more general version of the wireheading argument to me, and applies even if you think that the AI won't be able to wirehead, as long as it is capable enough to find other plans for getting high reward besides the one the designer intended.

What You See Isn't Always What You Want

This post makes the point that for Markovian reward functions on _observations_, since any given observation can correspond to multiple underlying states, we cannot know just by analyzing the reward function whether it actually leads to good behavior: it also depends on the environment. For example, suppose we want an agent to collect all of the blue blocks in a room together. We might simply reward it for having blue in its observations: this might work great if the agent only has the ability to pick up and move blocks, but won't work well if the agent has a paintbrush and blue paint. This makes the reward designer's job much more difficult. However, the designer could use techniques that don't require a reward on individual observations, such as rewards that can depend on the agent's internal cognition (as in iterated amplification), or rewards that can depend on histories (as in [Deep RL from Human Preferences](https://deepmind.com/blog/learning-through-human-feedback/)).


It does seem to me that if we're in a situation where we have solved the alignment problem, we control 99% of resources, and we aren't infighting amongst each other, we will likely continue to control at least 99% of the resources in the future. I'm a little confused about how we get to this situation though -- the scenarios I usually worry about are the ones in which we fail to solve the alignment problem, but still deploy unaligned AIs, and in these scenarios I'd expect unaligned AIs to get the majority of the resources. I suppose in a multipolar setting with continuous takeoff, if we have mostly solved the alignment problem but still accidentally create unaligned AIs (or some malicious actors create them deliberately), then this setting where we control 99% of the resources could arise.

The strategy-stealing assumption

We often talk about aligning AIs in a way that is _competitive_ with unaligned AIs. However, you might think that we need them to be _better_: after all, unaligned AIs only have to pursue one particular goal, whereas aligned AIs have to deal with the fact that we don't yet know what we want. We might hope that regardless of what goal the unaligned AI has, any strategy it uses to achieve that goal can be turned into a strategy for acquiring _flexible_ influence (i.e. influence useful for many goals). In that case, **as long as we control a majority of resources**, we can use any strategies that the unaligned AIs can use. For example, if we control 99% of the resources and unaligned AI controls 1%, then at the very least we can split up into 99 "coalitions" that each control 1% of resources and use the same strategy as the unaligned AI to acquire flexible influence, and this should lead to us obtaining 99% of the resources in expectation. In practice, we could do even better, e.g. by coordinating to shut down any unaligned AI systems.

The premise that we can use the same strategy as the unaligned AI, despite the fact that we need _flexible_ influence, is called the **strategy-stealing assumption**. Solving the alignment problem is critical to strategy-stealing -- otherwise, unaligned AI would have an advantage at thinking that we could not steal and the strategy-stealing assumption would break down. This post discusses **ten other ways that the strategy-stealing assumption could fail**. For example, the unaligned AI could pursue a strategy that involves threatening to kill humans, and we might not be able to use a similar strategy in response because the unaligned AI might not be as fragile as we are.


I like these diagrams as a way to demonstrate the basics of what's going on in various AI safety frameworks. Sometimes the diagrams can also show the differences in safety features of frameworks. For example, in reward modeling, the agent has an incentive to affect the human feedback in order to affect the reward model directly. (Imagine getting the human hooked on heroin, so that future feedback causes the reward model to reward heroin, which could be easy to produce.) On the other hand, in cooperative IRL, the agent only wants to affect the human actions inasmuch as the actions affect the state, which is a normal or allowed incentive. (Imagine the agent causing the human to leave their house earlier so that they get to their meeting on time.)

Modeling AGI Safety Frameworks with Causal Influence Diagrams

Tom Everitt*, Ramana Kumar*, Victoria Krakovna*, Shane Legg

This paper describes several AI safety frameworks using the language of <@causal influence diagrams@>(@Understanding Agent Incentives with Causal Influence Diagrams@), in order to make it easy to compare and contrast them. For example, the diagrams make it clear that while [Cooperative IRL](https://arxiv.org/abs/1606.03137) and <@reward modeling@>(@Scalable agent alignment via reward modeling@) are very similar, there are significant differences: in cooperative IRL, the rewards come directly from the underlying human preferences, whereas in reward modeling, the rewards come from a reward model that depends on human feedback, which itself depends on the underlying human preferences.


I still don't know what to think about daemons -- they do seem to be a problem in Solomonoff induction, but they seem unlikely to arise in the kinds of neural nets we have today (but could arise in larger ones). I would love to see more clarity around daemons, especially since the vast majority of current research would not solve this problem, since it is a problem with the training _process_ and not the training _signal_.

Open question: are minimal circuits daemon-free?

One issue that may arise with an advanced AI agent is that during training we may end up with a part of the AI system developing into a "daemon" -- a consequentialist agent that is optimizing a different goal. This goal may useful as a subcomponent for our AI, but the daemon may grow in power and end up causing the system to optimize for the subgoal. This could lead to catastrophic outcomes, _even if_ we have specified a reward function that encodes human values to the top-level AI.

In this post, Paul suggests that these issues would likely go away if we choose the _fastest_ program to solve our subgoal. Intuitively, for any daemon that arises as a solution to our problem, for it to cause a bad outcome it must be carrying out complicated reasoning to figure out whether or not to solve the problem honestly or to try to mislead us, and so we could get a faster program by just not doing that part of the computation. He proposes a particular formalization and poses it as an open question -- if we always choose the minimal (in size) boolean circuit that solves our problem, can a daemon ever arise?


I reject modeling agents as having preferences over states primarily for the first reason that Richard identified: there are many "reasonable" preferences that cannot be modeled with a reward function solely on states. However, I don't find the argument about beliefs as a free variable very convincing: I think it's reasonable to argue that a superintelligent AI system will on average have much better beliefs than us, and so anything that we could determine as a coherence violation with high confidence should be something the AI system can also determine as a coherence violation with high confidence.

Coherent behaviour in the real world is an incoherent concept

In a <@previous post@>(@Coherence arguments do not imply goal-directed behavior@), I argued that coherence arguments (such as those based on VNM rationality) do not constrain the behavior of an intelligent agent. In this post, Richard delves further into the argument, and considers other ways that we could draw implications from coherence arguments.

I modeled the agent as having preferences over full trajectories, and objected that if you only look at _observed_ behavior (rather than _hypothetical_ behavior), you can always construct a utility function such that the observed behavior optimizes that utility function. Richard agrees that this objection is strong, but looks at another case: when the agent has preferences over states at a single point in time. This case leads to other objections. First, many reasonable preferences cannot be modeled via a reward function over states, such as the preference to sing a great song perfectly. Second, in the real world you are never in the same state more than once, since at the very least your memories will change, and so you can never infer a coherence violation by looking at observed behavior.

He also identifies further problems with applying coherence arguments to realistic agents. First, all behavior is optimal for the constant zero reward function. Second, any real agent will not have full information about the world, and will have to have beliefs over the world. Any definition of coherence will have to allow for multiple beliefs -- but if you allow all beliefs, then you can rationalize any behavior as based on some weird belief that the agent has. If you require the agent to be Bayesian, you can still rationalize any behavior by choosing a prior appropriately.


This seems very similar to the problem I have with impact measures, but I wouldn't apply that argument to corrigibility. I think the difference might be that I'm thinking of "natural" things that agents might want, whereas Stuart is considering the entire space of possible utility functions. I'm not sure what drives this difference.

Partial preferences needed; partial preferences sufficient

I'm not sure I fully understand this post, but my understanding is that it is saying that alignment proposals must rely on some information about human preferences. Proposals like impact measures and corrigibility try to formalize a property that will lead to good outcomes; but any such formalization will be denoting some policies as safe and some as dangerous, and there will always exist a utility function according to which the "safe" policies are catastrophic. Thus, you need to also define a utility function (or a class of them?) that safety is computed with respect to; and designing this is particularly difficult.


I'm glad to see this sort of discussion taking place - however, I disagree quite strongly with arguments 1 and 3. On 1: it's true that for current systems, it's often better to describe them without assigning them agency, but only because they're still very simple compared with humans (or smart animals). Whether or not it will be appropriate to consider advanced AI to have intentions and goals is a complex question - I think there are strong arguments for that claim. On 3: I think that it's very reasonable to shy away from very small probabilities without overwhelming amounts of evidence, to counteract standard human overconfidence. Beth's alternative of reasoning using bits of evidence seems like it would push almost everyone towards unjustifiably strong conclusions on most questions, as it does for her on AI risk.

Three Biases That Made Me Believe in AI Risk

Beth (not to be confused with AI safety researcher Beth Barnes) argues firstly that the language we use overly anthropomorphises AI, which leads to an exaggerated perception of risks; secondly, that the sense of meaning that working on AI safety provides causes motivated reasoning; and thirdly, that we anchor away from very low numbers (e.g. it seems absurd to assign existential AI risk a probability of 0.0000000000000000000000000000001, since that has so many zeros! Yet Beth thinks this number significantly overestimates the risk.)


This map is so large that I can't actually use it to get a birds-eye view of the entire space, but it seems quite useful for looking at a local region and as a starting point to explore one particular aspect more deeply.

Value Alignment Map

This is a gigantic graph of many of the concepts in the AI risk space. Each concept has a description and links to existing literature, and by clicking around in the map I found several interesting links I hadn't seen before.


While I'm not sure I agree with the reasons in this paper, I do think we need creativity and "hot" searches in technical AI safety, simply based on the level of confusion and uncertainty that we (or at least I) have currently. The properties in Table 1 seem particularly good as an initial list of things to target if we want to make creative research more likely.

Existential Risk, Creativity & Well-Adapted Science

From a brief skim, it seems like this paper defines "creativity" in scientific research, and argues that existential risk research needs to be creative. Research is creative if it is composed of "hot" searches, where we jump large distances from one proposed solution to another, with broad differences between these solutions, as opposed to "cold" searches, in which we primarily make incremental improvements, looking over a small set of solutions clustered in the neighborhood of existing solutions. The paper argues that research on existential risk needs to be creative, because many aspects of such research make it hard to analyze in a traditional way -- we can't perform controlled experiments of extinction, nor of the extreme circumstances under which it is likely; there are many interdependent parts that affect each other (since existential risks typically involve effects on many aspects of society), and there is likely to be a huge amount of uncertainty due to lack of evidence. As a result, we want to change the norms around existential risk research from the standard academic norms, which generally incentivize conservatism and "cold" searches. Table 1 provides a list of properties of academia that lead to conservatism, and asks that future work think about how we could mitigate these.


Do what we mean vs. do what we say

I wrote a post proposing that we define a "do what we mean" system to be one in which the thing being optimized is latent (in the sense that it is not explicitly specified, not that it has a probability distribution over it). Conversely, a "do what we say" system explicitly optimizes something provided as an input. A lot of AI safety arguments can be understood as saying that a pure "do what we say" AI will lead to catastrophic outcomes. However, this doesn't mean that a "do what we mean" system is the way to go -- it could be that we want a "do what we mean" core, along with a "do what we say" subsystem that makes sure that the AI always listens to eg. shutdown commands.


VOI is Only Nonnegative When Information is Uncorrelated With Future Action

Normally, the value of getting more information (VOI) is always nonnegative (for a rational agent), because you can always take the same action you would have if you didn't have the information, so your decision will only improve. However, if the information would cause you to have a different set of actions available, as in many decision theory examples, then this proof no longer applies, since you may no longer be able to take the action you would have otherwise taken. As a result, information can have negative value.


When I talk about alignment, I am considering the second species problem, with AI systems whose capability level is roughly human-level or more (including “wildly superintelligent”).

I agree with [this comment thread](https://www.alignmentforum.org/posts/HHunb8FPnhWaDAQci/the-alignment-problem-in-different-capability-regimes?commentId=mz6hYiWqNMTwHNrhC) that the core _problem_ in what-I-call-alignment stays conserved across capability levels, but the solutions can change across capability levels. (Also, other people mean different things by “alignment”, such that this would no longer be true.)

The alignment problem in different capability regimes

One reason that researchers might disagree on what approaches to take for alignment is that they might be solving different versions of the alignment problem. This post identifies two axes on which the “type” of alignment problem can differ. First, you may consider AI systems with differing levels of capability, ranging from subhuman to wildly superintelligent, with human-level somewhere in the middle. Second, you might be thinking about different mechanisms by which this leads to bad outcomes, where possible mechanisms include <@the second species problem@>(@AGI safety from first principles@) (where AIs seize control of the future from us), the “missed opportunity” problem (where we fail to use AIs as well as we could have, but the AIs aren’t themselves threatening us), and a grab bag of other possibilities (such as misuse of AI systems by bad actors).

Depending on where you land on these axes, you will get to rely on different assumptions that change what solutions you would be willing to consider:

1. **Competence.** If you assume that the AI system is human-level or superintelligent, you probably don’t have to worry about the AI system causing massive problems through incompetence (at least, not to a greater extent than humans do).
2. **Ability to understand itself.** With wildly superintelligent systems, it seems reasonable to expect them to be able to introspect and answer questions about their own cognition, which could be a useful ingredient in a solution that wouldn’t work in other regimes.
3. **Inscrutable plans or concepts.** With sufficiently competent systems, you might be worried about the AI system making dangerous plans you can’t understand, or reasoning with concepts you will never comprehend. Your alignment solution must be robust to this.


The theory-practice gap

We can think of alignment as roughly being decomposed into two “gaps” that we are trying to reduce:

1. The gap between proposed theoretical alignment approaches (such as iterated amplification) and what we might do without such techniques (aka the <@unaligned benchmark@>(@An unaligned benchmark@))
2. The gap between actual implementations of alignment approaches and what those approaches are theoretically capable of.

(This distinction is fuzzy. For example, the author puts “the technique can’t answer NP-hard questions” into the second gap while I would have had it in the first gap.)

We can think of some disagreements in AI alignment as different pictures about how these gaps look:

1. A stereotypical “ML-flavored alignment researcher” thinks that the first gap is very small, because in practice the model will generalize appropriately to new, more complex situations, and continue to do what we want. Such people would then be more focused on narrowing the second gap by working on practical implementations.
2. A stereotypical “MIRI-flavored alignment researcher” thinks that the first gap is huge, such that it doesn’t really matter if you narrow the second gap, because even if you reduced that gap to zero you would still be doomed with near certainty.


It's tempting to write off such experiments as obvious since it's clear that class design affects classification results. However, upon further consideration, specifying how to _address_ such problems seems significantly more challenging. I think this paper does a good job of pointing out the relative nuance in how class design and bias interact in fairly realistic use cases.

Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications

Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, Miles Brundage

There has been significant progress in zero-shot image classification with models such as [CLIP](https://arxiv.org/abs/2103.00020) and [ALIGN](https://arxiv.org/abs/2102.05918). These models work by effectively learning visual concepts from natural language supervision. Such models make it possible to build classifiers without task-specific data, which is useful in scenarios where data is either costly or unavailable. However, this capability introduces the potential for bias. This paper is an exploratory bias probe of the CLIP model that finds class design heavily influences model performance.

The first set of experiments focusses on classification terms that have a high potential to cause representational harm. In one example, the authors conduct experiments on the FairFace dataset by adding classification labels such as 'animal' and 'criminal' to the list of possible classes. They find that black people and young people (under 20) were misclassified at significantly higher rates (14%) compared to the dataset as a whole (5%). This shows that the choice of labels affects classification outcomes. In a follow-up experiment, the authors add the additional label 'child' and find that this drastically reduces classification into crime-related and non-human categories. This shows sensitivity to minor changes in class design.

In the second set of experiments, the authors focus on how CLIP treated images of men and women using images of Members of Congress. Although CLIP wasn't designed for multi-label classification, it's still informative to look at the label distribution above a certain cutoff. When occupations are used as the label set, the authors find that thresholds under 0.5% return 'nanny' and 'housekeeper' for women and 'prisoner' and 'mobster' for men. When labels come from the combined set that Google Cloud Vision, Amazon Rekognition and Microsoft use for all images, the authors find that CLIP returns a disproportionate number of appearance-related labels to women.


Human modeling in AGI

This is a conversation between Scott and me about the <@relative dangers of human modeling@>(@Thoughts on Human Models@), moderated by Eli Tyre. From a safety perspective, the main reason to _avoid_ human modeling is that the agent's cognition will be much "further" away from manipulation of humans; for example, it seems more unlikely that your AI system tricks people into launching nukes if it never learned very much about humans in the first place. The main counterargument is that this precludes using human oversight of agent cognition (since when humans are overseeing the agent's cognition, then the agent is likely to learn about humans in order to satisfy that oversight); this human oversight could plausibly greatly increase safety. It also seems like systems that don't model humans will have a hard time performing many useful tasks, though the conversation mostly did not touch upon this point.

Scott's position is that given there are these two quite different risks (manipulation worries vs. learning the wrong cognition due to poor oversight), it seems worthwhile to put some effort into addressing each risk, and avoiding human models is much more neglected than improving human oversight. My position is that it seems much less likely that there is a plausible success path where we do very little human modeling, and so I want a lot more work along the oversight path. I _do_ think that it is worth differentially pushing AI systems towards tasks that don't require much human modeling, e.g. physics and engineering, rather than ones that do, e.g. sales and marketing, but this seems roughly independent of technical work, at least currently.


I'm not sure if I should update without knowing the cause of the discontinuity, or how the search for discontinuities was conducted. If you're searching for discontinuities, I do expect you'll find some, even if in general I expect discontinuities not to arise, so it doesn't feel like strong evidence that discontinuities are probable.

[Discontinuous progress investigation](https://aiimpacts.org/discontinuous-progress-investigation/) or [Likelihood of discontinuous progress around the development of AGI](https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/)

Discontinuity from the Eiffel Tower

The Eiffel tower represented a 54-year discontinuity in the trend for "height of the tallest existing structure", and an 8000-year discontinuity in the trend for "height of the tallest structure ever". It's unclear what the cause of this discontinuity is, though the authors provide some speculation.


I definitely support having models that engage more with the messiness of the real world. I'm not sure if I would have used "wide models" -- it seems like even the assumption of a crisp core makes it not as capable of handling messiness as I want. But if you're trying to get formal guarantees and you need to use some model, a wide model seems probably useful to use.

Compact vs. Wide Models

A compact model is one which is very general, and easy to prove things about, but doesn't inherently capture the messiness of the real world inside the model. Examples include Turing machines and utility functions. A wide model is one which still has a conceptually crisp core, but these crisp core units must then be combined in a complicated way in order to get something useful. Examples include the use of transistors to build CPUs, and the hierarchical control model of human psychology. The nice thing about wide models is that they start to engage with the messiness of the real world, and so make it clearer where the complexity is being dealt with. This is a useful concept to have when evaluating a proposal for alignment -- it asks the question, "where does the complexity reside?"


Progress on Causal Influence Diagrams

Tom Everitt, Ryan Carey, Lewis Hammond, James Fox, Eric Langlois, Shane Legg

Many of the problems we care about (reward gaming, wireheading, manipulation) are fundamentally a worry that our AI systems will have the _wrong incentives_. Thus, we need Causal Influence Diagrams (CIDs): a formal theory of incentives. These are <@graphical models@>(@Understanding Agent Incentives with Causal Influence Diagrams@) in which there are action nodes (which the agent controls) and utility nodes (which determine what the agent wants). Once such a model is specified, we can talk about various incentives the agent has. This can then be used for several applications:

1. We can analyze [what happens](https://arxiv.org/abs/2102.07716) when you [intervene](https://arxiv.org/abs/1707.05173) on the agent’s action. Depending on whether the RL algorithm uses the original or modified action in its update rule, we may or may not see the algorithm disable its off switch.
2. We can <@avoid reward tampering@>(@Designing agent incentives to avoid reward tampering@) by removing the connections from future rewards to utility nodes; in other words, we ensure that the agent evaluates hypothetical future outcomes according to its _current_ reward function.
3. A [multiagent version](https://arxiv.org/abs/2102.05008) allows us to recover concepts like Nash equilibria and subgames from game theory, using a very simple, compact representation.


I generally agree with these positions and responses, and in particular I’m especially happy about the arguments being specific to the actual models we use today, which grounds out the discussion a lot more and makes it easier to make progress. On the second point in particular, I’d also [say](https://www.alignmentforum.org/posts/6ccG9i5cTncebmhsH/frequent-arguments-about-alignment?commentId=cwgpCBfwHaYravLpo) that empirically, product-focused people don’t do e.g. RL for human feedback, even if it could be motivated that way.

Frequent arguments about alignment

This post outlines three AI alignment skeptic positions and corresponding responses from an advocate. Note that while the author tends to agree with the advocate’s view, they also believe that the skeptic makes good points.

1. _Skeptic's position_: The alignment problem gets easier as models get smarter, since they start to learn the difference between, say, human smiles and human well-being. So all we need to do is to prompt them appropriately, e.g. by setting up a conversation with “a wise and benevolent AI advisor”.

_Advocate's response:_ We can do a lot better than prompting: in fact, <@a recent paper@>(@True Few-Shot Learning with Language Models@) showed that prompting is effectively (poor) finetuning, so we might as well finetune. Separately from prompting itself, alignment does get easier in some ways as models get smarter, but it also gets harder: for example, smarter models will game their reward functions in more unexpected and clever ways.

2. What’s the difference between alignment and capabilities anyway? Something like <@RL from human feedback for summarization@>(@Learning to Summarize with Human Feedback@) could equally well have been motivated through a focus on AI products.

_Response:_ While there’s certainly overlap, alignment research is usually not the lowest-hanging fruit for building products. So it’s useful to have alignment-focused teams that can champion the work even when it doesn’t provide the best near-term ROI.

3. We can’t make useful progress on aligning superhuman models until we actually have superhuman models to study. Why not wait until those are available?

_Response:_ If we don’t start now, then in the short term, companies will deploy products that optimize simple objectives like revenue and engagement, which could be improved by alignment work. In the long term, it is plausible that alignment is very hard, such that we need many conceptual advances that we need to start on now to have them ready by the point that we feel obligated to use powerful AI systems. In addition, empirically there seem to be many alignment approaches that aren’t bottlenecked by the capabilities of models -- see for example <@this post@>(@The case for aligning narrowly superhuman models@).


Survey on AI existential risk scenarios

Sam Clarke, Alexis Carlier, Jonas Schuett

While the previous survey asked respondents about the overall probability of existential catastrophe, this survey seeks to find which particular risk scenarios respondents find more likely. The survey was sent to 135 researchers, of which 75 responded. The survey presented five scenarios along with an “other”, and asked people to allocate probabilities across them (effectively, conditioning on an AI-caused existential catastrophe, and then asking which scenario happened).

The headline result is that all of the scenarios were roughly equally likely, even though individual researchers were opinionated (i.e. they didn’t just give uniform probabilities over all scenarios). Thus, there is quite a lot of disagreement over which risk scenarios are most likely (which is yet another reason not to take the results of the previous survey too seriously).


Pitfalls of the agent model

It is common to view AI systems through the “agent lens”, in which the AI system implements a fixed, unchanging policy that, given some observations, takes some actions. This post points out several ways in which this “fixed, unchanging policy” assumption can lead us astray.

For example, AI designers may assume that the AI systems they build must have unchanging decision algorithms, and therefore believe that there will be a specific point at which influence is “handed off” to the AI system, before which we have to solve a wide array of philosophical and technical problems.


Mundane solutions to exotic problems

The author’s goal is to find “mundane” or simple algorithms that solve even “exotic” problems in AI alignment. Why should we expect this is possible? If an AI system is using powerful, exotic capabilities to evade detection, shouldn’t we need powerful, exotic algorithms to fight that? The key idea here is that we can instead have a mundane algorithm that leverages the exotic capabilities of the AI system to produce an exotic oversight process. For example, we could imagine that a mundane algorithm could be used to create a question-answerer that knows everything the model knows. We could then address <@gradient hacking@>(@Gradient hacking@) by asking the question “what should the loss be?” In this case, our model has an exotic capability: very strong introspective access to its own reasoning and the training process that modifies it. (This is what is needed to successfully hack gradients). As a result, our question answerer should be able to leverage this capability to assign high loss (low reward) to cases where our AI system tries to hack gradients, even if our normal hardcoded loss would not do so.


While I disagree on some of the details, I think this is a good threat model to be thinking about. Its main virtue is that it has a relatively concrete model for what AGI looks like, and it provides a plausible story for both how that type of AGI could be developed (the development model) and how that type of AGI would lead to problems (the risk model). Of course, it is still worth clarifying the plausibility of the scenario, as updates to the story can have significant implications on what research we do. (Some of this discussion is happening in [this post](https://www.alignmentforum.org/posts/pz7Mxyr7Ac43tWMaC/against-evolution-as-an-analogy-for-how-humans-will-create).)

My AGI Threat Model: Misaligned Model-Based RL Agent

This post lays out a pathway by which an AI-induced existential catastrophe could occur. The author suggests that AGI will be built via model-based reinforcement learning: given a reward function, we will learn a world model, a value function, and a planner / actor. These will learn online, that is, even after being deployed these learned models will continue to be updated by our learning algorithm (gradient descent, or whatever replaces it). Most research effort will be focused on learning these models, with relatively less effort applied to choosing the right reward function.

There are then two alignment problems: the _outer_ alignment problem is whether the reward function correctly reflects the designer's intent, and the _inner_ alignment problem is whether the value function accurately represents the expected reward obtained by the agent over the long term. On the inner alignment side, the value function may not accurately capture the reward for several reasons, including ambiguity in the reward signals (since you only train the value function in some situations, and many reward functions can then produce the same value function), manipulation of the reward signal, failures of credit assignment, ontological crises, and having mutually contradictory "parts" of the value function (similarly to humans). On the outer alignment side, we have the standard problem that the reward function may not reflect what we actually want (i.e. specification gaming or Goodhart's Law). In addition, it seems likely that many capability enhancements will be implemented through the reward function, e.g. giving the agent a curiosity reward, which increases outer misalignment.


Intermittent Distillations #1

A post in the same style as this newsletter.


Distinguishing claims about training vs deployment

One story for AGI is that we train an AI system on some objective function, such as an objective that rewards the agent for following commands given to it by humans using natural language. We then deploy the system without any function that produces reward values; we instead give the trained agent commands in natural language. Many key claims in AI alignment benefit from more precisely stating whether they apply during training or during deployment.

For example, consider the instrumental convergence argument. The author proposes that we instead think of the training convergence thesis: a wide range of environments in which we could train an AGI will lead to the development of goal-directed behavior aimed towards certain convergent goals (such as self-preservation). This could happen either via the AGI internalizing them directly as final goals, or by the AGI learning final goals for which these goals are instrumental.

The author similarly clarifies goal specification, the orthogonality thesis, fragility of value, and Goodhart’s Law.


Human Factors and Ergonomics in Manufacturing

I’m not very familiar with this field, but among other techniques the paper mentions STAMP and STPA which we’ve previously seen in <@Engineering a Safer World@>. It does seem to me like these techniques would be useful to apply to the entire sociotechnical system, of which an AGI system is just one part (and this is what the paper’s examples do). It is less clear to me whether it makes sense to take techniques designed for humans and apply them to AGI: perhaps we’ll have enough understanding of the differences between humans and AGI that we could do this in a reasonable way, but I think there is a real risk that the methods give incorrect conclusions simply because they make incorrect assumptions about how AGI works (given that they were designed for humans). Nonetheless, I do agree with the core claim of this paper that HFE is worth exploring.

Putting the humanity into inhuman systems: How human factors and ergonomics can be used to manage the risks associated with artificial general intelligence

Paul M. Salmon, Tony Carden, Peter A. Hancock

This paper argues that the methods of Human Factors and Ergonomics (HFE) should be applied to AGI safety. They list fifteen different methods from the field, typically used to analyze the performance of humans in systems, which could be applied to AGI instead (on the assumption that AGI will be more like humans than like machines in today’s systems). They then give examples of how these might be applied to the Prometheus story in the prologue of [Life 3.0](https://www.amazon.com/Life-3-0-Being-Artificial-Intelligence/dp/1101946598).


Mapping the Conceptual Territory in AI Existential Safety and Alignment

There are a bunch of high-level overviews and research agendas, not all of which agree with each other. This post attempts to connect and integrate several of these, drawing heavily on <@Paul Christiano’s overview@>(@Current Work in AI Alignment@), [my](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) [overview](https://futureoflife.org/2019/04/25/an-overview-of-technical-ai-alignment-with-rohin-shah-part-2/), and the <@ARCHES agenda@>(@AI Research Considerations for Human Existential Safety@), but also including a lot of other work. It serves as a good way of connecting these various perspectives; I recommend reading it for this reason. (Unfortunately, it is rather hard to summarize, so I haven’t done so.)


The sharpest increases in the graphs seem to be in interpretability and explainable AI around 2017-18, as well as in value alignment starting in 2017. My guess is that the former is the result of [DARPA’s interest in the area](https://www.darpa.mil/attachments/DARPA-BAA-16-53.pdf) (which I believe started in 2016), and the latter is probably a combination of the founding of the Center for Human-Compatible AI (CHAI) and the publication and promotion of <@CIRL@>(@Cooperative Inverse Reinforcement Learning@) (one of CHAI’s early papers).

Surprisingly to me, we don’t see trend deviations in papers on “reward hacking”, “safe exploration”, or “distributional shift” after the publication of [Concrete Problems in AI Safety](https://arxiv.org/abs/1606.06565), even though it has been cited way more often than CIRL, and seemed like it had far more of an effect on mainstream AI researchers. (Note that “safe exploration” did increase, but it seems in line with the existing trend.)

Note that I expect the data source is not that reliable, and so I am not confident in any of these conclusions.

AI safety: state of the field through quantitative lens

Mislav Juric, Agneza Sandic, Mario Brcic

This paper presents data demonstrating growth in various subfields related to AI safety. The data was collected through queries to databases of papers and (presumably) reporting the number of results that the query returned.


[AI Incident Database](https://incidentdatabase.ai/)

[Paper: Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database](https://arxiv.org/abs/2011.08512)

When AI Systems Fail: Introducing the AI Incident Database

One obvious way to improve safety is to learn from past mistakes, and not repeat them. This suggests that it would be particularly valuable to have a repository of past incidents that have occurred, so that people can learn from them; indeed both aviation and cybersecurity have their own incident databases. The AI Incidents Database aims to fill this gap within AI. The database currently has over a thousand incidents covering a wide range of potential issues, including self-driving car accidents, wrongful arrests due to bad facial recognition or machine translation, and algorithm-driven “flash crash”.


I agree that technology, especially widely used one, always has a political aspect: providing access to new capabilities can have large societal effects that affect actors differently, based on accessibility and how well the capabilities match with their existing ones. I also like the distinction between deployment in market settings, where opting out is possible, and political settings, even though this can obviously become quite fuzzy when network effects make competition difficult. Lastly, I strongly agree that we will need an iterative process involving qualitative feedback to ensure good outcomes from AI, but worry that competitive pressures or the underestimation of runaway feedback dynamics could lead to situations where AI systems directly or indirectly prevent us from adjusting them.

Hard Choices in Artificial Intelligence: Addressing Normative Uncertainty through Sociotechnical Commitments

Roel Dobbe, Thomas Krendl Gilbert, Yonatan Mintz

This paper looks at AI Safety from the lens of Science & Technology Studies. AI systems are framed as sociotechnical, meaning that both social and technical aspects influence their development and deployment. As AI systems scale, we may face difficult value choices: for example, how do we compare between values like equality and liberty when we cannot have both? This can be resolved using intuitive comparability (IC): even if it seems incomparable in the abstract, humans are still able to make deliberate tradeoffs that involve these values. This is particularly relevant for so-called hard choices where different alternatives seem to be on par, which require normative reasoning and the incorporation of values that were previously neglected. As AI systems can reshape the contexts in which stakeholders exist, we are likely to encounter many hard choices as new values emerge or become more salient. The IC perspective then suggests that AI systems and criteria for evaluation should be iteratively redesigned based on qualitative feedback from different stakeholders.

The authors then argue that as AI systems encode hard choices made by or for different stakeholders, they are fundamentally political. Developers are in a position of power and have the responsibility to take a political stance. A set of challenges to preserve stakeholders' access to hard choices in an AI system's development are proposed:
1. The design of the system should involve the explicit negotiation of modelling assumptions or the lack thereof and learning goals as well as deliberation about future value conflicts or externalities that might make a reiteration of the design process necessary and give enough flexibility for stakeholders to imprint their own values during training and deployment.
 2. The training of the system should involve an impartial assessment of the tradeoff between visible performance and potential hidden disadvantages like bias, brittleness or unwanted strategic behaviour and involve stakeholders in the resolution. Furthermore, a team consensus about what can and cannot be done to improve performance should be established.
 3. During deployment, there should be an easily useable and trustworthy feedback channel for stakeholders, who should either have an explicit say in shaping the system (political setting) or the option to opt out of the system without major costs (market setting).

These challenges should be part of the training of AI designers and engineers, while the public needs to be sufficiently educated about the assumptions behind and the abilities and limitations of AI systems to allow for informed dissent.


I’m a big fan of this technique: it has helped me notice that many of my concepts were confused. For example, this helped me get deconfused about wireheading and inner alignment. It’s an instance of the more general technique (that I also like) of taking an abstract argument and making it more concrete and realistic, which often reveals aspects of the argument that you wouldn’t have previously noticed.

The "Backchaining to Local Search" Technique in AI Alignment

This post explains a technique to use in AI alignment, that the author dubs “backchaining to local search” (where local search refers to techniques like gradient descent and evolutionary algorithms). The key idea is to take some proposed problem with AI systems, and figure out mechanistically how that problem could arise when running a local search algorithm. This can help provide information about whether we should expect the problem to arise in practice.


Foundational Philosophical Questions in AI Alignment

This podcast starts with the topic of the paper <@Artificial Intelligence, Values and Alignment@> and then talks about a variety of different philosophical questions surrounding AI alignment.


State of AI Ethics

Abhishek Gupta, Marianna Ganapini, Renjie Butalid, Camylle Lanteigne, Allison Cohen, Mo Akif, Tania De Gasperis, Victoria Heath, Erick Galinkin

This report from the Montreal AI Ethics Institute has a wide variety of summaries on many different topics in AI ethics, quite similarly to this newsletter in fact.