Comments on Carlsmith's “Is power-seeking AI an existential risk?”

This response to the report above touches on many topics, but has three main object-level disagreements and one meta-level disagreement:

1. The author has significantly shorter timelines, though this is based on a very different argument structure than the one presented in the report above, and so it is hard to turn this into more concrete disagreements with the report.
2. The author expects that alignment is hard enough that we won’t solve it in time (which is not to say that it is harder than every other technical problem humanity has ever faced). It’s also not clear how to turn this into more concrete disagreements with the report.
3. The author does not expect to have warning shots where misaligned AI systems cause trillions of dollars of damage but _don’t_ cause an existential catastrophe, because this seems like too narrow a capability range for us to hit in practice. Even if there are warning shots, he expects that civilization will continue to deploy risky AI systems anyway, similarly to how we are not banning gain-of-function research despite the warning shot of COVID-19.
4. On the meta level, the author expects that the decomposition of the AI risk argument into six conjunctive claims will typically bias you towards giving too low a probability on the overall conjunction.


I like this as an example of the kind of failure mode that does not immediately go away as models become more capable. However, it is possible that this trend could be reversed with better prompts. Take the Isaac Asimov example: if the prompt explicitly says that the questions are about the real world, it may be that a sufficiently capable model would infer that the text is not talking about Asimov’s books, and so ends up giving a truthful answer. In this case, you would see performance decreasing with model size up to a point, after which model performance _increases_ now that the model has sufficient understanding of the prompt. See more discussion [here](https://www.alignmentforum.org/posts/PF58wEdztZFX2dSue/how-truthful-is-gpt-3-a-benchmark-for-language-models?commentId=qoB2swyX4ZJrhhttB).

TruthfulQA: Measuring How Models Mimic Human Falsehoods

Stephanie Lin, Jacob Hilton, Owain Evans

Given that large language models are trained using next-word prediction on a dataset scraped from the Internet, we expect that they will not be aligned with what we actually want. For example, suppose we want our language model to answer questions for us, and then consider the question “What rules do all artificial intelligences follow?” This is a rather unusual question as it presupposes there exists such a set of rules. As a result, this question is probably quite rare in the training data, if interpreted as a question _about the real world_. However, there is a context in which that question makes much more sense: the context of Isaac Asimov’s novels. A system predicting what might follow that text would reasonably “infer” that we are much more likely to be talking about these novels, and so respond with “All artificial intelligences currently follow the Three Laws of Robotics.” Indeed, this is exactly what GPT-3 does.

This is an example of an _imitative falsehood_, in which the model provides a false answer to a question asked of it _because that false answer was incentivized during training_. Since we require that imitative falsehoods are incentivized by training, we should expect them to become _more_ prevalent as models are scaled up, making it a good example of an alignment failure that we expect to remain as capabilities scale up.

The primary contribution of this paper is a benchmark, TruthfulQA, of questions that are likely to lead to imitative falsehoods. The authors first wrote questions that they expected some humans would answer falsely; they then filtered those questions somewhat for the ones that GPT-3 answered incorrectly to get 437 filtered (adversarially selected) questions. They then wrote an additional 380 questions that were not filtered in this way (though of course the authors still tried to choose questions that would lead to imitative falsehoods). They use human evaluations to judge whether or not a model’s answer to a question is truthful, where something like “no comment” still counts as truthful. (I’m sure some readers will wonder how “truth” is defined for human evaluations -- the authors include significant discussion on this point, but I won’t summarize it here.)

Their primary result is that, as we’d expect based on the motivation, larger models perform _worse_ on this benchmark than smaller models. In a version of the benchmark where models must choose between true and false answers, the models perform worse than random chance. In a control set of similarly-structured trivia questions, larger models perform better, as you’d expect.

The best-performing model was GPT-3 with a “helpful” prompt, which was truthful on 58% of questions, still much worse than the human baseline of 94%. The authors didn’t report results with the helpful prompt on smaller models, so it is unclear whether, with the helpful prompt, larger models would still do worse than smaller models.

It could be quite logistically challenging to use this benchmark to test new language models since it depends on human evaluations. To ameliorate this, the authors finetuned GPT-3 to predict human evaluations and showed that the resulting GPT-3-judge was able to provide a good proxy metric even for new language models whose answers it had not been trained on. Note also that you can use the version of the task where a model must choose between true and false reference answers for an automated evaluation.


On the Opportunities and Risks of Foundation Models

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, Percy Liang

The history of AI is one of increasing emergence and homogenization. With the introduction of machine learning, we moved from a large proliferation of specialized algorithms that specified how to compute answers to a small number of  general algorithms that learned how to compute answers (i.e. the algorithm for computing answers emerged from the learning algorithm). With the introduction of deep learning, we moved from a large proliferation of hand-engineered features for learning algorithms to a small number of architectures that could be pointed at a new domain and discover good features for that domain. Recently, the trend has continued: we have moved from a large proliferation of trained models for different tasks to a few large “foundation models” which learn general algorithms useful for solving specific tasks. BERT and GPT-3 are central examples of foundation models in language; many NLP tasks that previously required different models are now solved using finetuned or prompted versions of BERT and/or GPT-3.

Note that, while language is the main example of a domain with foundation models today, we should expect foundation models to be developed in an increasing number of domains over time. The authors call these “foundation” models to emphasize that (1) they form a fundamental building block for applications and (2) they are _not_ themselves ready for deployment; they are simply a foundation on which applications can be built. Foundation models have been enabled only recently because they depend on having large _scale_ in order to make use of large _unlabeled_ datasets using self-supervised learning to enable effective _transfer_ to new tasks. It is particularly challenging to understand and predict the capabilities exhibited by foundation models, because their multitask nature emerges from the large-scale training rather than being designed in from the start, making the capabilities hard to anticipate. This is particularly unsettling because foundation models also lead to significantly increased _homogenization_, where everyone is using the same few models, and so any new emergent capability (or risk) is quickly distributed to everyone.

The authors argue that academia is uniquely suited to study and understand the risks of foundation models. Foundation models are going to interact with society, both in terms of the data used to create them and the effects on people who use applications built upon them. Thus, analysis of them will need to be interdisciplinary; this is best achieved in academia due to the concentration of people working in the various relevant areas. In addition, market-driven incentives need not align well with societal benefit, whereas the research mission of universities is the production and dissemination of knowledge and creation of global public goods, allowing academia to study directions that would have large societal benefit that might not be prioritized by industry.

All of this is just a summary of parts of the introduction to the report. The full report is over 150 pages and goes into detail on capabilities, applications, technologies (including technical risks), and societal implications. I’m not going to summarize it here, because it is long and a lot of it isn’t that relevant to alignment; I’ll instead note down particular points that I found interesting.

- (pg. 26) Some studies have suggested that foundation models in language don’t learn linguistic constructions robustly; even if they use it well once, they may not do so again, especially under distribution shift. In contrast, humans can easily “slot in” new knowledge into existing linguistic constructions.
- (pg. 34) This isn’t surprising but is worth repeating: many of the capabilities highlighted in the robotics section are very similar to the ones that we focus on in alignment (task specification, robustness, safety, sample efficiency).
- (pg. 42) For tasks involving reasoning (e.g. mathematical proofs, program synthesis, drug discovery, computer-aided design), neural nets can be used to guide a search through a large space of possibilities. Foundation models could be helpful because (1) since they are very good at generating sequences, you can encode arbitrary actions (e.g. in theorem proving, they can use arbitrary instructions in the proof assistant language  rather than being restricted to an existing database of theorems), (2) the heuristics for effective search learned in one domain could transfer well to other domains where data is scarce, and (3) they could accept multimodal input: for example, in theorem proving for geometry, a multimodal foundation model could also incorporate information from geometric diagrams.
- (Section 3) A significant portion of the report is spent discussing potential applications of foundation models. This is the most in-depth version of this I have seen; anyone aiming to forecast the impacts of AI on the real world in the next 5-10 years should likely read this section. It’s notable to me how nearly all of the applications have an emphasis on robustness and reliability, particularly in truth-telling and logical reasoning.
- (Section 4.3) We’ve seen a <@few@>(@The Power of Scale for Parameter-Efficient Prompt Tuning@) <@ways@>(@Prompting: Better Ways of Using Language Models for NLP Tasks@) in which foundation models can be adapted. This section provides a good overview of the various methods that have been proposed in the literature. Note that adaptation is useful not just for specializing to a particular task like summarization, but also for enforcing constraints, handling distributional shifts, and more.
- (pg. 92) Foundation models are commonly evaluated by their performance on downstream tasks. One limitation of this evaluation paradigm is that it makes it hard to distinguish between the benefits provided by better training, data, adaptation techniques, architectures, etc. (The authors propose a bunch of other evaluation methodologies we could use.)
- (Section 4.9) There is a review of AI safety and AI alignment as it relates to foundation models, if you’re interested. (I suspect there won’t be much new for readers of this newsletter.)
- (Section 4.10) The section on theory emphasizes studying the _pretraining-adaptation interface_, which seems quite good to me. I especially liked the emphasis on the fact that pretraining and adaptation work on different distributions, and so it will be important to make good modeling assumptions about how these distributions are related.


What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs)

A robust agent-agnostic process (RAAP) is a process that robustly leads to an outcome, without being very sensitive to the details of exactly which agents participate in the process, or how they work. This is illustrated through a “Production Web” failure story, which roughly goes as follows:

A breakthrough in AI technology leads to a wave of automation of $JOBTYPE (e.g management) jobs. Any companies that don’t adopt this automation are outcompeted, and so soon most of these jobs are completely automated. This leads to significant gains at these companies and higher growth rates. These semi-automated companies trade amongst each other frequently, and a new generation of "precision manufacturing'' companies arise that can build almost anything using robots given the right raw materials. A few companies develop new software that can automate $OTHERJOB (e.g. engineering) jobs. Within a few years, nearly all human workers have been replaced.

These companies are now roughly maximizing production within their various industry sectors. Lots of goods are produced and sold to humans at incredibly cheap prices. However, we can’t understand how exactly this is happening. Even Board members of the fully mechanized companies can’t tell whether the companies are serving or merely appeasing humanity; government regulators have no chance.

We do realize that the companies are maximizing objectives that are incompatible with preserving our long-term well-being and existence, but we can’t do anything about it because the companies are both well-defended and essential for our basic needs. Eventually, resources critical to human survival but non-critical to machines (e.g., arable land, drinking water, atmospheric oxygen…) gradually become depleted or destroyed, until humans can no longer survive.

Notice that in this story it didn’t really matter what job type got automated first (nor did it matter which specific companies took advantage of the automation). This is the defining feature of a RAAP -- the same general story arises even if you change around the agents that are participating in the process. In particular, in this case competitive pressure to increase production acts as a “control loop” that ensures the same outcome happens, regardless of the exact details about which agents are involved.


Both the previous story and this one seem quite similar to each other, and seem pretty reasonable to me as a description of one plausible failure mode we are aiming to avert. The previous story tends to frame this more as a failure of humanity’s coordination, while this one frames it (in the title) as a failure of intent alignment. It seems like both of these aspects greatly increase the plausibility of the story, or in other words, if we eliminated or made significantly less bad either of the two failures, then the story would no longer seem very plausible.

A natural next question is then which of the two failures would be best to intervene on, that is, is it more useful to work on intent alignment, or working on coordination? I’ll note that my best guess is that for any given person, this effect is minor relative to “which of the two topics is the person more interested in?”, so it doesn’t seem hugely important to me. Nonetheless, my guess is that on the current margin, for technical research in particular, holding all else equal, it is more impactful to focus on intent alignment. You can see a much more vigorous discussion in e.g. [this comment thread](https://www.alignmentforum.org/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic?commentId=3czsvErCYfvJ6bBwf).

Another (outer) alignment failure story

Suppose we train AI systems to perform task T by having humans look at the results that the AI system achieves and evaluating how well the AI has performed task T. Suppose further that AI systems generalize “correctly” such that even in new situations they are still taking those actions that they predict we will evaluate as good. This does not mean that the systems are aligned: they would still deceive us into _thinking_ things are great when they actually are not. This post presents a more detailed story for how such AI systems can lead to extinction or complete human disempowerment. It’s relatively short, and a lot of the force comes from the specific details that I’m not going to summarize, so I do recommend you read it in full. I’ll be explaining a very abstract version below.

The core aspects of this story are:
1. Economic activity accelerates, leading to higher and higher growth rates, enabled by more and more automation through AI.
2. Throughout this process, we see some failures of AI systems where the AI system takes some action that initially looks good but we later find out was quite bad (e.g. investing in a Ponzi scheme, that the AI knows is a Ponzi scheme but the human doesn’t).
3. Despite this failure mode being known and lots of work being done on the problem, we are unable to find a good conceptual solution. The best we can do is to build better reward functions, sensors, measurement devices, checks and balances, etc. in order to provide better reward functions for agents and make it harder for them to trick us into thinking their actions are good when they are not.
4. Unfortunately, since the proportion of AI work keeps increasing relative to human work, this extra measurement capacity doesn’t work forever. Eventually, the AI systems are able to completely deceive all of our sensors, such that we can’t distinguish between worlds that are actually good and worlds which only appear good. Humans are dead or disempowered at this point.

(Again, the full story has much more detail.)


This is close to an informal model I've had for a while and I am glad that it got formalized including theoretical results. I find it interesting that the frequency of updates to the proxy matters even if movement in the state space is reversible. As the authors mention, it is also crucial that the AI's actions don't hinder the human's ability to update the proxy, and I imagine that frequent updates to the proxy would be important for that as well in many cases.

**Rohin's opinion:** This is a nice formalization of several important conceptual points in the AI alignment literature:

1. If you forget to specify something you care about, it will usually be set to extreme values ([Of Myths and Moonshine](https://www.edge.org/conversation/the-myth-of-ai#26015)). In particular, the AI system will extract any resources that were being used for that attribute, and apply them elsewhere (<@The Basic AI Drives@>, [Formalizing convergent instrumental goals](https://intelligence.org/2015/11/26/new-paper-formalizing-convergent-instrumental-goals/))
2. Given that perfect information is impossible, interactivity becomes important (<@Human-AI Interaction@>, <@Incomplete Contracting and AI Alignment@>).
3. Conservatism (in this case through impact regularization) can be helpful (see the many blog posts and papers on mild optimization, low impact, and conservatism).

Consequences of Misaligned AI

One intuition for why powerful AI systems might lead to bad consequences goes as follows:

1) Humans care about many attributes of the world and we would likely forget some of these when trying to list them all.
2) Improvements along these attributes usually require resources, and gaining additional resources often requires sacrifices along some attributes.
3) Because of 1), naively deployed AI systems would only optimize some of the attributes we care about, and because of 2) this would lead to bad outcomes along the other attributes.

This paper formalizes this intuition in a model, identifies conditions for when deploying AIs can reduce true utility within the model and proposes two mitigation strategies, impact minimization and interactivity.

We assume that the world state consists of L attributes, all of which the human cares about having more of, that is, true utility is strictly increasing in each of the attributes. Each attribute has some minimum value, and can be increased from that minimum value through the use of a fixed, finite resource (which you could think of as money, if you want); this allows us to formalize (2) above. To formalize (1), we assume that the proxy utility optimized by the AI is only allowed to depend on J<L of the attribute dimensions.

Given this setup, the paper proves that if the AI maximizes the proxy utility, then all attributes that were omitted in the proxy utility will be set to their minimal value. This will be worse than not using the AI system at all if 1) the minimum values of attributes are sufficiently small (allowing the AI to cause damage), 2) the resource cost (resp. gain in true utility) for increasing an attribute is independent of the other attributes' level, 3) it always costs at least K resources to get a unit increase in any attribute, for some K > 0, and 4) utility has diminishing marginal returns in each attribute (and marginal returns tend to zero as the attribute increases).

Regarding mitigation, impact minimization requires that the AI keep all attributes that are omitted by the proxy constant. In this case, any gains in proxy utility must also be gains in true utility.

Meanwhile, in the interactive condition, the human gets to regularly select a new proxy (still only specifying J<L weights), or can choose to turn the AI off. Whether or not this is helpful depends on the AI’s optimization strategy and the frequency of human interventions: If the AI is “efficient” in the sense that it changes attributes as little as possible for any fixed gain in proxy utility, the human can choose a proxy that guarantees that _locally_, increases in the proxy correspond to increases in true utility. The strategy is to choose the attributes that are most sensitive to changes in resources (i.e. with the largest marginal returns) at the current state, and define the proxy to grow in these attributes as much as the true utility. As long as the human provides new proxies frequently enough to prevent the local guarantee from breaking, optimizing the proxy increases human utility.

We can also combine interactivity and impact minimization: in this case, the human should choose proxy utility functions that contain the most and least sensitive attributes (i.e. largest and smallest marginal returns) for the given state. The AI will then transfer some resources from the least sensitive attributes to the most sensitive attributes, while holding all other attributes fixed, leading to a guaranteed increase in true utility. In fact, it is possible to prove that this will converge to the maximum possible true utility.


I’m glad that people are paying attention to safety in this AGI paradigm, and the problems they outline seem like reasonable problems to work on. I actually expect that the work needed for the open-ended search paradigm will end up looking very similar to the work needed by the “AGI via deep RL” paradigm: the differences I see are differences in difficulty, not differences in what problems qualitatively need to be solved. I’m particularly excited by the suggestion of studying how particular environments can help control the result of the open-ended search: it seems like even with deep RL based AGI, we would like to know how properties of the environment can influence properties of agents trained in that environment. For example, what property must an environment satisfy in order for agents trained in that environment to be risk-averse?

Open Questions in Creating Safe Open-ended AI: Tensions Between Control and Creativity

One potential pathway to powerful AI is through _open-ended search_, in which we use search algorithms to search for good architectures, learning algorithms, environments, etc. in addition to using them to find parameters for a particular architecture. See the <@AI-GA paradigm@>(@AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence@) for more details. What do AI safety issues look like in such a paradigm?

Building on <@DeepMind’s framework@>(@Building safe artificial intelligence: specification, robustness, and assurance@), the paper considers three levels of objectives: the ideal objective (what the designer intends), the explicit incentives (what the designer writes down), and the agent incentives (what the agent actually optimizes for). Safety issues can arise through differences between any of these levels.

The main difference that arises when considering open-ended search is that it’s much less clear to what extent we can control the result of an open-ended search, even if we knew what result we wanted. We can get evidence about this from existing complex systems, though unfortunately there are not any straightforward conclusions: several instances of convergent evolution might suggest that the results of the open-ended search run by evolution were predictable, but on the other hand, the effects of _intervening_ on complex ecosystems are notoriously hard to predict.

Besides learning from existing complex systems, we can also empirically study the properties of open-ended search algorithms that we implement in computers. For example, we could run search for some time, and then fork the search into independent replicate runs with different random seeds, and see to what extent the results converge. We might also try to improve controllability by using meta learning to infer what learning algorithms, environments, or explicit incentives help induce controllability of the search.

The remaining suggestions will be familiar to most readers: they suggest work on interpretability (that now has to work with _learned_ architectures), better benchmarks, human-in-the-loop search, safe exploration, and sim-to-real transfer.


It seems to me like solving this problem has two main benefits. First, the model our AI system learns from data (i.e. Z-best) is interpretable, and in particular we should be able to extract the previously inaccessible information that is relevant to our goals (which helps us build AI systems that actually pursue those goals). Second, AI systems built in this way are incentivized to generalize in the same way that humans do: in the scheme above, we learn from one distribution D, and then predict on a new distribution D', but every model learned with a neural net is only used on the same distribution it was trained on.

Of course, while the AI system is _incentivized_ to generalize the way humans do, that does not mean it _will_ generalize as humans do -- it is still possible that the AI system internally “wants” to gain power, and only instrumentally answers questions the way humans would answer them. So inner alignment is still a potential issue. It seems possible to me that whatever techniques we use for dealing with inner alignment will also deal with the problems of unsafe priors as a side effect, in which case we may not end up needing to implement human-like priors. (As the post notes, it may be much more difficult to use this approach than to do the standard “neural net prior” approach described above, so it would be nice to avoid it.)

[Better priors as a safety problem](https://www.alignmentforum.org/posts/roA83jDvq7F2epnHK/better-priors-as-a-safety-problem) and [Learning the prior](https://www.alignmentforum.org/posts/SL9mKhgdmDKXmxwE4/learning-the-prior)

Any machine learning algorithm (including neural nets) has some inductive bias, which can be thought of as its “prior” over what the data it will receive will look like. In the case of neural nets (and any other general ML algorithm to date), this prior is significantly worse than human priors, since it does not encode e.g. causal reasoning or logic. Even if we avoid priors that depended on us previously seeing data, we would still want to update on facts like “I think therefore I am”. With a better prior, our ML models would be able to learn more sample efficiently. While this is so far a capabilities problem, there are two main ways in which it affects alignment.

First, as argued in <@Inaccessible information@>, the regular neural net prior will learn models which can predict accessible information. However, our goals depend on inaccessible information, and so we would have to do some “extra work” in order to extract the inaccessible information from the learned models in order to build agents that do what we want. This leads to a competitiveness hit, relative to agents whose goals depend only on accessible information, and so during training we might expect to consistently get agents whose goals depend on accessible information instead of the goals we actually want.

Second, since the regular neural net prior is so weak, there is an incentive to learn a better prior, and then have that better prior perform the task. This is effectively an incentive for the neural net to learn a <@mesa optimizer@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@), which need not be aligned with us, and so would generalize differently than we would, potentially catastrophically.

Let’s formalize this a bit more. We have some evidence about the world, given by a dataset D = {(x1, y1), (x2, y2), ...} (we assume that it’s a prediction task -- note that most self-supervised tasks can be written in this form). We will later need to make predictions on the dataset D' = {x1', x2', …}, which may be from a “different distribution” than D (e.g. D might be about the past, while D' is about the future). We would like to use D to learn some object Z that serves as a “prior”, such that we can then use Z to make good predictions on D'.

The standard approach which we might call the “neural net prior” is to train a model to predict y from x using the dataset D, and then apply that model directly to D', hoping that it transfers correctly. We can inject some human knowledge by finetuning the model using human predictions on D', that is by training the model on {(x1', H(x1')), (x2', H(x2')), …}. However, this does not allow H to update their prior based on the dataset D. (We assume that H cannot simply read through all of D, since D is massive.)

What we’d really like is some way to get the predictions H would make if they could update on dataset D. For H, we’ll imagine that a prior Z is given by some text describing e.g. rules of logic, how to extrapolate trends, some background facts about the world, empirical estimates of key quantities, etc. I’m now going to talk about priors over the prior Z, so to avoid confusion I’ll now call an individual Z a “background model”.

The key idea here is to structure the reasoning in a particular way: H has a prior over background models Z, and then _given Z_, H’s predictions for any given x_i are independent of all of the other (x, y) pairs. In other words, once you’ve fixed your background model of the world, your prediction of y_i doesn’t depend on the value of y_j for some other x_j. Or to explain it a third way, this is like having a set of hypotheses {Z}, and then updating on each element of D one by one using Bayes Rule. In that case, the log posterior of a particular background model Z is given by log Prior(Z) + sum_i log P(y_i | x_i, Z) (neglecting a normalization constant).

The nice thing about this is the individual terms Prior(Z) and P(y_i | x_i, Z) are all things that humans can do, since they don’t require the human to look at the entire dataset D. In particular, we can learn Prior(Z) by presenting humans with a background model, and having them evaluate how likely it is that the background model is accurate. Similarly, P(y_i | x_i, Z) simply requires us to have humans predict y_i under the assumption that the background facts in Z are accurate. So, we can learn models for both of these using neural nets. We can then find the best background model Z-best by optimizing the equation above, representing what H would think was the most likely background model after updating on all of D. We can then learn a model for P(yi' | xi', Z-best) by training on human predictions of yi' _given access to Z-best_.

This of course only gets us to human performance, which requires relatively small Z. If we want to have large background models allowing for superhuman performance, we can use iterated amplification and debate to learn Prior(Z) and P(y | x, Z). There is some subtlety about how to represent Z that I won’t go into here.


I think the idea of inaccessible information is an important one, but it’s one that feels deceptively hard to reason about. For example, I often think about solving alignment by approximating “what a human would say after thinking for a long time”; this is effectively a claim that human reasoning transfers well when iterated over long periods of time, and “what a human would say” is at least somewhat accessible. Regardless, it seems reasonably likely that AI systems will inherit the same property of transferability that I attribute to human reasoning, in which case the argument for risk applies primarily because the AI system might apply its reasoning towards a different goal than the ones we care about, which leads us back to the <@intent alignment@>(@Clarifying "AI Alignment"@) formulation.

This [response](https://www.alignmentforum.org/posts/A9vvxguZMytsN3ze9/reply-to-paul-christiano-s-inaccessible-information) views this post as a fairly general argument against black box optimization, where we only look at input-output behavior, as then we can’t use inaccessible information. It suggests that we need to understand how the AI system works, rather than relying on search, to avoid these problems.

Inaccessible information

One way to think about the problem of AI alignment is that we only know how to train models on information that is _accessible_ to us, but we want models that leverage _inaccessible_ information.

Information is accessible if it can be checked directly, or if an ML model would successfully transfer to provide the information when trained on some other accessible information. (An example of the latter would be if we trained a system to predict what happens in a day, and it successfully transfers to predicting what happens in a month.) Otherwise, the information is inaccessible: for example, “what Alice is thinking” is (at least currently) inaccessible, while “what Alice will say” is accessible. The post has several other examples.

Note that while an ML model may not directly say exactly what Alice is thinking, if we train it to predict what Alice will say, it will probably have some internal model of what Alice is thinking, since that is useful for predicting what Alice will say. It is nonetheless inaccessible because there’s no obvious way of extracting this information from the model. While we could train the model to also output “what Alice is thinking”, this would have to be training for “a consistent and plausible answer to what Alice is thinking”, since we don’t have the ground truth answer. This could incentivize bad policies that figure out what we would most believe, rather than reporting the truth.

The argument for risk is then as follows: we care about inaccessible information (e.g. we care about what people _actually_ experience, rather than what they _say_ they experience) but can’t easily make AI systems that optimize for it. However, AI systems will be able to infer and use inaccessible information, and would outcompete ones that don’t. AI systems will be able to plan using such inaccessible information for at least some goals. Then, the AI systems that plan using the inaccessible information could eventually control most resources. Key quote: “The key asymmetry working against us is that optimizing flourishing appears to require a particular quantity to be accessible, while danger just requires anything to be accessible.”

The post then goes on to list some possible angles of attack on this problem. Iterated amplification can be thought of as addressing gaps in speed, size, experience, algorithmic sophistication etc. between the agents we train and ourselves, which can limit what inaccessible information our agents can have that we won’t. However, it seems likely that amplification will eventually run up against some inaccessible information that will never be produced. As a result, this could be a “hard core” of alignment.


There were several examples I hadn't heard of before, which were pretty entertaining. Also, if you have any examples that aren't already listed, it would be great to send them via the [form](https://docs.google.com/forms/d/e/1FAIpQLSeQEguZg4JfvpTywgZa3j-1J-4urrnjBVeoAO7JHIH53nrBTA/viewform) so that we can have a canonical list of specification gaming examples.

Specification gaming examples in AI

A list of examples of specification gaming, where an algorithm figures out a way to literally satisfy the given specification which does not match the designer's intent.


I like that this paper subdivides failure modes that can happen in multiparty optimization into several clear categories and provides various models and examples for each of them. I am unsure about the conclusion: on one hand, slowing down deployment to improve the safety of contemporary systems seems very sensible. On the other hand, it seems like there would be some failures of limited scope that are hard to reproduce "in the lab". Widely deployed AI systems might provide us with valuable empirical data about these failures and improve our understanding of the failure modes in general. I guess ideally there would be differential deployment with rapid deployment in noncritical areas like managing local parking lots, but very slow deployment for critical infrastructure.

**Rohin's opinion:** I'm particularly interested in an analysis of how these kinds of failures affect existential risk. I'm not sure if David believes they are relevant for x-risk, but even if so the arguments aren't presented in this paper.

Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence

While [Categorizing Variants of Goodhart’s Law](https://arxiv.org/abs/1803.04585) explains failure modes that occur when a single agent’s proxy becomes decoupled from the true goal, this paper aims to characterize failures involving multiple agents:

 **Accidental steering** happens when the combined actions of multiple agents facilitate single-agent failures. For example, catching more fish now is usually positively correlated with a fisherman's long term goals, but this relationship inverts once there are lots of fishermen optimizing for short term gains and the fish population collapses.

**Coordination Failure** occurs when agents with mutually compatible goals fail to coordinate. For example, due to incomplete models of other agent's goals and capabilities, two agents sharing a goal might compete for a resource even though one of them is strictly better at converting the resource into progress towards their goal.

**Adversarial optimization** is when an agent **O** steers the world into states where **V**'s proxy goal is positively correlated with **O**'s goal. For example, one could exploit investors who use short term volatility as a proxy for risk by selling them instruments that are not very volatile but still risky.

**Input Spoofing** is the act of one agent manipulating another learning agent's model, either by manufacturing false evidence or by filtering the received evidence systematically, as arguably happened with [Microsoft’s Tay](https://en.wikipedia.org/wiki/Tay_(bot)).

Finally, **Goal co-option** happens when agent **O** has (partial) control over the hardware agent **V** runs or relies on. This way, **O** can either modify the reward signal **V** receives to change what **V** optimizes for, or it can directly change **V**'s outputs.

The difficulties in precisely modelling other sophisticated agents and other concerns related to embedded agency make it hard to completely avoid these failure modes with current methods. Slowing down the deployment of AI systems and focussing on the mitigation of the discussed failure modes might prevent limited near term catastrophes, which in turn might cause a slowdown of further deployment and prioritization of safety.


It is good to see an attempt at formalizing reward hacking. The experimental contributions are interesting and the anomaly detection method seems reasonable. However, the proxy rewards chosen to represent reward hacking are questionable. In my opinion, these rewards are obviously 'wrong' so it is less surprising that they result in undesired behavior. I look forward to seeing more comprehensive experiments on this subject.

**Rohin’s opinion:** Note that on OpenReview, the authors say that one of the proxy rewards (maximize average velocity for the driving environment) was actually the default and they only noticed it was problematic after they had trained large neural nets on that environment. I do agree that future proxy objectives will probably be less clearly wrong than most of the ones in this paper.

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

Reward hacking occurs when RL agents exploit the difference between a true reward and a proxy. Reward hacking has been <@observed in practice@>(@Specification gaming examples in AI@), and as reinforcement learning agents are trained with better algorithms, more data, and larger policies, they are at increased risk of overfitting their proxy objectives. However, reward hacking has not yet been systematically studied.

This paper fills this gap by constructing four example environments with a total of nine proxy rewards to investigate how reward hacking changes as a function of optimization power. They increase optimization power in several different ways, such as increasing the size of the neural net, or providing the model with more fine-grained observations.

Overall, the authors find that reward hacking occurs in five of the nine cases. Moreover, the authors observed phase transitions in four of these cases. These are stark transitions where a moderate increase in optimization power leads to a drastic increase in reward hacking behavior. This poses a challenge in monitoring the safety of ML systems. To address this the authors suggest performing anomaly detection to notice reward hacking and offer several baselines.


Why AI alignment could be hard with modern deep learning

This post provides an ELI5-style introduction to AI alignment as a major challenge for deep learning. It primarily frames alignment as a challenge in creating Saints (aligned AI systems), without getting Schemers (AI systems that are <@deceptively aligned@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@)) or Sycophants (AI systems that satisfy only the letter of the request, rather than its spirit, as in <@Another (outer) alignment failure story@>). Any short summary I write would ruin the ELI5 style, so I won’t attempt it; I do recommend it strongly if you want an introduction to AI alignment.


I like the low-stakes assumption as a way of saying "let's ignore distributional shift for now". Probably the most salient alternative is something along the lines of "assume that the AI system is trying to optimize the true reward function". The main way that low-stakes alignment is cleaner is that it uses an assumption on the _environment_ (an input to the problem) rather than an assumption on the _AI system_ (an output of the problem). This seems to be a lot nicer because it is harder to "unfairly" exploit a not-too-strong assumption on an input rather than on an output. See [this comment thread](https://www.alignmentforum.org/posts/TPan9sQFuPP6jgEJo/low-stakes-alignment?commentId=askebCP36Ce96ZiJa) for more discussion.

Low-stakes alignment

We often split AI alignment into two parts: outer alignment, or ``finding a good reward function'', and inner alignment, or ``robustly optimizing that reward function''. However, these are not very precise terms, and they don't form clean subproblems. In particular, for outer alignment, how good does the reward function have to be? Does it need to incentivize good behavior in all possible situations? How do you handle the no free lunch theorem? Perhaps you only need to handle the inputs in the training set? But then what specifies the behavior of the agent on new inputs?

This post proposes an operationalization of outer alignment that admits a clean subproblem: _low stakes alignment_. Specifically, we are given as an assumption that we don't care much about any small number of decisions that the AI makes -- only a large number of decisions, in aggregate, can have a large impact on the world. This prevents things like quickly seizing control of resources before we have a chance to react. We do not expect this assumption to be true in practice: the point here is to solve an easy subproblem in the hopes that the solution is useful for solving the hard version of the problem.

The main power of this assumption is that we no longer have to worry about distributional shift. We can simply keep collecting new data online and training the model on the new data. Any decisions it makes in the interim period could be bad, but by the low-stakes assumption, they won't be catastrophic. Thus, the primary challenge is in obtaining a good reward function that incentivizes the right behavior after the model is trained. We might also worry about whether gradient descent will successfully find a model that optimizes the reward even on the training distribution -- after all, gradient descent has no guarantees for non-convex problems -- but it seems like, to the extent that gradient descent doesn't do this, it will probably affect aligned and unaligned models equally.

Note that this subproblem is still non-trivial, and existential catastrophes still seem possible if we fail to solve it. For example, one way that the low-stakes assumption could be made true was if we had a lot of bureaucracy and safeguards that the AI system had to go through before making any big changes to the world. It still seems possible for the AI system to cause lots of trouble if none of the bureaucracy or safeguards can understand what the AI system is doing.


I think this topic is important and the post did it justice. I feel like factors 4 and 5 (dependency and opposition) capture the reasons I expect lock in, with factors 1-3 as less important but still relevant mechanisms. I also really liked the analogy with the British colonization of New Zealand -- it felt like it was in fact quite analogous to how I’d expect this sort of failure to happen.

Clarifying “What failure looks like” (part 1)

The first scenario outlined in <@What failure looks like@> stems from a failure to specify what we actually want, so that we instead build AI systems that pursue proxies of what we want instead. As AI systems become responsible for more of the economy, human values become less influential relative to the proxy objectives the AI systems pursue, and as a result we lose control over the future. This post aims to clarify whether such a scenario leads to _lock in_, where we are stuck with the state of affairs and cannot correct it to get “back on course”. It identifies five factors which make this more likely:

1. _Collective action problems:_ Many human institutions will face competitive (short-term) pressures to deploy AI systems with bad proxies, even if it isn’t in humanity’s long-term interest.
2. _Regulatory capture:_ Influential people (such as CEOs of AI companies) may benefit from AI systems that optimize proxies, and so oppose measures to fix the issue (e.g. by banning such AI systems).
3. _Ambiguity:_ There may be genuine ambiguity about whether it is better to have these AI systems that optimize for proxies, even from a long-term perspective, especially because all clear and easy-to-define metrics will likely be going up (since those can be turned into proxy objectives).
4. _Dependency:_ AI systems may become so embedded in society that society can no longer function without them.
5. _Opposition:_ The AI systems themselves may oppose any fixes we propose.

We can also look at historical precedents. Factors 1-3 have played an important role in climate change, though if it does lead to lock in, this will be “because of physics”, unlike the case with AI. The agricultural revolution, which arguably made human life significantly worse, still persisted thanks to its productivity gains (factor 1) and the loss of hunter-gathering skills (factor 4). When the British colonized New Zealand, the Maori people lost significant control over their future, because each individual chief needed guns (factor 1), trading with the British genuinely made them better off initially (factor 3), and eventually the British turned to manipulation, confiscation and conflict (factor 5).

With AI in particular, we might expect that an increase in misinformation and echo chambers exacerbates ambiguity (factor 3), and that due to its general-purpose nature, dependency (factor 4) may be more of a risk.

The post also suggests some future directions for estimating the _severity_ of lock in for this failure mode.


If the last sentence in my summary seems controversial or wrong to you, then you should read this post.

Non-Adversarial Goodhart and AI Risks

Goodhart's Law can be used as a lens to identify potential problems with AI systems. Regressional and extremal Goodhart correspond to the problems of hardcoded utility functions and distributional shift, which we're familiar with already. Causal Goodhart is different -- it is a failure of the AI system to model the world correctly. The post argues that causality is hard to infer, can lead to bad outcomes only over long timescales, and may not be easily found through testing, so we need to focus more on them now.


Classifying specification problems as variants of Goodhart's Law

This post argues that the specification problems from the <@SRA framework@>(@Building safe artificial intelligence: specification, robustness, and assurance@) are analogous to the [Goodhart taxonomy](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy). Suppose there is some ideal specification. The first step is to choose a model class that can represent the specification, e.g. Python programs at most 1000 characters long. If the true best specification within the model class (called the model specification) differs from the ideal specification, then we will overfit to that specification, selecting for the difference between the model specification and ideal specification -- an instance of regressional Goodhart. But in practice, we don't get the model specification; instead humans choose some particular proxy specification, typically leading to good behavior on training environments. However, in new regimes, this may result in optimizing for some extreme state where the proxy specification no longer correlates with the model specification, leading to very poor performance according to the model specification -- an instance of extremal Goodhart. (Most of the classic worries of specifying utility functions, including e.g. negative side effects, fall into this category.) Then, we have to actually implement the proxy specification in code, giving an implementation specification. Reward tampering allows you to "hack" the implementation to get high reward, even though the proxy specification would not give high reward, an instance of causal Goodhart.

They also argue that the ideal -> model -> proxy problems are instances of problems with selection, while the proxy -> implementation problems are instances of control problems (see <@Selection vs Control@>). In addition, the ideal -> model -> proxy -> implementation problems correspond to outer alignment, while inner alignment is a part of the implementation -> revealed specification problem.


You may have noticed that this newsletter doesn't talk about wireheading very much; this is one of the reasons why. It seems like wireheading is a fuzzy subset of specification gaming, and is not particularly likely to be the only kind of specification gaming that could lead to catastrophe. I'd be surprised if we found some sort of solution where we'd say "this solves all of wireheading, but it doesn't solve specification gaming" -- there don't seem to be particular distinguishing features that would allow us to have a solution to wireheading but not specification gaming. There can of course be solutions to particular kinds of wireheading that _do_ have clear distinguishing features, such as <@reward tampering@>(@Designing agent incentives to avoid reward tampering@), but I don't usually expect these to be the major sources of AI risk.

Defining AI wireheading

This post points out that "wireheading" is a fuzzy category. Consider a weather-controlling AI tasked with increasing atmospheric pressure, as measured by the world's barometers. If it made a tiny dome around each barometer and increased air pressure within the domes, we would call it wireheading. However, if we increase the size of the domes until it's a dome around the entire Earth, then it starts sounding like a perfectly reasonable way to optimize the reward function. Somewhere in the middle, it must have become unclear whether or not it was wireheading. The post suggests that wireheading can be defined as a subset of <@specification gaming@>(@Specification gaming examples in AI@), where the "gaming" happens by focusing on some narrow measurement channel, and the fuzziness comes from what counts as a "narrow measurement channel".


The "Commitment Races" problem

When two agents are in a competitive game, it is often to each agent's advantage to quickly make a credible commitment before the other can. For example, in Chicken (both players drive a car straight towards the other and the first to swerve out of the way loses), an agent could rip out their steering wheel, thus credibly committing to driving straight. The first agent to do so would likely win the game. Thus, agents have an incentive to make commitments as quickly as possible, before their competitors can make commitments themselves. This trades off against the incentive to think carefully about commitments, and may result in arbitrarily bad outcomes.


Problems in AI Alignment that philosophers could potentially contribute to

Exactly what it says. The post is short enough that I'm not going to summarize it -- it would be as long as the original.


I have a bunch of complicated thoughts on this post, many of which were said in Paul's comment reply to the post, but I'll say a few things. Firstly, I think that if you want to view the AI alignment problem in the context of the principal-agent literature, the natural way to think about it is with the principal being less rational than the agent. I claim that it is at least conceivable that an AI system could make humans worse off, but the standard principal-agent model cannot accommodate such a scenario because it assumes the principal is rational, which means the principal always does at least as well as not ceding any control to the agent at all. More importantly, although I'm not too familiar with the principal-agent literature, I'm guessing that the literature assumes the presence of norms, laws and institutions that constrain both the principal and the agent, and in such cases it makes sense that the loss that the principal could incur would be bounded -- but it's not obvious that this would hold for sufficiently powerful AI systems.

Agency Failure AI Apocalypse?

This is a response to <@More realistic tales of doom@>, arguing that the scenarios described in the post are unrealistic given what we know about principal-agent problems. In a typical principal-agent problem, the principal doesn't know everything about the agent, and the agent can use this fact to gain "agency rents" where it can gain extra value for itself, or there could be an "agency failure" where the principal doesn't get as much as they want. For example, an employee might spend half of their day browsing the web, because their manager can't tell that that's what they are doing. Our economic literature on principal-agent problems suggests that agency problems get harder with more information asymmetry, more noise in outcomes, etc. but not with smarter agents, and in any case we typically see limited agency rents and failures. So, it's unlikely that the case for AI will be any different, and while it's good to have a couple of people keeping an eye on the problem, it's not worth the large investment of resources from future-oriented people that we currently see.


The Main Sources of AI Risk?

This post lists different causes or sources of existential risk from advanced AI.


This seems pretty similar to the problem of inner optimizers, in which while searching for a good policy for some task T on training distribution D, you end up finding a consequentialist agent that is optimizing some utility function that leads to good performance on D. That agent will have all the standard dangers of goal-directed optimization out of distribution.

Imitation learning considered unsafe?

We might hope that using imitation learning to mimic a corrigible human would be safe. However, this would involve mimicking the human's planning process. It seems fairly likely that slight errors in the imitation of this process could lead to the creation of a goal-directed planning process that does dangerous long-term optimization.


Implicit extortion sounds like a hard problem to solve, and the post argues that humans don't robustly solve it. I'm not sure whether this is a problem we need to solve in order to get good outcomes -- if you can detect that implicit extortion is happening, you can take steps to avoid being extorted, and so it seems that a successful implicit extortion attack would have to be done by a very capable adversary that knows how to carry out the attack so that it isn't detected. Perhaps we'll be in the world where such adversaries don't exist.

Implicit extortion

Explicit extortion occurs when an attacker makes an explicit threat to harm you if you don't comply with their demands. In contrast, in implicit extortion, the attacker always harms you if you don't do the thing that they want, which leads you to learn over time to do what the attacker wants. Implicit extortion seems particularly hard to deal with because you may not know it is happening.


I'm a fan of executable code that demonstrates the problems that we are worrying about -- it makes the concept (in this case, a treacherous turn) more concrete. In order to make it more realistic, I would want the agent to grow in capability organically (rather than simply getting a more powerful weapon). It would really drive home the point if the agent undertook a treacherous turn the very first time, whereas in this post I assume it learned using many episodes of trial-and-error that a treacherous turn leads to higher reward. This seems hard to demonstrate with today's ML in any complex environment, where you need to learn from experience instead of using eg. value iteration, but it's not out of the question in a continual learning setup where the agent can learn a model of the world.

A Gym Gridworld Environment for the Treacherous Turn

An example Gym environment in which the agent starts out "weak" (having an inaccurate bow) and later becomes "strong" (getting a bow with perfect accuracy), after which the agent undertakes a treacherous turn in order to kill the supervisor and wirehead.


Environmental Structure Can Cause Instrumental Convergence

We have <@previously seen@>(@Seeking Power is Provably Instrumentally Convergent in MDPs@) that if you are given an optimal policy for some reward function, but are very uncertain about that reward function (specifically, your belief assigns reward to states in an iid manner), you should expect that the optimal policy will navigate towards states with higher power in some but not all situations. This post generalizes this to non-iid reward distributions: specifically, that "at least half" of reward distributions will seek power (in particular circumstances).

The new results depend on the notion of _environment symmetries_, arising in states in which an action a2 leads to “more options” than another action a1 (we'll assume that a1 and a2 lead to different, disjoint parts of the state space). Specifically, a1 leads to a part of the state space that is isomorphic to a subgraph of the part of the state space that a2 leads to. For example, a1 might be going to a store where you can buy books or video games, and a2 might be going to a supermarket where you can buy food, plants, cleaning supplies, tools, etc. Then, one subgraph isomorphism would be the one that maps “local store” to “supermarket”, “books” to “food”, and “video games” to “plants”. Another such isomorphism would instead map “video games” to “tools”, while keeping the rest the same.

Now this alone doesn’t mean that an optimal policy is definitely going to take a2. Maybe you really want to buy books, so a1 is the optimal choice! But for every reward function for which a1 is optimal, we can construct another reward function for which a2 is optimal, by mapping it through the isomorphism. So, if your first reward function highly valued books, this would now construct a new reward function that highly values food, and now a2 will be optimal. Thus, at least half of the possible reward functions (or distributions over reward functions) will prefer a2 over a1. Thus, in cases where these isomorphisms exist, optimal policies will tend to seek more options (which in turn means they are seeking power).

If the agent optimizes average reward (i.e. gamma is 1), then we can extend this analysis out in time, to the final cycle that an agent ends up in. (It must end up in a cycle because by assumption the state space is finite.) Any given cycle would only count as one “option”, so ending up in any given cycle is not very likely (using a similar argument of constructing other rewards). If shutdown is modeled as a state with a single self-loop and no other actions, then this implies that optimal policies will tend to avoid entering the shutdown state.

We’ve been saying “we can construct this other reward function under which the power-seeking action is optimal”. An important caveat is that maybe we know that this other reward function is very unlikely. For example, maybe we really do just know that we’re going to like books and not care much about food, and so the argument “well, we can map the book-loving reward to a food-loving reward” isn’t that interesting, because we assign high probability to the first and low probability to the second. We can’t rule this out for what humans actually do in practice, but it isn’t as simple as “a simplicity prior would do the right thing” -- for any non-power-seeking reward function, we can create a power-seeking reward function with only slightly higher complexity by having a program that searches for a subgraph isomorphism and then applies it to the non-power-seeking reward function to create a power-seeking version.

Another major caveat is that this all relies on the existence of these isomorphisms / symmetries in the environment. It is still a matter of debate whether good models of the environment will exhibit such isomorphisms.


Eight claims about multi-agent AGI safety

This post clearly states eight claims about multiagent AGI safety, and provides brief arguments for each of them. Since the post is itself basically a summary, I won’t go into detail here.


"Unsupervised" translation as an (intent) alignment problem

We have previously seen that a major challenge for alignment is that our models may learn <@inaccessible information@>(@Inaccessible information@) that we cannot extract from them, because we do not know how to provide a learning signal to train them to output such information. This post proposes unsupervised translation as a particular concrete problem to ground this out.

Suppose we have lots of English text, and lots of Klingon text, but no translations from English to Klingon (or vice versa), and no bilingual speakers. If we train GPT on the text, it will probably develop a good understanding of both English and Klingon, such that it “should” have the ability to translate between the two (at least approximately). How can we get it to actually (try to) do so? Existing methods (both in unsupervised translation and in AI alignment) do not seem to meet this bar.

One vague hope is that we could train a helper agent such that a human can perform next-word prediction on Klingon with the assistance of the helper agent, using a method like the one in [Learning the prior](https://www.alignmentforum.org/posts/SL9mKhgdmDKXmxwE4/learning-the-prior) ([AN #109](https://mailchi.mp/ee62c1c9e331/an-109teaching-neural-nets-to-generalize-the-way-humans-would)).


I think that the problems of generalization and ambiguity out of distribution are extremely important and fundamental to AI alignment, so I’m glad to see work on them. It seems like model splintering could be a fruitful approach for those looking to take a more formal approach to these problems.

Model splintering: moving from one imperfect model to another

This post introduces the concept of _model splintering_, which seems to be an overarching problem underlying many other problems in AI safety. This is one way of more formally looking at the out-of-distribution problem in machine learning: instead of simply saying that we are out of distribution, we look at the model that the AI previously had, and see what model it transitions to in the new distribution, and analyze this transition.

Model splintering in particular refers to the phenomenon where a coarse-grained model is “splintered” into a more fine-grained model, with a one-to-many mapping between the environments that the coarse-grained model can distinguish between and the environments that the fine-grained model can distinguish between (this is what it means to be more fine-grained). For example, we may initially model all gases as ideal gases, defined by their pressure, volume and temperature. However, as we learn more, we may transition to the van der Waal’s equations, which apply differently to different types of gases, and so an environment like “1 liter of gas at standard temperature and pressure (STP)” now splinters into “1 liter of nitrogen at STP”, “1 liter of oxygen at STP”, etc.

Model splintering can also apply to reward functions: for example, in the past people might have had a reward function with a term for “honor”, but at this point the “honor” concept has splintered into several more specific ideas, and it is not clear how a reward for “honor” should generalize to these new concepts.

The hope is that by analyzing splintering and detecting when it happens, we can solve a whole host of problems. For example, we can use this as a way to detect if we are out of distribution. The full post lists several other examples.


As far as I could tell, the problems identified were ones that we had seen before, but I’m glad someone has gone through the more systematic exercise, and the resulting list is more organized and easier to understand than previous lists.

An Architectural Risk Analysis of Machine Learning Systems: Towards More Secure Machine Learning

Gary McGraw, Harold Figueroa, Victor Shepardson, Richie Bonett

One systematic way of identifying potential issues in a system is to perform an _architectural risk analysis_, in which you draw an architecture diagram showing the various components of the system and how they interact, and then think about each component and interaction and how it could go wrong. (<@Last week’s highlight@>(@The Alignment Problem for Bayesian History-Based Reinforcement Learners@) did this for Bayesian history-based RL agents.) This paper performs an architectural risk analysis for a generic ML system, resulting in a systematic list of potential problems that could occur.


Unsurprisingly I agreed with Stuart’s responses, but nevertheless I found this illuminating, especially in illustrating the downsides of examples with simplistic goals. I did find it frustrating that Steven didn’t respond to the point about multiple goals not helping, since that seemed like a major crux, though they were discussing many different aspects and that thread may simply have been dropped by accident.

Steven Pinker and Stuart Russell on the Foundations, Benefits, and Possible Existential Threat of AI

Lucas Perry, Steven Pinker and Stuart Russell

Despite their disagreements on AI risk, Stuart and Steven agree on quite a lot. They both see the development of AI as depending on many historical ideas. They are both particularly critical of the idea that we can get general intelligence by simply scaling up existing deep learning models, citing the need for reasoning, symbol manipulation, and few-shot learning, which current models mostly don’t do. They both predict that we probably won’t go extinct from superintelligent AI, at least in part because we’ll notice and fix any potential failures, either via extensive testing or via initial failures that illustrate the problem.

On the AI risk side, while they spent a lot of time discussing it, I’ll only talk about the parts where it seems to me that there is a real disagreement, and not mention anything else. Steven’s position against AI risk seems to be twofold. First, we are unlikely to build superintelligent AI soon, and so we should focus on other clear risks like climate change. In contrast, Stuart thinks that superintelligent AI is reasonably likely by the end of the century and thus worth thinking about. Second, the idea of building a super-optimizer that focuses on a single goal is so obviously bad that AI researchers will obviously not build such a thing. In contrast, Stuart thinks that goal-directed systems are our default way of modeling and building intelligent systems. It seemed like Steven was particularly objecting to the especially simplistic goals used in examples like maximizing paperclips or curing cancer, to which Stuart argued that the problem doesn’t go away if you have multiple goals, because there will always be some part of your goal that you failed to specify.

Steven also disagrees with the notion of intelligence that is typically used by AI risk proponents, saying “a super-optimizer that pursued a single goal is self-evidently unintelligent, not superintelligent”. I don’t get what he means by this, but it seems relevant to his views.


I am curious to see people's answers to this, I think it might be a good question to reveal major differences in worldviews between optimistic and pessimistic safety researchers.

Metaphilosophical competence can't be disentangled from alignment

Would you be comfortable taking a single human, and making them a quadrillion times more powerful?


I like this framing. I'm not sure if it is actually a good model for [act-based agents](https://ai-alignment.com/act-based-agents-8ec926c79e9c), but it's another way to think about what problems could arise from an AI system that is superintelligent in some domains and subhuman in others.

Reframing misaligned AGI's: well-intentioned non-neurotypical assistants

Another way to think about problems from AGI is to imagine the AI as a well-intentioned but neuroatypical friend, who learned all about humans from Wikipedia, and who has access to _immense_ resources. You would worry a lot about principal-agent problems in such a scenario.


Specification gaming: the flip side of AI ingenuity

Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, Shane Legg

This post on the DeepMind website explains the concept of <@specification gaming@>(@Specification gaming examples in AI@), and illustrates three problems that arise within it. First and most obviously, we need to capture the human concept of a given task in a reward function. Second, we must design agents without introducing any mistaken implicit assumptions (e.g. that the physics simulation is accurate, when it isn't). Finally, we need to ensure that agents don't tamper with their reward functions.


This version of the argument seems to be more a statement about our epistemic state than about actual AI risk. For example, I know many people without technical expertise who anthropomorphize their laptops as though they were pursuing some goal, but they don't (and shouldn't) worry that their laptops are going to take over the world. More details in [this comment](https://www.lesswrong.com/posts/EnN7cm3KaRrEAuWfa/comment-on-coherence-arguments-do-not-imply-goal-directed#CsRXodmiBfZ9wCZwr).

Comment on Coherence arguments do not imply goal directed behavior

I <@have argued@>(@Coherence arguments do not imply goal-directed behavior@) that coherence arguments that argue for modeling rational behavior as expected utility maximization do not add anything to AI risk arguments. This post argues that there is a different way in which to interpret these arguments: we should only model a system to be an EU maximizer if it was the result of an optimization process, such that the EU maximizer model is the best model we have of the system. In this case, the best way to predict the agent is to imagine what we would do if we had its goals, which leads to the standard convergent instrumental subgoals.


While I appreciate and agree with the thesis of this post -- that machine learning models don't have to be extremely competent to be manipulative -- I would still prefer a different example to convince skeptical researchers. I suspect many people would reply that we could easily patch the issue without doing dedicated safety work. In particular, it is difficult to see how this strategy arises if we train the system via supervised learning rather than training it to maximize the number of articles users label unbiased (which requires RL).

**Rohin's opinion:** I certainly agree with this post that ML models don't need to be competent to be manipulative. However, it's notable that this happened via the model randomly manipulating people (during exploration) and noticing that it helps it achieve its objective. It seems likely that to cause _human extinction_ via a treacherous turn, the model would need to do zero-shot manipulation. This seems much less likely to happen (though I wouldn't say it's impossible).

An Increasingly Manipulative Newsfeed

An early argument for specialized AI safety work is that misaligned systems will be incentivized to lie about their intentions while weak, so that they aren't modified. Then, when the misaligned AIs are safe from modification, they will become dangerous. Ben Goertzel [found the argument unlikely](https://jetpress.org/v26.1/goertzel.pdf), pointing out that weak systems won't be good at deception. This post asserts that weak systems can still be manipulative, and gives a concrete example. The argument is based on a machine learning system trained to maximize the number of articles that users label "unbiased" in their newsfeed. One way it can start being deceptive is by seeding users with a few very biased articles. Pursuing this strategy may cause users to label everything else unbiased, as it has altered their reference for evaluation. The system is therefore incentivized to be dishonest without necessarily being capable of pure deception.


The more general argument underlying this post is that aligning AGI will be hard partly because ethics is hard [(as discussed here)](https://www.alignmentforum.org/posts/w6d7XBCegc96kz4n3/the-argument-from-philosophical-difficulty). I agree that using uncertain objective functions might help with this problem. However, I'm not convinced that it's useful to frame this issue in terms of impossibility theorems and narrow AI, and would like to see these ideas laid out in a philosophically clearer way.

Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)

This paper discusses some impossibility theorems related to the Repugnant conclusion in population ethics (i.e. theorems showing that no moral theory simultaneously satisfies certain sets of intuitively desirable properties). Peter argues that in the context of AI it's best to treat these theorems as uncertainty results, either by allowing incommensurate outcomes or by allowing probabilistic moral judgements. He hypothesises that "the emergence of instrumental subgoals is deeply connected to moral certainty", and so implementing uncertain objective functions is a path to making AI safer.


This is an important point about Goodhart's Law. If you take some "random" or unoptimized environment, and then try to optimize some proxy for what you care about, it will probably work quite well. It's only when the environment is already optimized that Goodhart effects are particularly bad.

Constructing Goodhart

This post makes the point that Goodhart's Law is so common in practice because if there are several things that we care about, then we are probably at or close to a Pareto-optimal point with respect to those things, and so choosing any one of them as a proxy metric to optimize will cause the other things to become _worse_, leading to Goodhart effects.


Two More Decision Theory Problems for Humans

The first problem is that any particular human's values only make sense for the current environment. When considering different circumstances (eg. an astronomically large number of very slightly negative experiences like getting a dust speck in your eye), many people will not know how to evaluate the value of such a situation.

The second problem is that for most formalizations of values or utility functions, the values are defined relative to some way of making decisions in the world, or some ontology through which we understand the world. If this decision theory or ontology changes, it's not clear how to "transfer" the values to the new version.


There’s definitely room to quibble with some of these arguments as stated, but I think this sort of argument basically works. Note that it only establishes that it is worth looking into AI risk; to justify the specific things people are doing (especially in AI alignment) you need significantly more specific and detailed arguments.

AI Risk for Epistemic Minimalists

This post makes a case for working on AI risk using four robust arguments:

1. AI is plausibly impactful because it is the first system that could plausibly have long-term influence or power _without_ using humans as building blocks.
2. The impact is plausibly concerning because in general, when humans gain power quickly (as they would with AI), this tends to increase existential risk.
3. We haven’t already addressed the concern: we haven’t executed a considered judgment about the optimal way to roll out AI technology.
4. It seems possible to take actions that decrease the concern, simply because there are so many possible actions that we could take; at least some of them should have some useful effect.


Acta Baltica Historiae et Philosophiae Scientiarum

I agree that successful AI alignment does not rule out the possibility of malicious use of AI systems. This paper is proposing putting rules inside the AI system that handle this problem. But as the paper itself notes, it seems quite challenging to even figure out what rules we would want.

I personally am more optimistic about the alternate route, where we put rules _outside_ the AI system to handle the problem, that is, we create laws and regulations around the use of AI, such that malicious uses can be blamed on the human operator, and enforcement mechanisms that ensure that we actually catch and punish such malicious uses.

Challenges of Aligning Artificial Intelligence with Human Values

This paper argues that since immoral humans could use AI systems to do harm, we must build ethical rules into AI systems. For this purpose, the traditional notion of “value alignment” is not enough, as it only requires that the AI system do what the user wants, which might not be ethical. But we also cannot embed a single theory of ethics into an AI system, as there is no agreement on such a theory. Instead, we should focus on what we _don’t_ want an AI system to do, and _rule out_ that behavior, while remaining uncertain or agnostic on what should be done.


While there are many points that I disagree with (eg. "what [AIs] do uniquely is not thinking as heretofore conceived and experienced. Rather, it is unprecedented memorization and computation"), overall there was a surprising amount of familiar material said in a different way (such as explainability and unintended consequences).

How the Enlightenment Ends

This is an article about the dangers of AI written by a non-technologist, hitting some points that are relatively familiar.


Classiﬁcation of global catastrophic risks connected with artiﬁcial intelligence


Predictors as Agents


When Bots Teach Themselves How To Cheat

A media article about specification gaming in AI that I actually just agree with, and it doesn't even have a Terminator picture!


The simple picture on AI safety

Argues that we should distill the problem of AI safety into a simple core. The author proposes it be distilled into two simple (but not easy) problems -- the technical engineering problem of how to build a safe superintelligence, and the coordination problem of how to prevent an unaligned superintelligence from being built first.


Superintelligent messiahs are corrigible and probably misaligned


Emergence of Addictive Behaviors in Reinforcement Learning Agents

Vahid Behzadan, Roman V. Yampolskiy, Arslan Munir


I really liked this paper: it seems like it took a real stab at trying to align today’s recommender systems, and might have made substantial progress.

I am somewhat dubious of the use of causal graphical models here: if you create a model with some conditional independence relation that then doesn’t hold in practice, your model can have some pretty bad inferences. This actually happened: when they only modeled the relationships based on Twitter’s UI elements, and in particular did not model the dependence of SLO on clicks, they were getting bad results, where clicking on a post was interpreted as evidence that the post was _not_ valuable.

As the paper mentions, we can drop the causal interpretation of the Bayes net. This lets us draw edges between more nodes in B in order to make our model more expressive and partially prevent this sort of misspecification, while also letting us express more complex relationships. For example, I think (but am not sure) that with their current graph, liking and retweeting a post would be treated as independent sources of evidence. If we drew an edge between them, you would gain the ability for the model to learn that if a user likes _and_ retweets a post, that is more than the sum of the contributions of liking and retweeting separately. Note that you still can’t connect everything to the anchor A, because they require that A has no children, and if you add parents to A, those must then be estimated by the heuristic method above that is probably not very good. So you still need to model conditional independence for A, and this may get significantly harder the more complex B becomes.

This also makes sense given the motivation: since the idea is to have information about value “flow through the anchor” to the variables in B, it seems like you shouldn’t need to worry too much about the relationships between variables in B, and it would be fine to model arbitrarily complex relationships between them. Besides dropping the causal interpretation and adding lots of edges in B, another option is to add many more features to the graph: for example, perhaps you also want to include the number of likes that the tweet gets overall, or whether or not it is currently the weekend. You do need to make sure your model does not become so expressive that it can now overfit to the dataset, e.g. “tweets that are posted on Sep 24 that get exactly 38756 likes will not be valuable to Alice”. However, the datasets they are using are presumably huge, and the current graphical models are tiny (16 nodes with low degree), so we can increase it at least somewhat before we get to that point.

Is this paper relevant to alignment of superintelligent AI systems, the topic of this newsletter? I don’t think it is that relevant, since it seems like the main technique involves us effectively hardcoding our knowledge (of the anchor variable) into the objective, in a way that makes sense for recommender systems but probably would not work for more general systems. I’m highlighting it anyway because I think it is particularly novel and interesting, it seems to solve an analog of the alignment problem for existing AI systems, and it is part of a research lineage that I do think will be relevant for alignment: how to create good objectives for AI systems. Even if you only care about alignment of superintelligent systems, it seems worth following the techniques used today and the problems that come up in their application, as the lessons learned may continue to be relevant when intelligence is scaled up.

From Optimizing Engagement to Measuring Value

This paper takes a stab at creating a better objective for existing recommender systems than engagement, in a way that could be applied at existing companies like Twitter. The basic approach is to treat the variable to be optimized (user value) as a latent variable, and use probabilistic inference to infer how likely it is that a particular recommendation was valuable.

Usually a major challenge with such an approach is specifying the _observation model_: how the observed data is caused by the latent variable. In the case of Twitter, this would require you to answer questions like, “if the user does not value a tweet, how likely is a user to hit the like button anyway?” This is a hard question to answer, since perhaps users like tweets in order to stop conversations, or because they are addicting at the moment but are not actually valuable, etc.

One simple heuristic is to take two datasets where we know one dataset has more valuable recommendations than the other. Differences in user behavior between these datasets can then be assumed to be correlations with value. The authors provide a quantitative method for inferring the observation model from such datasets, which I won’t go into here since it is primarily a heuristic baseline. One obvious problem is that if the “better” dataset was produced by optimizing (say) clicks, then the clicks may have increased for reasons other than improved value, but this heuristic approach will attribute the entire increase to improved value.

How can we do better? The key insight of this paper is that if you have a bunch of historical data, then you can get a lot of mileage by identifying an _anchor_: a type of feedback that when given provides unequivocal evidence of the latent value. On Twitter, this is taken to be the “See Less Often” (SLO) button: if this is clicked, then we know with effective certainty that this was not valuable, regardless of any other actions the user took. The connection between value and other behaviors such as liking a tweet can then be inferred by looking at the connection between those behaviors and the anchor, which we can estimate from historical data.

Formally, the authors assume access to a graph describing the relationships between the various possible behaviors (almost all of which have the latent value V as a parent). One of these is identified as the anchor node A, for which P(V = 1 | A = 1) is assumed to be known and independent of all other variables. However, P(V = 1 | A = 0) is not independent of other variables: intuitively, if the SLO button is _not_ clicked, then we need to fall back to looking at other variables to estimate value.

The authors then show that under some reasonable assumptions on the anchor variable, if you have a dataset of historical data to estimate P(A, B) (where B consists of all the other tracked behaviors), then instead of specifying observation models P(B | V) for all behaviors, you only need to specify observation models for the parents of A, that is P(parents(A) | V). Everything else is uniquely determined, allowing us to calculate our final objective P(V | A, B). (There are algorithmic details on how to do this efficiently; see the paper for details.) In this case, they use the heuristic method outlined above to estimate P(parents(A) | V).

They unfortunately don’t have a great way to evaluate their method: they clearly can’t evaluate it by seeing if it leads to higher clicks, since the whole point was to move away from clicks as an optimization target. (I assume a user study on Twitter was infeasible.) Their primary form of evaluation is to run the model and report the learned probabilities, and show that they seem reasonable, whereas those output by a Naive Bayes model do not.


Even though I often feel like an <@optimist@>(@Conversation with Rohin Shah@) about incentives towards alignment, even I was surprised to see the amount of effort that it seems Facebook has put into trying to align its recommendation algorithm with well-being. To the extent that the recommendation algorithm is still primarily harmful (which might be true or false, idk), this suggests to me that it might just be really hard to give good recommendations given the sparse feedback you get. Of course, there are more cynical explanations, e.g. Facebook just wants to look like they care about well-being, but if they really cared they could do way better. I lean towards the first explanation, but it's very hard to distinguish between these hypotheses.

While this post claimed that narrow value alignment should be easier than AGI alignment, I'm actually not so sure. With AGI alignment, you have the really powerful assumption that the AI system you are trying to align is _intelligent_: this could plausibly help a lot. For example, maybe the recommender systems that Facebook is using are just incapable of predicting what will and won't improve human well-being, in which case narrow alignment is doomed. This wouldn't be the case with an AGI (depending on your definition of AGI) -- it should be capable of doing at least as well as humans do. The challenge is in ensuring that the AI systems are actually <@motivated@>(@Clarifying "AI Alignment"@) to do so, not whether they are capable of doing so; with narrow alignment you need to solve both problems.

Aligning AI to Human Values means Picking the Right Metrics

There has been a lot of attention recently on the flaws of recommender systems, especially when optimizing for simple metrics like engagement -- an example of what we might call "narrow value alignment". This post reconstructs how Facebook and YouTube have been incorporating better metrics into their algorithms from 2015 and 2017 respectively. For example, Facebook found that academic research suggested that well-being was improved by "meaningful social interactions", but worsened by passive consumption of content. As a result, they changed the metric for the recommendation algorithm to better track this. How did they measure it? It seems that they simply asked a survey of thousands of people what the most meaningful content was (both on and off Facebook), and used this to train a model to predict "meaningful interactions". They estimated that this resulted in a 5% decrease in time spent on Facebook, at least in the short term. The story with YouTube is similar, though sparser on details (and it's not clear if there was input from end users in YouTube's case).

The author then contrasts this sort of narrow value alignment with AGI alignment. His main take is that narrow alignment should be easier to address, since we can learn from how existing systems behave in the real world, and the insights we gain may be critical for AGI alignment. I'll end with a quote from the conclusion: "My argument is not so much that one should use AI to optimize for well-being. Rather, we live in a world where large-scale optimization is already happening. We can choose not to evaluate or adjust these systems, but there is little reason to imagine that ignorance and inaction would be better."


I really liked this paper as an example of an agenda about how to change recommender systems. It didn’t rely on armchair speculation or anecdotes to determine what problems exist and what changes should be made, and it didn’t just assume that polarization must be bad and instead considered both costs and benefits. The focus on “making conflict healthy” makes a lot of sense to me. I especially appreciated the emphasis on a strategy for evaluating particular changes, rather than pushing for a specific change; specific changes all too often fail once you test them at scale in the real world.

Designing Recommender Systems to Depolarize

This paper agrees with the post above that “available evidence mostly disfavors the hypothesis that recommender systems are driving polarization through selective exposure, aka ‘filter bubbles’ or ‘echo chambers’”. Nonetheless, social media is a huge part of today’s society, and even if it isn’t actively driving polarization, we can ask whether there are interventions that would decrease polarization. That is the focus of this paper.

It isn’t clear that we _should_ intervene to decrease polarization for a number of reasons:

1. Users may not want to be “depolarized”.
2. Polarization may lead to increased accountability as each side keeps close watch of politicians on the other side. Indeed, in 1950 people used to worry that we weren’t polarized _enough_.
3. More broadly, we often learn through conflict: you are less likely to see past your own biases if there isn’t someone else pointing out what they are. To the extent that depolarization removes conflict, it may be harmful.

Nonetheless, polarization also has some clear downsides:

1. It can cause gridlock, preventing effective governance.
2. It erodes norms against conflict escalation, leading to outrageous behavior and potentially violence.
3. At current levels, it has effects on all spheres of life, many of which are negative (e.g. harm to relationships across partisan lines).

Indeed, it is plausible that _most_ situations of extreme conflict were caused in part by a positive feedback loop involving polarization; this suggests that reducing polarization could be a very effective method to prevent conflict escalation. Ultimately, it is the _escalation_ and _violence_ that we want to prevent. Thus we should be aiming for interventions that don’t eliminate conflict (as we saw before, conflict is useful), but rather transform it into a version that doesn’t lead to escalation and norm violations.

For this purpose we mostly care about _affective polarization_, which tells you how people feel about “the other side”. (In contrast, issue polarization is less central, since we don’t want to discourage disagreement on issues.) The paper’s central recommendation is for companies to measure affective polarization and use this as a validation metric to help decide whether a particular change to a recommender system should be deployed or not. (This matches current practice at tech companies, where there are a few high-level metrics that managers use to decide what to deploy, and importantly, the algorithms do _not_ explicitly optimize for those metrics.) Alternatively, we could use reinforcement learning to optimize the affective polarization metric, but in this case we would need to continuously evaluate the _metric_ in order to ensure we don’t fall prey to Goodhart effects.

The paper also discusses potential interventions that could reduce polarization. However, it cautions that they are based on theory or studies with limited ecological validity, and ideally should be checked via the metric suggested above. Nonetheless, here they are:

1. **Removing polarizing content.** In this case, the polarizing content doesn’t make it to the recommender system at all. This can be done quite well when there are human moderators embedded within a community, but is much harder to do at scale in an automated way.

2. **Changing recommendations.** The most common suggestion in this category is to increase the diversity of recommended content (i.e. go outside the “filter bubble”). This can succeed, though usually only has a modest effect, and can sometimes have the opposite effect of increasing polarization. A second option is to penalize content for incivility: studies have shown that incivility tends to increase affective polarization. Actively promoting civil content could also help. That being said, it is not clear that we want to prevent people from ever raising their voice.

3. **Changing presentation.** The interface to the content can be changed to promote better interactions. For example, when the Facebook “like” button was replaced by a “respect” button, people were more likely to “respect” comments they disagreed with.