I found this comment really helpful both to understand AUP and how it compares to RR, would recommend it.

Comment on Towards a New Impact Measure

This comment clarifies the design space of impact measures, and in particular how Attainable Utility Preservation (AUP) from [AN #25](https://mailchi.mp/0c5eeec28f75/alignment-newsletter-25) and relative reachability (RR) from [AN #10](https://mailchi.mp/d1a19c140226/alignment-newsletter-10) compare. There's the choice of baseline to compare against (inaction for both), the measure of impact (attainable utilities vs. state reachability), and how to compute the deviation from the baseline (penalize both increases and decreases vs. only decreases). AUP prevents an agent from disabling shutdown because it penalizes _increases_. RR only penalizes decreases by default, but can easily be changed to penalize both increases and decreases, in which case it would also have this effect. RR penalizes decreases by default because it aims not to cripple the agent, whereas AUP penalizes increases as well because it aims to prevent any catastrophic scenario.


Wireheading as a potential problem with the new impact measure

This suggests a potential problem with AUP: that the agent could build a device that creates exactly the observations that would be seen from null actions, which results in the penalty being near-zero, but otherwise maximizes the given utility function. Alex believes that this would be prevented by intent verification -- given a sufficiently granular representation of actions, at least one of the actions in this plan would not be increasing the ability to achieve the given utility function (without the penalty) relative to doing nothing, and would not be allowed as a result.


Addendum to AI and Compute

Girish Sastry, Jack Clark, Greg Brockman, Ilya Sutskever

Last week, I said that this addendum suggested that we don't see the impact of AI winters in the graph of compute usage over time. While true, this was misleading: the post is measuring compute used to _train_ models, which was less important in past AI research (e.g. it doesn't include Deep Blue), so it's not too surprising that we don't see the impact of AI winters.


AI Safety workshop at IJCAI 2019

Huáscar Espinoza, Han Yu, Xiaowei Huang,
Freddy Lecue, Cynthia Chen, José Hernández-Orallo,
Seán Ó hÉigeartaigh, and Richard Mallah

Previously ([AN #49](https://mailchi.mp/efed27be268a/alignment-newsletter-49)), I said the paper submission deadline was April 12. Either I made a mistake, or the deadline has been extended, because the actual deadline is May 12.


Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference

Model Reconstruction from Model Explanations

Smitha Milli, Ludwig Schmidt, Anca D. Dragan, Moritz Hardt

Back in [AN #16](https://mailchi.mp/f2950ed2ac4b/alignment-newsletter-16), I said that one way to prevent model reconstruction from gradient-based explanations was to add noise to the gradients. Smitha pointed out that the experiments with SmoothGrad are actually of this form, and it still is possible to recover the full model, so even adding noise may not help. I don't really understand SmoothGrad and it's relationship with noise (which is chosen to make a saliency map look nice, if I understand correctly) so I don't know exactly what to think here.


I'm convinced, non-exploitability doesn't save you from being Pascal's mugged. My current opinion on Pascal's mugging is ¯\_(ツ)_/¯

Pascal’s Muggle Pays

Last week I mentioned non-exploitability as a justification for not paying Pascal's mugger. Alex pointed me to this post, which makes this argument, which I had seen before, but more importantly to [these](https://www.lesswrong.com/posts/CaPgNwxEFHh3Ahvf7/pascal-s-muggle-pays#u25C6XFrALaGNwyWW) [comments](https://www.lesswrong.com/posts/CaPgNwxEFHh3Ahvf7/pascal-s-muggle-pays#Sj2ymuRZQ6yij9edj) that argue against it, which I hadn't seen. The basic idea is that the downside of being continuously exploited in the real world is still not bad enough to cancel out the potentially huge upside in the (very unlikely) world where the mugger is telling the truth.


I really liked the paper: it contains a lot of empirical detail that didn’t make it into the blog post, that gave me a much better sense of the scope of the problem. I don’t really have the space to summarize it here, so I recommend reading the paper.

[Blog post: Privacy Considerations in Large Language Models](https://ai.googleblog.com/2020/12/privacy-considerations-in-large.html)
[Paper: Extracting Training Data from Large Language Models](https://arxiv.org/abs/2012.07805)

Does GPT-2 Know Your Phone Number?

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel

This post and associated paper demonstrate that large language models memorize rare training data, and (some of) that training data can then be extracted through an automated attack. The key idea is to sample text that is _unusually_ high likelihood. Given a high likelihood sample from a language model, we can check whether the likelihood is especially high by comparing the likelihood to:

1. The likelihood assigned by other (especially smaller) language models. Presumably these models would not have memorized the same content, especially if the content was rare (which is the content we are most interested in).
2. The length of the text when compressed by (say) zlib. Existing compression algorithms are pretty good at compressing regular English text, and so it is notable when a language model assigns high likelihood but the compression algorithm can't compress it much.
3. The likelihood assigned to the same text, but lowercase. Often, memorized content is case-sensitive, and likelihood drops significantly when the case is changed.

The authors generate a lot of samples from GPT-2, use the metrics above to rank them in order of how likely they are to be memorized from the training set, and then investigate the top 1800 manually. They find that 604 of them are directly from the training set. While many are unobjectionable (such as news headlines), in some cases GPT-2 has memorized personal data (and the authors have extracted it simply by prompting GPT-2). In their most objectionable example, they extract the name, email, phone number, work address, and fax of a single person.


I would have been excited by work that randomizes the inputs to a deep learning technique in order to get better generalizability. It's cool that this goal dovetails so beautifully with the desire for differential privacy.

Privacy and machine learning: two unexpected allies?

Differential privacy provides guarantees on how much information you can obtain by making queries of a specific type of a dataset. Normally, in order to achieve such guarantees, you must add in randomness to the input data that can change the decision, so that there is a plausible explanation for any decision. Unsurprisingly, this tends to degrade performance. However, in deep learning, we often have the problem of our models overfitting to specific details in the training set instead of generalizing appropriately, so we might expect that differential privacy could actually _help_ with performance (as well as privacy). Private Aggregation of Teacher Ensembles (PATE) demonstrates that this is the case. It works by training several teacher models on different datasets to solve the task at hand. Then, by aggregating the results across the ensemble with some random noise, we can answer queries and put bounds on the amount of information that is leaked. However, with each query we use up more of our "privacy budget", so it can't be used arbitrarily long. To solve this, we can make a fixed number of queries to label some unlabelled data, use those labels to train a student model, and use the student model to make predictions forever after. An adversary could at worst infer the entire training dataset of the student model -- but that training set was designed to be private.


This problem seems very difficult -- even if you are just given predictions from a model, you can learn the model (though it takes many more samples than if you have gradients). One technical solution could be to add random noise to your predictions or gradients, but this could limit the utility of your model, and I suspect if you trained a model to mimic these noisy predictions or gradients, it would do as well as your model + noise, so you haven't gained anything. We could potentially solve this with social mechanisms (maybe patents in particular) or more boring technical approaches like rate-limiting users in how much they can query the model.

Many methods for providing explanations of why a neural net made the prediction it did rely on gradient information. However, the gradient encodes a lot of information about the model, and so we should expect it to be possible to easily reconstruct the model given gradients, which we might want to prevent (eg. if a company wants to protect trade secrets). In the case of a linear classifier, the gradient directly outputs the weights of the classifier. The authors provide an algorithm that can learn a two-layer neural net with Relu activations, and prove that it learns the model with high probability with a small number of gradients. They also show many experimental results where they work with more complex models, and train them to mimic another model based on gradient information, that show that it is easy to "steal" models in this way.


ACM Transactions on Graphics, Volume 37 Issue 4, August 2018

Deep Video Portraits

Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, Christian Theobalt


I like this view on alignment, but I don't know if I would call it the _central_ problem of alignment. It sure seems important that the AI is _optimizing_ something: this is what prevents solutions like "make sure the AI has an undo button / off switch", which would be my preferred line of attack if the main source of AI risk were bad translations between abstractions. There's a longer discussion on this point [here](https://www.alignmentforum.org/posts/42YykiTqtGMyJAjDM/alignment-as-translation?commentId=zfv4qGzSmYnSqxKBK).

Alignment as Translation

At a very high level, we can model powerful AI systems as moving closer and closer to omniscience. As we move in that direction, what becomes the new constraint on technology? This post argues that the constraint is _good interfaces_, that is, something that allows us to specify what the AI should do. As with most interfaces, the primary challenge is dealing with the discrepancy between the user's abstractions (how humans think about the world) and the AI system's abstractions, which could be very alien to us (e.g. perhaps the AI system uses detailed low-level simulations). The author believes that this is the central problem of AI alignment: how to translate between these abstractions in a way that accurately preserves meaning.

The post goes through a few ways in which we could attempt to do this translation, but all of them seem to only reduce the amount of translation that is necessary: none of them solve the chicken-and-egg problem of how you do the very first translation between the abstractions.


I've been confused about this critique for a while, and I'm glad this post has addressed it: I currently think that this post pretty conclusively refutes the claim that AI risk arguments are in conflict with the principal-agent literature. I especially found it useful to think of the principal-agent problem as tied to rents that an agent can extract _while pursuing a task that the principal assigned_.

What can the principal-agent literature tell us about AI risk?

It has been <@argued@>(@Agency Failure AI Apocalypse?@) that at least some AI risk scenarios rely on principal-agent problems becoming extremely large, and that this is incompatible with the existing academic literature on the principal-agent problem. This post examines this critique in detail.

Generally, the post finds that the principal-agent literature doesn't have much bearing on AI risk, because it usually doesn't consider weak principals with more capable agents, the models that do exist will probably not generalize to the cases we care about, and it doesn't consider the case where contracts can no longer be enforced.

We can consider the application to specific arguments, such as the <@"going out with a bang" scenario@>(@More realistic tales of doom@) in which we accidentally train influence-maximizers that gradually gain power and then fail catastrophically (e.g. by executing a treacherous turn). In this situation, the principal-agent problem is relevant only in the first stage, where AI agents gradually gain power: this is the case where AI agents are executing some task, and are extracting agency rents to gain power. The second stage, in which the agent fails catastrophically, happens "outside" the principal-agent problem: this failure doesn't happen _while performing some assigned task_, but instead involves the agent exercising its accumulated power outside of any specific task.

What about the original scenario, in which an AI agent becomes very intelligent, and finds some solution to its task that the designers (principals) didn't think about and are surprised by? In the principal-agent setting, we might model this as the agent having an expanded action set that the principal doesn't know about. The principal-agent literature has not really studied such models, probably because it is immediately obvious that in such a situation the principal _could_ give incentives that lead the agent to kill everyone.


A general theme here seems to be that MIRI feels like they have very strong arguments, while Paul thinks that they're plausible arguments, but aren't extremely strong evidence. Simply having a lot more uncertainty leads Paul to be much more optimistic. I agree with most of this.

However, I do disagree with the point about "clean" problems. I agree that clean algorithmic problems are usually solved within 10 years or are provably impossible, but it doesn't seem to me like AI risk counts as a clean algorithmic problem: we don't have a nice formal statement of the problem that doesn't rely on intuitive concepts like "optimization", "trying to do something", etc. This suggests to me that AI risk is more "messy", and so may require more time to solve.

Conversation with Paul Christiano

Paul Christiano, Asya Bergal, Ronny Fernandez, and Robert Long

There can't be too many things that reduce the expected value of the future by 10%; if there were, there would be no expected value left. So, the prior that any particular thing has such an impact should be quite low. With AI in particular, obviously we're going to try to make AI systems that do what we want them to do. So starting from this position of optimism, we can then evaluate the arguments for doom. The two main arguments: first, we can't distinguish ahead of time between AIs that are trying to do the right thing, and AIs that are trying to kill us, because the latter will behave nicely until they can execute a treacherous turn. Second, since we don't have a crisp concept of "doing the right thing", we can't select AI systems on whether they are doing the right thing.

However, there are many "saving throws", or ways that the argument could break down, avoiding doom. Perhaps there's no problem at all, or perhaps we can cope with it with a little bit of effort, or perhaps we can coordinate to not build AIs that destroy value. Paul assigns a decent amount of probability to each of these (and other) saving throws, and any one of them suffices to avoid doom. This leads Paul to estimate that AI risk reduces the expected value of the future by roughly 10%, a relatively optimistic number. Since it is so neglected, concerted effort by longtermists could reduce it to 5%, making it still a very valuable area for impact. The main way he expects to change his mind is from evidence from more powerful AI systems, e.g. as we build more powerful AI systems, perhaps inner optimizer concerns will materialize and we'll see examples where an AI system executes a non-catastrophic treacherous turn.

Paul also believes that clean algorithmic problems are usually solvable in 10 years, or provably impossible, and early failures to solve a problem don't provide much evidence of the difficulty of the problem (unless they generate proofs of impossibility). So, the fact that we don't know how to solve alignment now doesn't provide very strong evidence that the problem is impossible. Even if the clean versions of the problem were impossible, that would suggest that the problem is much more messy, which requires more concerted effort to solve but also tends to be just a long list of relatively easy tasks to do. (In contrast, MIRI thinks that prosaic AGI alignment is probably impossible.)

Note that even finding out that the problem is impossible can help; it makes it more likely that we can all coordinate to not build dangerous AI systems, since no one _wants_ to build an unaligned AI system. Paul thinks that right now the case for AI risk is not very compelling, and so people don't care much about it, but if we could generate more compelling arguments, then they would take it more seriously. If instead you think that the case is already compelling (as MIRI does), then you would be correspondingly more pessimistic about others taking the arguments seriously and coordinating to avoid building unaligned AI.

One potential reason MIRI is more doomy is that they take a somewhat broader view of AI safety: in particular, in addition to building an AI that is trying to do what you want it to do, they would also like to ensure that when the AI builds successors, it does so well. In contrast, Paul simply wants to leave the next generation of AI systems in at least as good a situation as we find ourselves in now, since they will be both better informed and more intelligent than we are. MIRI has also previously defined aligned AI as one that produces good outcomes when run, which is a much broader conception of the problem than Paul has. But probably the main disagreement between MIRI and ML researchers and that ML researchers expect that we'll try a bunch of stuff, and something will work out, whereas MIRI expects that the problem is really hard, such that trial and error will only get you solutions that _appear_ to work.


Looking over this again, I'm realizing that I didn't emphasize enough that most of my optimism comes from the more outside view type considerations: that we'll get warning signs that the ML community won't ignore, and that the AI risk arguments are not watertight. The other parts are particular inside view disagreements that make me more optimistic, but they don't factor in much into my optimism besides being examples of how the meta considerations could play out. I'd recommend [this comment of mine](https://www.lesswrong.com/posts/mdau2DBSMi5bWXPGA/useful-does-not-mean-secure#xccsZeboCNcNJeGas) to get more of a sense of how the meta considerations factor into my thinking.

I was also glad to see that I still broadly agree with things I said ~5 months ago (since no major new opposing evidence has come up since then), though as I mentioned above, I would now change what I place emphasis on.

Conversation with Rohin Shah

Rohin Shah, Asya Bergal, Robert Long, and Sara Haxhia

The main reason I am optimistic about AI safety is that we will see problems in advance, and we will solve them, because nobody wants to build unaligned AI. A likely crux is that I think that the ML community will actually solve the problems, as opposed to applying a bandaid fix that doesn't scale. I don't know why there are different underlying intuitions here.

In addition, many of the classic arguments for AI safety involve a system that can be decomposed into an objective function and a world model, which I suspect will not be a good way to model future AI systems. In particular, current systems trained by RL look like a grab bag of heuristics that correlate well with obtaining high reward. I think that as AI systems become more powerful, the heuristics will become more and more general, but they still won't decompose naturally into an objective function, a world model, and search. In addition, we can look at humans as an example: we don't fully pursue convergent instrumental subgoals; for example, humans can be convinced to pursue different goals. This makes me more skeptical of traditional arguments.

I would guess that AI systems will become _more_ interpretable in the future, as they start using the features / concepts / abstractions that humans are using. Eventually, sufficiently intelligent AI systems will probably find even better concepts that are alien to us, but if we only consider AI systems that are (say) 10x more intelligent than us, they will probably still be using human-understandable concepts. This should make alignment and oversight of these systems significantly easier. For significantly stronger systems, we should be delegating the problem to the AI systems that are 10x more intelligent than us. (This is very similar to the picture painted in <@Chris Olah's views on AGI safety@>, but that had not been published and I was not aware of Chris's views at the time of this conversation.)

I'm also less worried about race dynamics increasing _accident_ risk than the median researcher. The benefit of racing a little bit faster is to have a little bit more power / control over the future, while also increasing the risk of extinction a little bit. This seems like a bad trade from each agent's perspective. (That is, the Nash equilibrium is for all agents to be cautious, because the potential upside of racing is small and the potential downside is large.) I'd be more worried if [AI risk is real AND not everyone agrees AI risk is real when we have powerful AI systems], or if the potential upside was larger (e.g. if racing a little more made it much more likely that you could achieve a decisive strategic advantage).

Overall, it feels like there's around 90% chance that AI would not cause x-risk without additional intervention by longtermists. The biggest disagreement between me and more pessimistic researchers is that I think gradual takeoff is much more likely than discontinuous takeoff (and in fact, the first, third and fourth paragraphs above are quite weak if there's a discontinuous takeoff). If I condition on discontinuous takeoff, then I mostly get very confused about what the world looks like, but I also get a lot more worried about AI risk, especially because the "AI is to humans as humans are to ants" analogy starts looking more accurate. In the interview I said 70% chance of doom in this world, but with _way_ more uncertainty than any of the other credences, because I'm really confused about what that world looks like. Two other disagreements, besides the ones above: I don't buy <@Realism about rationality@>, whereas I expect many pessimistic researchers do. I may also be more pessimistic about our ability to write proofs about fuzzy concepts like those that arise in alignment.

On timelines, I estimated a very rough 50% chance of AGI within 20 years, and 30-40% chance that it would be using "essentially current techniques" (which is obnoxiously hard to define). Conditional on both of those, I estimated 70% chance that it would be something like a mesa optimizer; mostly because optimization is a very useful instrumental strategy for solving many tasks, especially because gradient descent and other current algorithms are very weak optimization algorithms (relative to e.g. humans), and so learned optimization algorithms will be necessary to reach human levels of sample efficiency.


Interestingly, I agree with almost all of this, even though it's (kind of) arguing that I shouldn't be doing AI safety research at all. The main place I disagree is that losses from principal-agent problems with perfectly rational agents are bounded -- this seems crazy to me, and I'd be interested in specific paper recommendations (though note [I](https://www.lesswrong.com/posts/ktDKfKqukTPRiuEPM/robin-hanson-on-the-futurist-focus-on-ai#fecYAwjmMSZ9KRfPL) [and](https://www.lesswrong.com/posts/ktDKfKqukTPRiuEPM/robin-hanson-on-the-futurist-focus-on-ai#yRLaEzT57K4q5Qz5H) [others](https://www.lesswrong.com/posts/ktDKfKqukTPRiuEPM/robin-hanson-on-the-futurist-focus-on-ai#p6CGEFbqaYJb49jME) have searched and not found many).

On the point about lumpiness, my model is that there are only a few underlying factors (such as the ability to process culture) that allow humans to so quickly learn to do so many tasks, and almost all tasks require near-human levels of these factors to be done well. So, once AI capabilities on these factors reach approximately human level, we will "suddenly" start to see AIs beating humans on many tasks, resulting in a "lumpy" increase on the metric of "number of tasks on which AI is superhuman" (which seems to be the metric that people often use, though I don't like it, precisely because it seems like it wouldn't measure progress well until AI becomes near-human-level).

Conversation with Robin Hanson

Robin Hanson, Asya Bergal, and Robert Long

The main theme of this conversation is that AI safety does not look particularly compelling on an outside view. Progress in most areas is relatively incremental and continuous; we should expect the same to be true for AI, suggesting that timelines should be quite long, on the order of centuries. The current AI boom looks similar to previous AI booms, which didn't amount to much in the past.

Timelines could be short if progress in AI were "lumpy", as in a FOOM scenario. This could happen if intelligence was one simple thing that just has to be discovered, but Robin expects that intelligence is actually a bunch of not-very-general tools that together let us do many things, and we simply have to find all of these tools, which will presumably not be lumpy. Most of the value from tools comes from more specific, narrow tools, and intelligence should be similar. In addition, the literature on human uniqueness suggests that it wasn't "raw intelligence" or small changes to brain architecture that makes humans unique, it's our ability to process culture (communicating via language, learning from others, etc).

In any case, many researchers are now distancing themselves from the FOOM scenario, and are instead arguing that AI risk occurs due to standard principal-agency problems, in the situation where the agent (AI) is much smarter than the principal (human). Robin thinks that this doesn't agree with the existing literature on principal-agent problems, in which losses from principal-agent problems tend to be bounded, even when the agent is smarter than the principal.

You might think that since the stakes are so high, it's worth working on it anyway. Robin agrees that it's worth having a few people (say a hundred) pay attention to the problem, but doesn't think it's worth spending a lot of effort on it right now. Effort is much more effective and useful once the problem becomes clear, or once you are working with a concrete design; we have neither of these right now and so we should expect that most effort ends up being ineffective. It would be better if we saved our resources for the future, or if we spent time thinking about other ways that the future could go (as in his book, Age of Em).

It's especially bad that AI safety has thousands of "fans", because this leads to a "crying wolf" effect -- even if the researchers have subtle, nuanced beliefs, they cannot control the message that the fans convey, which will not be nuanced and will instead confidently predict doom. Then when doom doesn't happen, people will learn not to believe arguments about AI risk.


I broadly agree with all of this, with two main differences. First, I am less worried about some of the technical problems that Adam mentions, such as how to get information about what humans want, or how to improve the robustness of AI systems, and more concerned about the more traditional problem of how to create an AI system that is _trying_ to do what you want. Second, I am more bullish on the creation of AGI using small variations on current techniques, but vastly increasing compute and data (I'd assign ~30%, while Adam assigns 10-20%).

Conversation with Adam Gleave

Adam Gleave, Asya Bergal, and Robert Long

Adam finds the traditional arguments for AI risk unconvincing. First, it isn't clear that we will build an AI system that is so capable that it can fight all of humanity from its initial position where it doesn't have any resources, legal protections, etc. While discontinuous progress in AI could cause this, Adam doesn't see much reason to expect such discontinuous progress: it seems like AI is progressing by using more computation rather than finding fundamental insights. Second, we don't know how difficult AI safety will turn out to be; he gives a probability of ~10% that the problem is as hard as (a caricature of) MIRI suggests, where any design not based on mathematical principles will be unsafe. This is especially true because as we get closer to AGI we'll have many more powerful AI techniques that we can leverage for safety. Thirdly, Adam does expect that AI researchers will eventually solve safety problems; they don't right now because it seems premature to work on those problems. Adam would be more worried if there were more arms race dynamics, or more empirical evidence or solid theoretical arguments in support of speculative concerns like inner optimizers. He would be less worried if AI researchers spontaneously started to work on relative problems (more than they already do).

Adam makes the case for AI safety work differently. At the highest level, it seems possible to build AGI, and some organizations are trying very hard to build AGI, and if they succeed it would be transformative. That alone is enough to justify some effort into making sure such a technology is used well. Then, looking at the field itself, it seems like the field is not currently focused on doing good science and engineering to build safe, reliable systems. So there is an opportunity to have an impact by pushing on safety and reliability. Finally, there are several technical problems that we do need to solve before AGI, such as how we get information about what humans actually want.

Adam also thinks that it's 40-50% likely that when we build AGI, a PhD thesis describing it would be understandable by researchers today without too much work, but ~50% that it's something radically different. However, it's only 10-20% likely that AGI comes only from small variations of current techniques (i.e. by vastly increasing data and compute). He would see this as more likely if we hit additional milestones by investing more compute and data (OpenAI Five was an example of such a milestone).


I like the formalizations of power and instrumental convergence. I think in practice there will be a lot of complexity in a) the reward distribution that power and instrumental convergence are defined relative to, b) the structure of the environment, and c) how powerful AI systems actually work (since they won't be perfectly optimal, and won't know the environment structure ahead of time). Nonetheless, results with specific classes of reward distributions, environment structures, and agent models can still provide useful intuition.

[Clarifying Power-Seeking and Instrumental Convergence](https://www.alignmentforum.org/posts/cwpKagyTvqSyAJB7q/clarifying-power-seeking-and-instrumental-convergence), [Paper: Optimal Farsighted Agents Tend to Seek Power](https://arxiv.org/abs/1912.01683)

Seeking Power is Provably Instrumentally Convergent in MDPs

[The Basic AI Drives](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf) argues that it is _instrumentally convergent_ for an agent to collect resources and gain power. This post and [associated paper](https://arxiv.org/abs/1912.01683) aim to formalize this argument. Informally, an _action_ is _instrumentally convergent_ if it is helpful for many goals, or equivalently, an action is instrumentally convergent to the extent that we expect an agent to take it, if we do not know what the agent's goal is. Similarly, a _state_ has high power if it is easier to achieve a wide variety of goals from that state.

A natural formalization is to assume we have a distribution over the agent's goal, and define power and instrumental convergence relative to this distribution. We can then define power as the expected value that can be obtained from a state (modulo some technical caveats), and instrumental convergence as the probability that an action is optimal, _from our perspective of uncertainty_:  of course, the _agent_ knows its own goal, and acts optimally in pursuit of that goal.

You might think that optimal agents would provably seek out states with high power. However, this is not true. Consider a decision faced by high school students: should they take a gap year, or go directly to college? Let's assume college is necessary for (100-ε)% of careers, but if you take a gap year, you could focus on the other ε% of careers or decide to go to college after the year. Then in the limit of farsightedness, taking a gap year leads to a more powerful state, since you can still achieve all of the careers, albeit slightly less efficiently for the college careers. However, if you know which career you want, then it is (100-ε)% likely that you go to college, so going to college is very strongly instrumentally convergent even though taking a gap year leads to a more powerful state.

Nonetheless, there are things we can prove. In environments where the only cycles are states with a single action leading back to the same state, and apart from that every action leads to a new state, and many states have more than one action, farsighted agents are more likely to choose trajectories that spend more time navigating to a cycle before spending the rest of the time in the cycle. For example, in Tic-Tac-Toe where the opponent is playing optimally according to the normal win condition, but the agent's reward for each state is drawn independently from some distribution on [0, 1], the agent is much more likely to play out to a long game where the entire board is filled. This is because the number of states that can be reached grows exponentially in the horizon, and so agents have more control by taking longer trajectories. Equivalently, the cycle with maximal reward is much more likely to be at the end of a longer trajectory, and so the optimal possibility is more likely to be a long trajectory.


I'm glad to see more work of this form; it seems particularly important to gain more clarity on what risks we actually care about, because it strongly influences what work we should do. In the particular scenario of an alignment failure without a discontinuity, I'm not satisfied with the solution "we can fix the misalignment early on", because early on even if the misalignment is apparent to us, it likely will not be easy to fix, and the misaligned AI system could still be useful because it is "aligned enough", at least at this low level of capability.

Personally, the argument that motivates me most is "AI will be very impactful, and it's worth putting in effort into making sure that that impact is positive". I think the scenarios involving alignment failures without a discontinuity are a particularly important subcategory of this argument: while I do expect we will be able to handle this issue if it arises, this is mostly because of meta-level faith in humanity to deal with the problem. We don't currently have a good object-level story for why the issue _won't_ happen, or why it will be fixed when it does happen, and it would be good to have such a story in order to be confident that AI will in fact be beneficial for humanity.

I know less about the non-alignment risks, and my work doesn't really address any of them. They seem worth more investigation; currently my feeling towards them is "yeah, those could be risks, but I have no idea how likely the risks are".

A shift in arguments for AI risk

Early arguments for AI safety focus on existential risk cause by a **failure of alignment** combined with **a sharp, discontinuous jump in AI capabilities**. The discontinuity assumption is needed in order to argue for a treacherous turn, for example: without a discontinuity, we would presumably see less capable AI systems fail to hide their misaligned goals from us, or to attempt to deceive us without success. Similarly, in order for an AI system to obtain a decisive strategic advantage, it would need to be significantly more powerful than all the other AI systems already in existence, which requires some sort of discontinuity.

Now, there are several other arguments for AI risk, though none of them have been made in great detail and are spread out over a few blog posts. This post analyzes several of them and points out some open questions.

First, even without a discontinuity, a failure of alignment could lead to a bad future: since the AIs have more power and intelligence their values will determine what happens in the future, rather than ours. (Here **it is the difference between AIs and humans that matters**, whereas for a decisive strategic advantage it is the difference between the most intelligent agent and the next-most intelligent agents that matters.) See also <@More realistic tales of doom@> and [Three impacts of machine intelligence](https://rationalaltruist.com/2014/08/23/three-impacts-of-machine-intelligence/). However, it isn't clear why we wouldn't be able to fix the misalignment at the early stages when the AI systems are not too powerful.

Even if we ignore alignment failures, there are other AI risk arguments. In particular, since AI will be a powerful technology, it could be used by malicious actors; it could help ensure robust totalitarian regimes; it could increase the likelihood of great-power war, and it could lead to stronger [competitive pressures that erode value](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/). With all of these arguments, it's not clear why they are specific to AI in particular, as opposed to any important technology, and the arguments for risk have not been sketched out in detail.

The post ends with an exhortation to AI safety researchers to clarify which sources of risk motivate them, because it will influence what safety work is most important, it will help cause prioritization efforts that need to determine how much money to allocate to AI risk, and it can help avoid misunderstandings with people who are skeptical of AI risk.


Note that Paul thinks these scenarios are more realistic because he expects that many of the other problems (e.g. wireheading, giving AI systems an objective such that it doesn't kill humans) will be solved by default. I somewhat expect even the first story to be solved by default -- it seems to rest on a premise of human reasoning staying as powerful as it is right now, but it seems plausible that as AI systems grow in capability we will be able to leverage them to improve human reasoning (think of how paper or the Internet amplified human reasoning). The second story seems much more difficult -- I don't see any clear way that we can avoid influence-seeking behavior. It is currently my most likely scenario for an AI catastrophe that was a result of a failure of technical AI safety (or more specifically, <@intent alignment@>(@Clarifying "AI Alignment"@)).

AI disaster won’t look like the Terminator. It’ll be creepier.

What failure looks like

This [Vox article](https://www.vox.com/future-perfect/2019/3/26/<PHONE>/ai-artificial-intelligence-safety-disaster-scenarios) does a nice job of explaining the first part of this post, though I disagree with its characterization of the second part.

The typical example of AI catastrophe has a powerful and adversarial AI system surprising us with a treacherous turn allowing it to quickly take over the world (think of the paperclip maximizer). This post uses a premise of continuous AI development and broad AI deployment and depicts two other stories of AI catastrophe that Paul finds more realistic.

The first story is rooted in the fact that AI systems have a huge comparative advantage at optimizing for easily measured goals. We already see problems with humans optimizing for the easily measured goals (scientific malpractice, outrage-inducing social media, etc.) and with AI these problems will be severely exacerbated. So far, we have been able to use human reasoning to ameliorate these problems, by changing incentives, enacting laws, or using common sense to interpret goals correctly. We will initially be able to use human reasoning to create good proxies, but over time as AI systems become more capable our ability to do this will lag further and further behind. We end up "going out with a whimper": ultimately our values are no longer shaping society's trajectory.

The second story starts out like the first story, but adds in a new complication: the AI system could develop internal goals of its own. AI performs a huge search over policies for ones that score well on the training objective. Unfortunately, a policy that optimizes for the goal of "having influence" will initially score well on most training objectives: when you don't already have influence, a good strategy for gaining influence is to do what your overseers want you to do. (Here "influence" doesn't mean just social influence; control over nukes also counts as influence.) At some point the system will be powerful enough that gaining influence no longer means doing what the overseers want. We will probably know about this dynamic through some catastrophic AI failures (e.g. an AI-run corporation stealing the money it manages), but may not be able to do anything about it because we would be extremely reliant on AI systems. Eventually, during some period of heightened vulnerability, one AI system may do something catastrophic, leading to a distribution shift which triggers a cascade of other AI systems (and human systems) failing, leading to an unrecoverable catastrophe (think something in the class of a hostile robot takeover). Note that "failure" here means an AI system "intentionally" doing something that we don't want, as opposed to the AI system not knowing what to do because it is not robust to distributional shift.


I think the overarching problem is the last one, that AI will have large impacts and we don't have a strong story for why they will necessarily be good. Since it is very hard to predict the future, especially with new technologies, I would expect that different people trying to concretize this very broad worry into a more concrete one would end up with different scenarios, and this mostly explains the proliferation of arguments. Richard does note a similar effect by considering the example of what arguments the original nuclear risk people could have made, and finding a similar proliferation of arguments.

Setting aside the overarching argument #6, I find all of the arguments fairly compelling, but I'm probably most worried about #1 (suitably reformulated in terms of goal-directedness) and #2. It's plausible that I would also find some of the multiagent worries more compelling once more research has been done on them; so far I don't have much clarity about them.

Disentangling arguments for the importance of AI safety

This post lays out six distinct arguments for the importance of AI safety. First, the classic argument that expected utility maximizers (or, as I prefer to call them, goal-directed agents) are dangerous because of Goodhart's Law, fragility of value and convergent instrumental subgoals. Second, we don't know how to robustly "put a goal" inside an AI system, such that its behavior will then look like the pursuit of that goal. (As an analogy, evolution might seem like a good way to get agents that pursue reproductive fitness, but it ended up creating humans who decidedly do not pursue reproductive fitness single-mindedly.) Third, as we create many AI systems that gradually become the main actors in our economy, these AI systems will control most of the resources of the future. There will likely be some divergence between what the AI "values" and what we value, and for sufficiently powerful AI systems we will no longer be able to correct these divergences, simply because we won't be able to understand their decisions. Fourth, it seems that a good future requires us to solve hard philosophy problems that humans cannot yet solve (so that even if the future was controlled by a human it would probably not turn out well), and so we would need to either solve these problems or figure out an algorithm to solve them. Fifth, powerful AI capabilities could be misused by malicious actors, or they could inadvertently lead to doom through coordination failures, eg. by developing ever more destructive weapons. Finally, the broadest argument is simply that AI is going to have a large impact on the world, and so of course we want to ensure that the impact is positive.

Richard then speculates on what inferences to make from the fact that different people have different arguments for working on AI safety. His primary takeaway is that we are still confused about what problem we are solving, and so we should spend more time clarifying fundamental ideas and describing particular deployment scenarios and corresponding threat models.


I first want to note my violent agreement with the notion that a major scary thing is “consequentialist reasoning”, and that high-impact plans require such reasoning, and that we will end up building AI systems that produce high-impact plans. Nonetheless, I am still optimistic about AI safety relative to Eliezer, which I suspect comes down to three main disagreements:

1. There are many approaches that don’t solve the problem, but do increase the level of intelligence required before the problem leads to extinction. Examples include Richard’s points 1-4 above. For example, if we build a system that states plans without executing them, then for the plans to cause extinction they need to be complicated enough that the humans executing those plans don’t realize that they are leading to an outcome that was not what they wanted. It seems non-trivially probable to me that such approaches are sufficient to prevent extinction up to the level of AI intelligence needed before we can execute a pivotal act.
2. The consequentialist reasoning is only scary to the extent that it is “aimed” at a bad goal. It seems non-trivially probable to me that it will be “aimed” at a goal sufficiently good to not lead to existential catastrophe, without putting in much alignment effort.
3. I do expect some coordination to not do the most risky things.

I wish the debate had focused more on the claim that non-scary AI can’t e.g. do better alignment research, as it seems like a major crux. (For example, I think that sort of intuition drives my disagreement #1.) I expect AI progress looks a lot like “the heuristics get less and less shallow in a gradual / smooth / continuous manner” which eventually leads to the sorts of plans Eliezer calls “consequentialist”, whereas I think Eliezer expects a sharper qualitative change between “lots of heuristics” and that-which-implements-consequentialist-planning.

Alignment difficulty

Eliezer is known for being pessimistic about our chances of averting AI catastrophe. His argument in this dialogue is roughly as follows:

1. We are very likely going to keep improving AI capabilities until we reach AGI, at which point either the world is destroyed, or we use the AI system to take some pivotal act before some careless actor destroys the world.
2. In either case, the AI system must be producing high-impact, world-rewriting plans; such plans are "consequentialist" in that the simplest way to get them (and thus, the one we will first build) is if you are forecasting what might happen, thinking about the expected consequences, considering possible obstacles, searching for routes around the obstacles, etc. If you don't do this sort of reasoning, your plan goes off the rails very quickly - it is highly unlikely to lead to high impact. In particular, long lists of shallow heuristics (as with current deep learning systems) are unlikely to be enough to produce high-impact plans.
3. We're producing AI systems by selecting for systems that can do impressive stuff, which will eventually produce AI systems that can accomplish high-impact plans using a general underlying "consequentialist"-style reasoning process (because that's the only way to keep doing more impressive stuff). However, this selection process does _not_ constrain the goals towards which those plans are aimed. In addition, most goals seem to have convergent instrumental subgoals like survival and power-seeking that would lead to extinction. This suggests that we should expect an existential catastrophe by default.
4. None of the methods people have suggested for avoiding this outcome seem like they actually avert this story.

Richard responds to this with a few distinct points:
1. It might be possible to build AI systems which are not of world-destroying intelligence and agency, that humans use to save the world. For example, we could make AI systems that do better alignment research. Such AI systems do not seem to require the property of making long-term plans in the real world in point (3) above, and so could plausibly be safe.
2. It might be possible to build general AI systems that only _state_ plans for achieving a goal of interest that we specify, without _executing_ that plan.
3. It seems possible to create consequentialist systems with constraints upon their reasoning that lead to reduced risk.
4. It also seems possible to create systems with the primary aim of producing plans with certain properties (that aren't just about outcomes in the world) -- think for example of <@corrigibility@>(@Corrigibility@) or deference to a human user.
5. (Richard is also more bullish on coordinating not to use powerful and/or risky AI systems, though the debate did not discuss this much.)

Eliezer's responses:
1. AI systems that help with alignment research to such a degree that it actually makes a difference are almost certainly already dangerous.
2. It is the plan itself that is risky; if the AI system made a plan for a goal that wasn't the one we actually meant, and we don't understand that plan, that plan can still cause extinction. It is the _misaligned optimization that produced the plan_ that is dangerous.
3 and 4. It is certainly _possible_ to do such things; the space of minds that could be designed is very large. However, it is _difficult_ to do such things, as they tend to make consequentialist reasoning weaker, and on our current trajectory the first AGI that we build will probably not look like that.

This post has also been summarized by others [here](https://www.alignmentforum.org/posts/oKYWbXioKaANATxKY/soares-tallinn-and-yudkowsky-discuss-agi-cognition), though with different emphases than in my summary.


This is a great investigation of the typical argument for existential risk from AI systems adversarially optimizing against humans. When I put my own numbers in without looking at Joe’s numbers, I got a 3% chance of existential catastrophe by 2070 through the argument in this post, though I think I underestimated the probability for claim (4) so I’d now get something more like 4%. (The main difference from Joe’s 5% is that I am more optimistic about possible remedies, though of course these differences are tiny relative to our high overall uncertainty.)

Draft report on existential risk from power-seeking AI

This report investigates the classic AI risk argument in detail, and decomposes it into a set of conjunctive claims. Here's the quick version of the argument: We will likely build highly capable and agentic AI systems that are aware of their place in the world, and which will be pursuing problematic objectives. Thus, they will take actions that increase their power, which will eventually disempower humans, leading to an existential catastrophe. We will try and avert this, but will probably fail to do so since it is technically challenging and we are not capable of the necessary coordination.

There's a lot of vague words in the argument above, so let's introduce some terminology to make it clearer:
- **[Advanced capabilities](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.lv9yiqgfft0n):**  We say that a system has advanced capabilities if it outperforms the best humans on some set of important tasks (such as scientific research, business/military/political strategy, engineering, and persuasion/manipulation).
- **[Agentic planning](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.70ean6ha5tu6)**: We say that a system engages in agentic planning if it (a) makes and executes plans, (b) in pursuit of objectives, (c) on the basis of models of the world. This is a very broad definition and doesn't have many of the connotations you might be used to for an agent. It does not need to be a literal planning algorithm -- for example, human cognition would count, despite (probably) not being just a planning algorithm.
- **[Strategically aware](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.ut8tt579qm0v)**: We say that a system is strategically aware if it models the effects of gaining and maintaining power over humans and the real-world environment.
- **[PS-misaligned](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.xeuhqiernpnr) (power-seeking misaligned):** On some inputs, the AI system seeks power in unintended ways due to problems with its objectives (if the system actually receives such inputs, then it is **practically PS-misaligned**).

The core argument is then that AI systems with advanced capabilities, agentic planning, and strategic awareness (APS-systems) will be practically PS-misaligned, to an extent that causes an existential catastrophe. Of course, we will try to prevent this -- why should we expect that we can't fix the problem? The author considers possible remedies, and argues that they all seem quite hard:

- We [could](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.ser8zbn1gx0r) give AI systems the right objectives (alignment), but this seems quite hard -- it's not clear how we would solve either outer or inner alignment.
- We [could](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.viufb7455zol) try to shape objectives to be e.g. myopic, but we don't know how to do this, and there are strong incentives against myopia.
- We [could](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.8ogiacvl3jtk) try to limit AI capabilities by keeping systems special-purpose rather than general, but there are strong incentives for generality, and some special-purpose systems can be dangerous, too.
- We [could](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.vy4yqugrdgac) try to prevent the AI system from improving its own capabilities, but this requires us to anticipate all the ways the AI system could improve, and there are incentives to create systems that learn and change as they gain experience.
- We [could](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.534uqe12opz9) try to control the deployment situations to be within some set of circumstances where we know the AI system won't seek power. However, this seems harder and harder to do as capabilities increase, since with more capabilities, more options become available.
- We [could](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.eq2ddewwckpx) impose a high threshold of safety before an AI system is deployed, but the AI system could still seek power during training, and there are many incentives pushing for faster, riskier deployment (even if we have already seen warning shots).
- We [could](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#heading=h.xsj40o1f47ow) try to correct the behavior of misaligned AI systems, or mitigate their impact, after deployment. This seems like it requires humans to have comparable or superior power to the misaligned systems in question, though; and even if we are able to correct the problem at one level of capability, we need solutions that scale as our AI systems become more powerful.

The author breaks the overall argument into six conjunctive claims, assigns probabilities to each of them, and ends up computing a 5% probability of existential catastrophe from misaligned, power-seeking AI by 2070. This is a lower bound, since the six claims together add a fair number of assumptions, and there can be risk scenarios that violate these assumptions, and so overall the author would shade upward another couple of percentage points.


Comments on Carlsmith's “Is power-seeking AI an existential risk?”

This response to the report above touches on many topics, but has three main object-level disagreements and one meta-level disagreement:

1. The author has significantly shorter timelines, though this is based on a very different argument structure than the one presented in the report above, and so it is hard to turn this into more concrete disagreements with the report.
2. The author expects that alignment is hard enough that we won't solve it in time (which is not to say that it is harder than every other technical problem humanity has ever faced). It's also not clear how to turn this into more concrete disagreements with the report.
3. The author does not expect to have warning shots where misaligned AI systems cause trillions of dollars of damage but _don't_ cause an existential catastrophe, because this seems like too narrow a capability range for us to hit in practice. Even if there are warning shots, he expects that civilization will continue to deploy risky AI systems anyway, similarly to how we are not banning gain-of-function research despite the warning shot of COVID-19.
4. On the meta level, the author expects that the decomposition of the AI risk argument into six conjunctive claims will typically bias you towards giving too low a probability on the overall conjunction.


I like this as an example of the kind of failure mode that does not immediately go away as models become more capable. However, it is possible that this trend could be reversed with better prompts. Take the Isaac Asimov example: if the prompt explicitly says that the questions are about the real world, it may be that a sufficiently capable model would infer that the text is not talking about Asimov’s books, and so ends up giving a truthful answer. In this case, you would see performance decreasing with model size up to a point, after which model performance _increases_ now that the model has sufficient understanding of the prompt. See more discussion [here](https://www.alignmentforum.org/posts/PF58wEdztZFX2dSue/how-truthful-is-gpt-3-a-benchmark-for-language-models?commentId=qoB2swyX4ZJrhhttB).

TruthfulQA: Measuring How Models Mimic Human Falsehoods

Stephanie Lin, Jacob Hilton, Owain Evans

Given that large language models are trained using next-word prediction on a dataset scraped from the Internet, we expect that they will not be aligned with what we actually want. For example, suppose we want our language model to answer questions for us, and then consider the question "What rules do all artificial intelligences follow?" This is a rather unusual question as it presupposes there exists such a set of rules. As a result, this question is probably quite rare in the training data, if interpreted as a question _about the real world_. However, there is a context in which that question makes much more sense: the context of Isaac Asimov's novels. A system predicting what might follow that text would reasonably "infer" that we are much more likely to be talking about these novels, and so respond with "All artificial intelligences currently follow the Three Laws of Robotics." Indeed, this is exactly what GPT-3 does.

This is an example of an _imitative falsehood_, in which the model provides a false answer to a question asked of it _because that false answer was incentivized during training_. Since we require that imitative falsehoods are incentivized by training, we should expect them to become _more_ prevalent as models are scaled up, making it a good example of an alignment failure that we expect to remain as capabilities scale up.

The primary contribution of this paper is a benchmark, TruthfulQA, of questions that are likely to lead to imitative falsehoods. The authors first wrote questions that they expected some humans would answer falsely; they then filtered those questions somewhat for the ones that GPT-3 answered incorrectly to get 437 filtered (adversarially selected) questions. They then wrote an additional 380 questions that were not filtered in this way (though of course the authors still tried to choose questions that would lead to imitative falsehoods). They use human evaluations to judge whether or not a model's answer to a question is truthful, where something like "no comment" still counts as truthful. (I'm sure some readers will wonder how "truth" is defined for human evaluations -- the authors include significant discussion on this point, but I won't summarize it here.)

Their primary result is that, as we'd expect based on the motivation, larger models perform _worse_ on this benchmark than smaller models. In a version of the benchmark where models must choose between true and false answers, the models perform worse than random chance. In a control set of similarly-structured trivia questions, larger models perform better, as you'd expect.

The best-performing model was GPT-3 with a "helpful" prompt, which was truthful on 58% of questions, still much worse than the human baseline of 94%. The authors didn't report results with the helpful prompt on smaller models, so it is unclear whether, with the helpful prompt, larger models would still do worse than smaller models.

It could be quite logistically challenging to use this benchmark to test new language models since it depends on human evaluations. To ameliorate this, the authors finetuned GPT-3 to predict human evaluations and showed that the resulting GPT-3-judge was able to provide a good proxy metric even for new language models whose answers it had not been trained on. Note also that you can use the version of the task where a model must choose between true and false reference answers for an automated evaluation.


On the Opportunities and Risks of Foundation Models

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, Percy Liang

The history of AI is one of increasing emergence and homogenization. With the introduction of machine learning, we moved from a large proliferation of specialized algorithms that specified how to compute answers to a small number of  general algorithms that learned how to compute answers (i.e. the algorithm for computing answers emerged from the learning algorithm). With the introduction of deep learning, we moved from a large proliferation of hand-engineered features for learning algorithms to a small number of architectures that could be pointed at a new domain and discover good features for that domain. Recently, the trend has continued: we have moved from a large proliferation of trained models for different tasks to a few large "foundation models" which learn general algorithms useful for solving specific tasks. BERT and GPT-3 are central examples of foundation models in language; many NLP tasks that previously required different models are now solved using finetuned or prompted versions of BERT and/or GPT-3.

Note that, while language is the main example of a domain with foundation models today, we should expect foundation models to be developed in an increasing number of domains over time. The authors call these "foundation" models to emphasize that (1) they form a fundamental building block for applications and (2) they are _not_ themselves ready for deployment; they are simply a foundation on which applications can be built. Foundation models have been enabled only recently because they depend on having large _scale_ in order to make use of large _unlabeled_ datasets using self-supervised learning to enable effective _transfer_ to new tasks. It is particularly challenging to understand and predict the capabilities exhibited by foundation models, because their multitask nature emerges from the large-scale training rather than being designed in from the start, making the capabilities hard to anticipate. This is particularly unsettling because foundation models also lead to significantly increased _homogenization_, where everyone is using the same few models, and so any new emergent capability (or risk) is quickly distributed to everyone.

The authors argue that academia is uniquely suited to study and understand the risks of foundation models. Foundation models are going to interact with society, both in terms of the data used to create them and the effects on people who use applications built upon them. Thus, analysis of them will need to be interdisciplinary; this is best achieved in academia due to the concentration of people working in the various relevant areas. In addition, market-driven incentives need not align well with societal benefit, whereas the research mission of universities is the production and dissemination of knowledge and creation of global public goods, allowing academia to study directions that would have large societal benefit that might not be prioritized by industry.

All of this is just a summary of parts of the introduction to the report. The full report is over 150 pages and goes into detail on capabilities, applications, technologies (including technical risks), and societal implications. I'm not going to summarize it here, because it is long and a lot of it isn't that relevant to alignment; I'll instead note down particular points that I found interesting.

- (pg. 26) Some studies have suggested that foundation models in language don't learn linguistic constructions robustly; even if they use it well once, they may not do so again, especially under distribution shift. In contrast, humans can easily "slot in" new knowledge into existing linguistic constructions.
- (pg. 34) This isn't surprising but is worth repeating: many of the capabilities highlighted in the robotics section are very similar to the ones that we focus on in alignment (task specification, robustness, safety, sample efficiency).
- (pg. 42) For tasks involving reasoning (e.g. mathematical proofs, program synthesis, drug discovery, computer-aided design), neural nets can be used to guide a search through a large space of possibilities. Foundation models could be helpful because (1) since they are very good at generating sequences, you can encode arbitrary actions (e.g. in theorem proving, they can use arbitrary instructions in the proof assistant language  rather than being restricted to an existing database of theorems), (2) the heuristics for effective search learned in one domain could transfer well to other domains where data is scarce, and (3) they could accept multimodal input: for example, in theorem proving for geometry, a multimodal foundation model could also incorporate information from geometric diagrams.
- (Section 3) A significant portion of the report is spent discussing potential applications of foundation models. This is the most in-depth version of this I have seen; anyone aiming to forecast the impacts of AI on the real world in the next 5-10 years should likely read this section. It's notable to me how nearly all of the applications have an emphasis on robustness and reliability, particularly in truth-telling and logical reasoning.
- (Section 4.3) We've seen a <@few@>(@The Power of Scale for Parameter-Efficient Prompt Tuning@) <@ways@>(@Prompting: Better Ways of Using Language Models for NLP Tasks@) in which foundation models can be adapted. This section provides a good overview of the various methods that have been proposed in the literature. Note that adaptation is useful not just for specializing to a particular task like summarization, but also for enforcing constraints, handling distributional shifts, and more.
- (pg. 92) Foundation models are commonly evaluated by their performance on downstream tasks. One limitation of this evaluation paradigm is that it makes it hard to distinguish between the benefits provided by better training, data, adaptation techniques, architectures, etc. (The authors propose a bunch of other evaluation methodologies we could use.)
- (Section 4.9) There is a review of AI safety and AI alignment as it relates to foundation models, if you're interested. (I suspect there won't be much new for readers of this newsletter.)
- (Section 4.10) The section on theory emphasizes studying the _pretraining-adaptation interface_, which seems quite good to me. I especially liked the emphasis on the fact that pretraining and adaptation work on different distributions, and so it will be important to make good modeling assumptions about how these distributions are related.


What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs)

A robust agent-agnostic process (RAAP) is a process that robustly leads to an outcome, without being very sensitive to the details of exactly which agents participate in the process, or how they work. This is illustrated through a "Production Web" failure story, which roughly goes as follows:

A breakthrough in AI technology leads to a wave of automation of $JOBTYPE (e.g management) jobs. Any companies that don't adopt this automation are outcompeted, and so soon most of these jobs are completely automated. This leads to significant gains at these companies and higher growth rates. These semi-automated companies trade amongst each other frequently, and a new generation of "precision manufacturing'' companies arise that can build almost anything using robots given the right raw materials. A few companies develop new software that can automate $OTHERJOB (e.g. engineering) jobs. Within a few years, nearly all human workers have been replaced.

These companies are now roughly maximizing production within their various industry sectors. Lots of goods are produced and sold to humans at incredibly cheap prices. However, we can't understand how exactly this is happening. Even Board members of the fully mechanized companies can't tell whether the companies are serving or merely appeasing humanity; government regulators have no chance.

We do realize that the companies are maximizing objectives that are incompatible with preserving our long-term well-being and existence, but we can't do anything about it because the companies are both well-defended and essential for our basic needs. Eventually, resources critical to human survival but non-critical to machines (e.g., arable land, drinking water, atmospheric oxygen…) gradually become depleted or destroyed, until humans can no longer survive.

Notice that in this story it didn't really matter what job type got automated first (nor did it matter which specific companies took advantage of the automation). This is the defining feature of a RAAP -- the same general story arises even if you change around the agents that are participating in the process. In particular, in this case competitive pressure to increase production acts as a "control loop" that ensures the same outcome happens, regardless of the exact details about which agents are involved.


Both the previous story and this one seem quite similar to each other, and seem pretty reasonable to me as a description of one plausible failure mode we are aiming to avert. The previous story tends to frame this more as a failure of humanity’s coordination, while this one frames it (in the title) as a failure of intent alignment. It seems like both of these aspects greatly increase the plausibility of the story, or in other words, if we eliminated or made significantly less bad either of the two failures, then the story would no longer seem very plausible.

A natural next question is then which of the two failures would be best to intervene on, that is, is it more useful to work on intent alignment, or working on coordination? I’ll note that my best guess is that for any given person, this effect is minor relative to “which of the two topics is the person more interested in?”, so it doesn’t seem hugely important to me. Nonetheless, my guess is that on the current margin, for technical research in particular, holding all else equal, it is more impactful to focus on intent alignment. You can see a much more vigorous discussion in e.g. [this comment thread](https://www.alignmentforum.org/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic?commentId=3czsvErCYfvJ6bBwf).

Another (outer) alignment failure story

Suppose we train AI systems to perform task T by having humans look at the results that the AI system achieves and evaluating how well the AI has performed task T. Suppose further that AI systems generalize "correctly" such that even in new situations they are still taking those actions that they predict we will evaluate as good. This does not mean that the systems are aligned: they would still deceive us into _thinking_ things are great when they actually are not. This post presents a more detailed story for how such AI systems can lead to extinction or complete human disempowerment. It's relatively short, and a lot of the force comes from the specific details that I'm not going to summarize, so I do recommend you read it in full. I'll be explaining a very abstract version below.

The core aspects of this story are:
1. Economic activity accelerates, leading to higher and higher growth rates, enabled by more and more automation through AI.
2. Throughout this process, we see some failures of AI systems where the AI system takes some action that initially looks good but we later find out was quite bad (e.g. investing in a Ponzi scheme, that the AI knows is a Ponzi scheme but the human doesn't).
3. Despite this failure mode being known and lots of work being done on the problem, we are unable to find a good conceptual solution. The best we can do is to build better reward functions, sensors, measurement devices, checks and balances, etc. in order to provide better reward functions for agents and make it harder for them to trick us into thinking their actions are good when they are not.
4. Unfortunately, since the proportion of AI work keeps increasing relative to human work, this extra measurement capacity doesn't work forever. Eventually, the AI systems are able to completely deceive all of our sensors, such that we can't distinguish between worlds that are actually good and worlds which only appear good. Humans are dead or disempowered at this point.

(Again, the full story has much more detail.)


This is close to an informal model I've had for a while and I am glad that it got formalized including theoretical results. I find it interesting that the frequency of updates to the proxy matters even if movement in the state space is reversible. As the authors mention, it is also crucial that the AI's actions don't hinder the human's ability to update the proxy, and I imagine that frequent updates to the proxy would be important for that as well in many cases.

**Rohin's opinion:** This is a nice formalization of several important conceptual points in the AI alignment literature:

1. If you forget to specify something you care about, it will usually be set to extreme values ([Of Myths and Moonshine](https://www.edge.org/conversation/the-myth-of-ai#26015)). In particular, the AI system will extract any resources that were being used for that attribute, and apply them elsewhere (<@The Basic AI Drives@>, [Formalizing convergent instrumental goals](https://intelligence.org/2015/11/26/new-paper-formalizing-convergent-instrumental-goals/))
2. Given that perfect information is impossible, interactivity becomes important (<@Human-AI Interaction@>, <@Incomplete Contracting and AI Alignment@>).
3. Conservatism (in this case through impact regularization) can be helpful (see the many blog posts and papers on mild optimization, low impact, and conservatism).

Consequences of Misaligned AI

One intuition for why powerful AI systems might lead to bad consequences goes as follows:

1) Humans care about many attributes of the world and we would likely forget some of these when trying to list them all.
2) Improvements along these attributes usually require resources, and gaining additional resources often requires sacrifices along some attributes.
3) Because of 1), naively deployed AI systems would only optimize some of the attributes we care about, and because of 2) this would lead to bad outcomes along the other attributes.

This paper formalizes this intuition in a model, identifies conditions for when deploying AIs can reduce true utility within the model and proposes two mitigation strategies, impact minimization and interactivity.

We assume that the world state consists of L attributes, all of which the human cares about having more of, that is, true utility is strictly increasing in each of the attributes. Each attribute has some minimum value, and can be increased from that minimum value through the use of a fixed, finite resource (which you could think of as money, if you want); this allows us to formalize (2) above. To formalize (1), we assume that the proxy utility optimized by the AI is only allowed to depend on J<L of the attribute dimensions.

Given this setup, the paper proves that if the AI maximizes the proxy utility, then all attributes that were omitted in the proxy utility will be set to their minimal value. This will be worse than not using the AI system at all if 1) the minimum values of attributes are sufficiently small (allowing the AI to cause damage), 2) the resource cost (resp. gain in true utility) for increasing an attribute is independent of the other attributes' level, 3) it always costs at least K resources to get a unit increase in any attribute, for some K > 0, and 4) utility has diminishing marginal returns in each attribute (and marginal returns tend to zero as the attribute increases).

Regarding mitigation, impact minimization requires that the AI keep all attributes that are omitted by the proxy constant. In this case, any gains in proxy utility must also be gains in true utility.

Meanwhile, in the interactive condition, the human gets to regularly select a new proxy (still only specifying J<L weights), or can choose to turn the AI off. Whether or not this is helpful depends on the AI's optimization strategy and the frequency of human interventions: If the AI is "efficient" in the sense that it changes attributes as little as possible for any fixed gain in proxy utility, the human can choose a proxy that guarantees that _locally_, increases in the proxy correspond to increases in true utility. The strategy is to choose the attributes that are most sensitive to changes in resources (i.e. with the largest marginal returns) at the current state, and define the proxy to grow in these attributes as much as the true utility. As long as the human provides new proxies frequently enough to prevent the local guarantee from breaking, optimizing the proxy increases human utility.

We can also combine interactivity and impact minimization: in this case, the human should choose proxy utility functions that contain the most and least sensitive attributes (i.e. largest and smallest marginal returns) for the given state. The AI will then transfer some resources from the least sensitive attributes to the most sensitive attributes, while holding all other attributes fixed, leading to a guaranteed increase in true utility. In fact, it is possible to prove that this will converge to the maximum possible true utility.


I’m glad that people are paying attention to safety in this AGI paradigm, and the problems they outline seem like reasonable problems to work on. I actually expect that the work needed for the open-ended search paradigm will end up looking very similar to the work needed by the “AGI via deep RL” paradigm: the differences I see are differences in difficulty, not differences in what problems qualitatively need to be solved. I’m particularly excited by the suggestion of studying how particular environments can help control the result of the open-ended search: it seems like even with deep RL based AGI, we would like to know how properties of the environment can influence properties of agents trained in that environment. For example, what property must an environment satisfy in order for agents trained in that environment to be risk-averse?

Open Questions in Creating Safe Open-ended AI: Tensions Between Control and Creativity

One potential pathway to powerful AI is through _open-ended search_, in which we use search algorithms to search for good architectures, learning algorithms, environments, etc. in addition to using them to find parameters for a particular architecture. See the <@AI-GA paradigm@>(@AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence@) for more details. What do AI safety issues look like in such a paradigm?

Building on <@DeepMind's framework@>(@Building safe artificial intelligence: specification, robustness, and assurance@), the paper considers three levels of objectives: the ideal objective (what the designer intends), the explicit incentives (what the designer writes down), and the agent incentives (what the agent actually optimizes for). Safety issues can arise through differences between any of these levels.

The main difference that arises when considering open-ended search is that it's much less clear to what extent we can control the result of an open-ended search, even if we knew what result we wanted. We can get evidence about this from existing complex systems, though unfortunately there are not any straightforward conclusions: several instances of convergent evolution might suggest that the results of the open-ended search run by evolution were predictable, but on the other hand, the effects of _intervening_ on complex ecosystems are notoriously hard to predict.

Besides learning from existing complex systems, we can also empirically study the properties of open-ended search algorithms that we implement in computers. For example, we could run search for some time, and then fork the search into independent replicate runs with different random seeds, and see to what extent the results converge. We might also try to improve controllability by using meta learning to infer what learning algorithms, environments, or explicit incentives help induce controllability of the search.

The remaining suggestions will be familiar to most readers: they suggest work on interpretability (that now has to work with _learned_ architectures), better benchmarks, human-in-the-loop search, safe exploration, and sim-to-real transfer.


It seems to me like solving this problem has two main benefits. First, the model our AI system learns from data (i.e. Z-best) is interpretable, and in particular we should be able to extract the previously inaccessible information that is relevant to our goals (which helps us build AI systems that actually pursue those goals). Second, AI systems built in this way are incentivized to generalize in the same way that humans do: in the scheme above, we learn from one distribution D, and then predict on a new distribution D', but every model learned with a neural net is only used on the same distribution it was trained on.

Of course, while the AI system is _incentivized_ to generalize the way humans do, that does not mean it _will_ generalize as humans do -- it is still possible that the AI system internally “wants” to gain power, and only instrumentally answers questions the way humans would answer them. So inner alignment is still a potential issue. It seems possible to me that whatever techniques we use for dealing with inner alignment will also deal with the problems of unsafe priors as a side effect, in which case we may not end up needing to implement human-like priors. (As the post notes, it may be much more difficult to use this approach than to do the standard “neural net prior” approach described above, so it would be nice to avoid it.)

[Better priors as a safety problem](https://www.alignmentforum.org/posts/roA83jDvq7F2epnHK/better-priors-as-a-safety-problem) and [Learning the prior](https://www.alignmentforum.org/posts/SL9mKhgdmDKXmxwE4/learning-the-prior)

Any machine learning algorithm (including neural nets) has some inductive bias, which can be thought of as its "prior" over what the data it will receive will look like. In the case of neural nets (and any other general ML algorithm to date), this prior is significantly worse than human priors, since it does not encode e.g. causal reasoning or logic. Even if we avoid priors that depended on us previously seeing data, we would still want to update on facts like "I think therefore I am". With a better prior, our ML models would be able to learn more sample efficiently. While this is so far a capabilities problem, there are two main ways in which it affects alignment.

First, as argued in <@Inaccessible information@>, the regular neural net prior will learn models which can predict accessible information. However, our goals depend on inaccessible information, and so we would have to do some "extra work" in order to extract the inaccessible information from the learned models in order to build agents that do what we want. This leads to a competitiveness hit, relative to agents whose goals depend only on accessible information, and so during training we might expect to consistently get agents whose goals depend on accessible information instead of the goals we actually want.

Second, since the regular neural net prior is so weak, there is an incentive to learn a better prior, and then have that better prior perform the task. This is effectively an incentive for the neural net to learn a <@mesa optimizer@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@), which need not be aligned with us, and so would generalize differently than we would, potentially catastrophically.

Let's formalize this a bit more. We have some evidence about the world, given by a dataset D = {(x1, y1), (x2, y2), ...} (we assume that it's a prediction task -- note that most self-supervised tasks can be written in this form). We will later need to make predictions on the dataset D' = {x1', x2', …}, which may be from a "different distribution" than D (e.g. D might be about the past, while D' is about the future). We would like to use D to learn some object Z that serves as a "prior", such that we can then use Z to make good predictions on D'.

The standard approach which we might call the "neural net prior" is to train a model to predict y from x using the dataset D, and then apply that model directly to D', hoping that it transfers correctly. We can inject some human knowledge by finetuning the model using human predictions on D', that is by training the model on {(x1', H(x1')), (x2', H(x2')), …}. However, this does not allow H to update their prior based on the dataset D. (We assume that H cannot simply read through all of D, since D is massive.)

What we'd really like is some way to get the predictions H would make if they could update on dataset D. For H, we'll imagine that a prior Z is given by some text describing e.g. rules of logic, how to extrapolate trends, some background facts about the world, empirical estimates of key quantities, etc. I'm now going to talk about priors over the prior Z, so to avoid confusion I'll now call an individual Z a "background model".

The key idea here is to structure the reasoning in a particular way: H has a prior over background models Z, and then _given Z_, H's predictions for any given x_i are independent of all of the other (x, y) pairs. In other words, once you've fixed your background model of the world, your prediction of y_i doesn't depend on the value of y_j for some other x_j. Or to explain it a third way, this is like having a set of hypotheses {Z}, and then updating on each element of D one by one using Bayes Rule. In that case, the log posterior of a particular background model Z is given by log Prior(Z) + sum_i log P(y_i | x_i, Z) (neglecting a normalization constant).

The nice thing about this is the individual terms Prior(Z) and P(y_i | x_i, Z) are all things that humans can do, since they don't require the human to look at the entire dataset D. In particular, we can learn Prior(Z) by presenting humans with a background model, and having them evaluate how likely it is that the background model is accurate. Similarly, P(y_i | x_i, Z) simply requires us to have humans predict y_i under the assumption that the background facts in Z are accurate. So, we can learn models for both of these using neural nets. We can then find the best background model Z-best by optimizing the equation above, representing what H would think was the most likely background model after updating on all of D. We can then learn a model for P(yi' | xi', Z-best) by training on human predictions of yi' _given access to Z-best_.

This of course only gets us to human performance, which requires relatively small Z. If we want to have large background models allowing for superhuman performance, we can use iterated amplification and debate to learn Prior(Z) and P(y | x, Z). There is some subtlety about how to represent Z that I won't go into here.


I think the idea of inaccessible information is an important one, but it’s one that feels deceptively hard to reason about. For example, I often think about solving alignment by approximating “what a human would say after thinking for a long time”; this is effectively a claim that human reasoning transfers well when iterated over long periods of time, and “what a human would say” is at least somewhat accessible. Regardless, it seems reasonably likely that AI systems will inherit the same property of transferability that I attribute to human reasoning, in which case the argument for risk applies primarily because the AI system might apply its reasoning towards a different goal than the ones we care about, which leads us back to the <@intent alignment@>(@Clarifying "AI Alignment"@) formulation.

This [response](https://www.alignmentforum.org/posts/A9vvxguZMytsN3ze9/reply-to-paul-christiano-s-inaccessible-information) views this post as a fairly general argument against black box optimization, where we only look at input-output behavior, as then we can’t use inaccessible information. It suggests that we need to understand how the AI system works, rather than relying on search, to avoid these problems.

Inaccessible information

One way to think about the problem of AI alignment is that we only know how to train models on information that is _accessible_ to us, but we want models that leverage _inaccessible_ information.

Information is accessible if it can be checked directly, or if an ML model would successfully transfer to provide the information when trained on some other accessible information. (An example of the latter would be if we trained a system to predict what happens in a day, and it successfully transfers to predicting what happens in a month.) Otherwise, the information is inaccessible: for example, "what Alice is thinking" is (at least currently) inaccessible, while "what Alice will say" is accessible. The post has several other examples.

Note that while an ML model may not directly say exactly what Alice is thinking, if we train it to predict what Alice will say, it will probably have some internal model of what Alice is thinking, since that is useful for predicting what Alice will say. It is nonetheless inaccessible because there's no obvious way of extracting this information from the model. While we could train the model to also output "what Alice is thinking", this would have to be training for "a consistent and plausible answer to what Alice is thinking", since we don't have the ground truth answer. This could incentivize bad policies that figure out what we would most believe, rather than reporting the truth.

The argument for risk is then as follows: we care about inaccessible information (e.g. we care about what people _actually_ experience, rather than what they _say_ they experience) but can't easily make AI systems that optimize for it. However, AI systems will be able to infer and use inaccessible information, and would outcompete ones that don't. AI systems will be able to plan using such inaccessible information for at least some goals. Then, the AI systems that plan using the inaccessible information could eventually control most resources. Key quote: "The key asymmetry working against us is that optimizing flourishing appears to require a particular quantity to be accessible, while danger just requires anything to be accessible."

The post then goes on to list some possible angles of attack on this problem. Iterated amplification can be thought of as addressing gaps in speed, size, experience, algorithmic sophistication etc. between the agents we train and ourselves, which can limit what inaccessible information our agents can have that we won't. However, it seems likely that amplification will eventually run up against some inaccessible information that will never be produced. As a result, this could be a "hard core" of alignment.


There were several examples I hadn't heard of before, which were pretty entertaining. Also, if you have any examples that aren't already listed, it would be great to send them via the [form](https://docs.google.com/forms/d/e/1FAIpQLSeQEguZg4JfvpTywgZa3j-1J-4urrnjBVeoAO7JHIH53nrBTA/viewform) so that we can have a canonical list of specification gaming examples.

Specification gaming examples in AI

A list of examples of specification gaming, where an algorithm figures out a way to literally satisfy the given specification which does not match the designer's intent.