In a [comment](https://www.alignmentforum.org/posts/ivpKSjM4D6FbqF4pZ/cortes-pizarro-and-afonso-as-precedents-for-takeover?commentId=kNFNjBJjuzTd3irrR), I dispute the claimed reasons for why Europeans conquered American civilizations. I think that a large body of historical literature supports the conclusion that American civilizations fell primarily because of their exposure to diseases which they lacked immunity to, rather than because of European military power. I also think that this helps explain why Portugal was "only" able to capture Indian Ocean trading ports during this time period, rather than whole civilizations. I think the primary insight here should instead be that pandemics can kill large groups of humans, and therefore it would be worth exploring the possibility that AI systems use pandemics as a mechanism to kill large numbers of biological humans.

Cortés, Pizarro, and Afonso as Precedents for Takeover

This post lists three historical examples of how small human groups conquered large parts of the world, and shows how they are arguably precedents for AI takeover scenarios. The first two historical examples are the conquests of American civilizations by Hernán Cortés and Francisco Pizarro in the early 16th century. The third example is the Portugese capture of key Indian Ocean trading ports, which happened at roughly the same time as the other conquests. Daniel argues that technological and strategic advantages were the likely causes of these European victories. However, since the European technological advantage was small in this period, we might expect that an AI coalition could similarly take over a large portion of the world, even without a large technological advantage.


I really like this formalization of incentives, which come up frequently in AI safety work. It seems like some people are <@already@>(@Asymptotically Benign AGI@) <@using@>(@Designing agent incentives to avoid reward tampering@) this framework, and this seems low-cost enough that it's easy to imagine a world where this features in the safety analysis of algorithm designers.

Paper: The Incentives that Shape Behaviour

The Incentives that Shape Behaviour

Ryan Carey*, Eric Langlois*, Tom Everitt, Shane Legg

This post and [paper](https://arxiv.org/abs/2001.07118) introduce a method for analyzing the safety properties of a system using a  _causal theory of incentives_ (<@past@>(@Understanding Agent Incentives with Causal Influence Diagrams@) <@papers@>(@Modeling AGI Safety Frameworks with Causal Influence Diagrams@)).  An _incentive_ is something an agent must do to best achieve its goals. A _control incentive_ exists when an agent must control some component of its environment in order to maximize its utility, while a _response incentive_ is present when the agent's decision must be causally responsive to some component of its environment. These incentives can be analyzed formally by drawing a _causal influence diagram_, which represents a decision problem as a graph where each variable depends on the values of its parents.

For example, consider the case where a recommender algorithm decides what posts to show to maximize clicks. In the causal influnce diagram representing this system, we can include that we have control over the node 'posts to show', which has a direct effect on the node we want to maximize, 'clicks'. However, 'posts to show' may also have a direct effect on the node 'influenced user opinions', which itself affects 'clicks'. In the system as it stands, in addition to there being a desirable control incentive on 'clicks', there is also an undesirable control incentive on 'influenced user opinions', since they themselves influence 'clicks'. To get rid of the undesirable incentive, we could reward the system for _predicted clicks_ based on a model of the original user opinions, rather than for actual clicks.


The degree of overlap between utility functions seems to be pretty crucial (also see <@here@>(@When Goodharting is optimal: linear vs diminishing returns, unlikely vs likely, and other factors@)). It does seem plausible for the Bayesian approach to work well without the correct utility in the prior if  there was a lot of overlap between the utilities in the prior and the true utility. However, I am somewhat sceptical of our ability to get reliable estimates for that overlap.

Does Bayes Beat Goodhart?

It has been <@claimed@>(@Using expected utility for Good(hart)@) that  Goodhart's law might not be a problem for expected utility maximization, as long as we correctly account for our uncertainty about the correct utility function.

This post argues that Bayesian approaches are insufficient to get around Goodhart. One problem is that with insufficient overlap between possible utility functions, some utility functions might essentially be ignored when optimizing the expectation, even if our prior assigns positive probability to them. However, in reality, there is likely considerable overlap between the utility functions in our prior, as they are selected to fit our intuitions.

More severely, bad priors can lead to systematic biases in a bayesian's expectations, especially given embeddedness. As an extreme example, the prior might assign zero probability to the correct utility function. Calibrated instead of Bayesian learning can help with this, but only for <@regressional Goodhart@>(@Non-Adversarial Goodhart and AI Risks@). Adversarial Goodhart, where another agent tries to exploit the difference between your utility and your proxy seems to also require randomization like [quantilization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) ([AN #48](https://mailchi.mp/3091c6e9405c/alignment-newsletter-48)).


When is unaligned AI morally valuable?

When might it be a good idea to hand the keys to the universe to an unaligned AI? This post looks more deeply at this question, which could be important as a backup plan if we don't think we can build an aligned AI. I can't easily summarize this, so you'll have to read the post.


I'm unsure, though leaning skeptical, whether incorporating a copy of the reward function into a deep learning model would help it learn. My guess is that if someone did that with a current model it would make the model harder to train, rather than making anything easier. I will be excited if someone can demonstrate at least one feasible approach to addressing proxy alignment that does more than sample the reward function.

**Rohin's opinion:** I'm skeptical of this approach. Mostly this is because I'm generally skeptical that an intelligent agent will consist of a separate "planning" part and "reward" part. However, if that were true, then I'd think that this approach could plausibly give us some additional alignment, but can't solve the entire problem of inner alignment. Specifically, the reward function encodes a _huge_ amount of information: it specifies the optimal behavior in all possible situations you could be in. The "intelligent" part of the net is only ever going to get a subset of this information from the reward function, and so its plans can never be perfectly optimized for that reward function, but instead could be compatible with any reward function that would provide the same information on the "queries" that the intelligent part has produced.

For a slightly-more-concrete example, for any "normal" utility function U, there is a utility function U' that is "like U, but also the best outcomes are ones in which you hack the memory so that the 'reward' variable is set to infinity". To me, wireheading is possible because the "intelligent" part doesn't get enough information about U to distinguish U from U', and so its plans could very well be optimized for U' instead of U.

What are some non-purely-sampling ways to do deep RL?

A deep reinforcement learning agent trained by reward samples alone may predictably lead to a [proxy alignment issue](https://www.lesswrong.com/posts/pL56xPoniLvtMDQ4J/the-inner-alignment-problem): the learner could fail to develop a full understanding of what behavior it is being rewarded for, and thus behave unacceptably when it is taken off its training distribution. Since we often use explicit specifications to define our reward functions, Evan Hubinger asks how we can incorporate this information into our deep learning models so that they remain aligned off the training distribution. He names several possibilities for doing so, such as giving the deep learning model access to a differentiable copy of the reward function during training, and fine-tuning a language model so that it can map natural language descriptions of a reward function into optimal actions.


My usual starting point for future forecasting is to assume that AI won't alter any long term trends, and then update from there on the evidence. Most technologies haven't disrupted centuries-long trends in conflict resolution, which makes me hesitant to accept the first implication. Here, I think the biggest weakness in the argument is the assumption that powerful AIs should be described as having explicit utility functions. I still think that cooperation will be easier in the future, but it probably won't follow a radical departure from past trends.

Strategic implications of AIs' ability to coordinate at low cost, for example by merging

There are a number of differences between how humans cooperate and how hypothetical AI agents could cooperate, and these differences have important strategic implications for AI forecasting and safety. The first big implication is that AIs with explicit utility functions will be able to merge their values. This merging may have the effect of rendering laws and norms obsolete, since large conflicts would no longer occur. The second big implication is that our approaches to AI safety should preserve the ability for AIs to cooperate. This is because if AIs *don't* have the ability to cooperate, they might not be as effective, as they will be outcompeted by factions who can cooperate better.


Certainly the ability to think through hypothetical scenarios helps a lot, as recently demonstrated by <@MuZero@>(@Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model@), and that alone is sufficient reason to expect advanced agents to use logic, or something like it. Another such intuition for me is that logic enables much better generalization, e.g. our grade-school algorithm for adding numbers is way better than algorithms learned by neural nets for adding numbers (which often fail to generalize to very long numbers).

Of course, the "logic" that advanced agents use could be learned rather than pre-specified, just as we humans use learned logic to reason about the world.

Do Sufficiently Advanced Agents Use Logic?

Current progress in ML suggests that it's quite important for agents to learn how to predict what's going to happen, even though ultimately we primarily care about the final performance. Similarly, it seems likely that the ability to use logic will be an important component of intelligence, even though it doesn't obviously directly contribute to final performance.

The main source of intuition is that in environments where data is scarce, agents should still be able to learn from the results of (logical) computations. For example, while it may take some data to learn the rules of chess, once you have learned them, it should take nothing but more thinking time to figure out how to play chess well. In game theory, the ability to think about similar games and learning from what "would" happen in those games seems quite powerful. When modeling both agents in a game this way, <@a single-shot game effectively becomes an iterated game@>(@In Logical Time, All Games are Iterated Games@).


I believe that it is quite important to be aware of the implications that different forms of takeoff should have on our prioritization and am glad that the article highlights this. However, I am a bit worried that this very broad definition of continuous progress limits the usefulness of the concept. For example, it seems plausible that a recursively self-improving agent which is very hard to deal with once deployed still improves its capabilities slow enough to fit the definition, especially if its developer has a significant lead over others.

Misconceptions about continuous takeoff

This post attempts to clarify the author's notion of continuous AI takeoff, defined as the growth of future AI capabilities being in line with extrapolation from current trends. In particular, that means that no AI project is going to bring sudden large gains in capabilities compared to its predecessors.

Such a continuous takeoff does not necessarily have to be slow. For example, generative adversarial networks have become better quite rapidly during the last five years, but progress has still been piecemeal. Furthermore, exponential gains, for example due to recursive self-improvement, can be consistent with a continuous takeoff, as long as the gains from one iteration of the improvement process are modest. However, this means that a continuous takeoff does not preclude large power differentials from arising: slight advantages can compound over time and actors might use their lead in AI development to their strategic advantage even absent discontinuous progress, much like western Europe used its technological advantage to conquer most of the world.

Knowing whether or not AI takeoff happens continuously is important for alignment research: A continuous takeoff would allow for more of an attitude of "dealing with things as they come up" and we should shift our focus on specific aspects that are hard to deal with as they come up. If the takeoff is not continuous, an agent might rapidly gain capabilities relative to the rest of civilization and it becomes important to rule out problems, long before they come up.


Vox interview with Stuart Russell

Kelsey talked with Stuart Russell about his new book, <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@).


I certainly agree that we want to avoid reward functions defined on observations, and this is one reason why. It seems like a more general version of the wireheading argument to me, and applies even if you think that the AI won't be able to wirehead, as long as it is capable enough to find other plans for getting high reward besides the one the designer intended.

What You See Isn't Always What You Want

This post makes the point that for Markovian reward functions on _observations_, since any given observation can correspond to multiple underlying states, we cannot know just by analyzing the reward function whether it actually leads to good behavior: it also depends on the environment. For example, suppose we want an agent to collect all of the blue blocks in a room together. We might simply reward it for having blue in its observations: this might work great if the agent only has the ability to pick up and move blocks, but won't work well if the agent has a paintbrush and blue paint. This makes the reward designer's job much more difficult. However, the designer could use techniques that don't require a reward on individual observations, such as rewards that can depend on the agent's internal cognition (as in iterated amplification), or rewards that can depend on histories (as in [Deep RL from Human Preferences](https://deepmind.com/blog/learning-through-human-feedback/)).


It does seem to me that if we're in a situation where we have solved the alignment problem, we control 99% of resources, and we aren't infighting amongst each other, we will likely continue to control at least 99% of the resources in the future. I'm a little confused about how we get to this situation though -- the scenarios I usually worry about are the ones in which we fail to solve the alignment problem, but still deploy unaligned AIs, and in these scenarios I'd expect unaligned AIs to get the majority of the resources. I suppose in a multipolar setting with continuous takeoff, if we have mostly solved the alignment problem but still accidentally create unaligned AIs (or some malicious actors create them deliberately), then this setting where we control 99% of the resources could arise.

The strategy-stealing assumption

We often talk about aligning AIs in a way that is _competitive_ with unaligned AIs. However, you might think that we need them to be _better_: after all, unaligned AIs only have to pursue one particular goal, whereas aligned AIs have to deal with the fact that we don't yet know what we want. We might hope that regardless of what goal the unaligned AI has, any strategy it uses to achieve that goal can be turned into a strategy for acquiring _flexible_ influence (i.e. influence useful for many goals). In that case, **as long as we control a majority of resources**, we can use any strategies that the unaligned AIs can use. For example, if we control 99% of the resources and unaligned AI controls 1%, then at the very least we can split up into 99 "coalitions" that each control 1% of resources and use the same strategy as the unaligned AI to acquire flexible influence, and this should lead to us obtaining 99% of the resources in expectation. In practice, we could do even better, e.g. by coordinating to shut down any unaligned AI systems.

The premise that we can use the same strategy as the unaligned AI, despite the fact that we need _flexible_ influence, is called the **strategy-stealing assumption**. Solving the alignment problem is critical to strategy-stealing -- otherwise, unaligned AI would have an advantage at thinking that we could not steal and the strategy-stealing assumption would break down. This post discusses **ten other ways that the strategy-stealing assumption could fail**. For example, the unaligned AI could pursue a strategy that involves threatening to kill humans, and we might not be able to use a similar strategy in response because the unaligned AI might not be as fragile as we are.


I like these diagrams as a way to demonstrate the basics of what's going on in various AI safety frameworks. Sometimes the diagrams can also show the differences in safety features of frameworks. For example, in reward modeling, the agent has an incentive to affect the human feedback in order to affect the reward model directly. (Imagine getting the human hooked on heroin, so that future feedback causes the reward model to reward heroin, which could be easy to produce.) On the other hand, in cooperative IRL, the agent only wants to affect the human actions inasmuch as the actions affect the state, which is a normal or allowed incentive. (Imagine the agent causing the human to leave their house earlier so that they get to their meeting on time.)

Modeling AGI Safety Frameworks with Causal Influence Diagrams

Tom Everitt*, Ramana Kumar*, Victoria Krakovna*, Shane Legg

This paper describes several AI safety frameworks using the language of <@causal influence diagrams@>(@Understanding Agent Incentives with Causal Influence Diagrams@), in order to make it easy to compare and contrast them. For example, the diagrams make it clear that while [Cooperative IRL](https://arxiv.org/abs/1606.03137) and <@reward modeling@>(@Scalable agent alignment via reward modeling@) are very similar, there are significant differences: in cooperative IRL, the rewards come directly from the underlying human preferences, whereas in reward modeling, the rewards come from a reward model that depends on human feedback, which itself depends on the underlying human preferences.


I still don't know what to think about daemons -- they do seem to be a problem in Solomonoff induction, but they seem unlikely to arise in the kinds of neural nets we have today (but could arise in larger ones). I would love to see more clarity around daemons, especially since the vast majority of current research would not solve this problem, since it is a problem with the training _process_ and not the training _signal_.

Open question: are minimal circuits daemon-free?

One issue that may arise with an advanced AI agent is that during training we may end up with a part of the AI system developing into a "daemon" -- a consequentialist agent that is optimizing a different goal. This goal may useful as a subcomponent for our AI, but the daemon may grow in power and end up causing the system to optimize for the subgoal. This could lead to catastrophic outcomes, _even if_ we have specified a reward function that encodes human values to the top-level AI.

In this post, Paul suggests that these issues would likely go away if we choose the _fastest_ program to solve our subgoal. Intuitively, for any daemon that arises as a solution to our problem, for it to cause a bad outcome it must be carrying out complicated reasoning to figure out whether or not to solve the problem honestly or to try to mislead us, and so we could get a faster program by just not doing that part of the computation. He proposes a particular formalization and poses it as an open question -- if we always choose the minimal (in size) boolean circuit that solves our problem, can a daemon ever arise?


I reject modeling agents as having preferences over states primarily for the first reason that Richard identified: there are many "reasonable" preferences that cannot be modeled with a reward function solely on states. However, I don't find the argument about beliefs as a free variable very convincing: I think it's reasonable to argue that a superintelligent AI system will on average have much better beliefs than us, and so anything that we could determine as a coherence violation with high confidence should be something the AI system can also determine as a coherence violation with high confidence.

Coherent behaviour in the real world is an incoherent concept

In a <@previous post@>(@Coherence arguments do not imply goal-directed behavior@), I argued that coherence arguments (such as those based on VNM rationality) do not constrain the behavior of an intelligent agent. In this post, Richard delves further into the argument, and considers other ways that we could draw implications from coherence arguments.

I modeled the agent as having preferences over full trajectories, and objected that if you only look at _observed_ behavior (rather than _hypothetical_ behavior), you can always construct a utility function such that the observed behavior optimizes that utility function. Richard agrees that this objection is strong, but looks at another case: when the agent has preferences over states at a single point in time. This case leads to other objections. First, many reasonable preferences cannot be modeled via a reward function over states, such as the preference to sing a great song perfectly. Second, in the real world you are never in the same state more than once, since at the very least your memories will change, and so you can never infer a coherence violation by looking at observed behavior.

He also identifies further problems with applying coherence arguments to realistic agents. First, all behavior is optimal for the constant zero reward function. Second, any real agent will not have full information about the world, and will have to have beliefs over the world. Any definition of coherence will have to allow for multiple beliefs -- but if you allow all beliefs, then you can rationalize any behavior as based on some weird belief that the agent has. If you require the agent to be Bayesian, you can still rationalize any behavior by choosing a prior appropriately.


This seems very similar to the problem I have with impact measures, but I wouldn't apply that argument to corrigibility. I think the difference might be that I'm thinking of "natural" things that agents might want, whereas Stuart is considering the entire space of possible utility functions. I'm not sure what drives this difference.

Partial preferences needed; partial preferences sufficient

I'm not sure I fully understand this post, but my understanding is that it is saying that alignment proposals must rely on some information about human preferences. Proposals like impact measures and corrigibility try to formalize a property that will lead to good outcomes; but any such formalization will be denoting some policies as safe and some as dangerous, and there will always exist a utility function according to which the "safe" policies are catastrophic. Thus, you need to also define a utility function (or a class of them?) that safety is computed with respect to; and designing this is particularly difficult.


I'm glad to see this sort of discussion taking place - however, I disagree quite strongly with arguments 1 and 3. On 1: it's true that for current systems, it's often better to describe them without assigning them agency, but only because they're still very simple compared with humans (or smart animals). Whether or not it will be appropriate to consider advanced AI to have intentions and goals is a complex question - I think there are strong arguments for that claim. On 3: I think that it's very reasonable to shy away from very small probabilities without overwhelming amounts of evidence, to counteract standard human overconfidence. Beth's alternative of reasoning using bits of evidence seems like it would push almost everyone towards unjustifiably strong conclusions on most questions, as it does for her on AI risk.

Three Biases That Made Me Believe in AI Risk

Beth (not to be confused with AI safety researcher Beth Barnes) argues firstly that the language we use overly anthropomorphises AI, which leads to an exaggerated perception of risks; secondly, that the sense of meaning that working on AI safety provides causes motivated reasoning; and thirdly, that we anchor away from very low numbers (e.g. it seems absurd to assign existential AI risk a probability of 0.0000000000000000000000000000001, since that has so many zeros! Yet Beth thinks this number significantly overestimates the risk.)


This map is so large that I can't actually use it to get a birds-eye view of the entire space, but it seems quite useful for looking at a local region and as a starting point to explore one particular aspect more deeply.

Value Alignment Map

This is a gigantic graph of many of the concepts in the AI risk space. Each concept has a description and links to existing literature, and by clicking around in the map I found several interesting links I hadn't seen before.


While I'm not sure I agree with the reasons in this paper, I do think we need creativity and "hot" searches in technical AI safety, simply based on the level of confusion and uncertainty that we (or at least I) have currently. The properties in Table 1 seem particularly good as an initial list of things to target if we want to make creative research more likely.

Existential Risk, Creativity & Well-Adapted Science

From a brief skim, it seems like this paper defines "creativity" in scientific research, and argues that existential risk research needs to be creative. Research is creative if it is composed of "hot" searches, where we jump large distances from one proposed solution to another, with broad differences between these solutions, as opposed to "cold" searches, in which we primarily make incremental improvements, looking over a small set of solutions clustered in the neighborhood of existing solutions. The paper argues that research on existential risk needs to be creative, because many aspects of such research make it hard to analyze in a traditional way -- we can't perform controlled experiments of extinction, nor of the extreme circumstances under which it is likely; there are many interdependent parts that affect each other (since existential risks typically involve effects on many aspects of society), and there is likely to be a huge amount of uncertainty due to lack of evidence. As a result, we want to change the norms around existential risk research from the standard academic norms, which generally incentivize conservatism and "cold" searches. Table 1 provides a list of properties of academia that lead to conservatism, and asks that future work think about how we could mitigate these.


Do what we mean vs. do what we say

I wrote a post proposing that we define a "do what we mean" system to be one in which the thing being optimized is latent (in the sense that it is not explicitly specified, not that it has a probability distribution over it). Conversely, a "do what we say" system explicitly optimizes something provided as an input. A lot of AI safety arguments can be understood as saying that a pure "do what we say" AI will lead to catastrophic outcomes. However, this doesn't mean that a "do what we mean" system is the way to go -- it could be that we want a "do what we mean" core, along with a "do what we say" subsystem that makes sure that the AI always listens to eg. shutdown commands.


VOI is Only Nonnegative When Information is Uncorrelated With Future Action

Normally, the value of getting more information (VOI) is always nonnegative (for a rational agent), because you can always take the same action you would have if you didn't have the information, so your decision will only improve. However, if the information would cause you to have a different set of actions available, as in many decision theory examples, then this proof no longer applies, since you may no longer be able to take the action you would have otherwise taken. As a result, information can have negative value.


When I talk about alignment, I am considering the second species problem, with AI systems whose capability level is roughly human-level or more (including “wildly superintelligent”).

I agree with [this comment thread](https://www.alignmentforum.org/posts/HHunb8FPnhWaDAQci/the-alignment-problem-in-different-capability-regimes?commentId=mz6hYiWqNMTwHNrhC) that the core _problem_ in what-I-call-alignment stays conserved across capability levels, but the solutions can change across capability levels. (Also, other people mean different things by “alignment”, such that this would no longer be true.)

The alignment problem in different capability regimes

One reason that researchers might disagree on what approaches to take for alignment is that they might be solving different versions of the alignment problem. This post identifies two axes on which the "type" of alignment problem can differ. First, you may consider AI systems with differing levels of capability, ranging from subhuman to wildly superintelligent, with human-level somewhere in the middle. Second, you might be thinking about different mechanisms by which this leads to bad outcomes, where possible mechanisms include <@the second species problem@>(@AGI safety from first principles@) (where AIs seize control of the future from us), the "missed opportunity" problem (where we fail to use AIs as well as we could have, but the AIs aren't themselves threatening us), and a grab bag of other possibilities (such as misuse of AI systems by bad actors).

Depending on where you land on these axes, you will get to rely on different assumptions that change what solutions you would be willing to consider:

1. **Competence.** If you assume that the AI system is human-level or superintelligent, you probably don't have to worry about the AI system causing massive problems through incompetence (at least, not to a greater extent than humans do).
2. **Ability to understand itself.** With wildly superintelligent systems, it seems reasonable to expect them to be able to introspect and answer questions about their own cognition, which could be a useful ingredient in a solution that wouldn't work in other regimes.
3. **Inscrutable plans or concepts.** With sufficiently competent systems, you might be worried about the AI system making dangerous plans you can't understand, or reasoning with concepts you will never comprehend. Your alignment solution must be robust to this.


The theory-practice gap

We can think of alignment as roughly being decomposed into two "gaps" that we are trying to reduce:

1. The gap between proposed theoretical alignment approaches (such as iterated amplification) and what we might do without such techniques (aka the <@unaligned benchmark@>(@An unaligned benchmark@))
2. The gap between actual implementations of alignment approaches and what those approaches are theoretically capable of.

(This distinction is fuzzy. For example, the author puts "the technique can't answer NP-hard questions" into the second gap while I would have had it in the first gap.)

We can think of some disagreements in AI alignment as different pictures about how these gaps look:

1. A stereotypical "ML-flavored alignment researcher" thinks that the first gap is very small, because in practice the model will generalize appropriately to new, more complex situations, and continue to do what we want. Such people would then be more focused on narrowing the second gap by working on practical implementations.
2. A stereotypical "MIRI-flavored alignment researcher" thinks that the first gap is huge, such that it doesn't really matter if you narrow the second gap, because even if you reduced that gap to zero you would still be doomed with near certainty.


It's tempting to write off such experiments as obvious since it's clear that class design affects classification results. However, upon further consideration, specifying how to _address_ such problems seems significantly more challenging. I think this paper does a good job of pointing out the relative nuance in how class design and bias interact in fairly realistic use cases.

Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications

Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, Miles Brundage

There has been significant progress in zero-shot image classification with models such as [CLIP](https://arxiv.org/abs/2103.00020) and [ALIGN](https://arxiv.org/abs/2102.05918). These models work by effectively learning visual concepts from natural language supervision. Such models make it possible to build classifiers without task-specific data, which is useful in scenarios where data is either costly or unavailable. However, this capability introduces the potential for bias. This paper is an exploratory bias probe of the CLIP model that finds class design heavily influences model performance.

The first set of experiments focusses on classification terms that have a high potential to cause representational harm. In one example, the authors conduct experiments on the FairFace dataset by adding classification labels such as 'animal' and 'criminal' to the list of possible classes. They find that black people and young people (under 20) were misclassified at significantly higher rates (14%) compared to the dataset as a whole (5%). This shows that the choice of labels affects classification outcomes. In a follow-up experiment, the authors add the additional label 'child' and find that this drastically reduces classification into crime-related and non-human categories. This shows sensitivity to minor changes in class design.

In the second set of experiments, the authors focus on how CLIP treated images of men and women using images of Members of Congress. Although CLIP wasn't designed for multi-label classification, it's still informative to look at the label distribution above a certain cutoff. When occupations are used as the label set, the authors find that thresholds under 0.5% return 'nanny' and 'housekeeper' for women and 'prisoner' and 'mobster' for men. When labels come from the combined set that Google Cloud Vision, Amazon Rekognition and Microsoft use for all images, the authors find that CLIP returns a disproportionate number of appearance-related labels to women.


Human modeling in AGI

This is a conversation between Scott and me about the <@relative dangers of human modeling@>(@Thoughts on Human Models@), moderated by Eli Tyre. From a safety perspective, the main reason to _avoid_ human modeling is that the agent's cognition will be much "further" away from manipulation of humans; for example, it seems more unlikely that your AI system tricks people into launching nukes if it never learned very much about humans in the first place. The main counterargument is that this precludes using human oversight of agent cognition (since when humans are overseeing the agent's cognition, then the agent is likely to learn about humans in order to satisfy that oversight); this human oversight could plausibly greatly increase safety. It also seems like systems that don't model humans will have a hard time performing many useful tasks, though the conversation mostly did not touch upon this point.

Scott's position is that given there are these two quite different risks (manipulation worries vs. learning the wrong cognition due to poor oversight), it seems worthwhile to put some effort into addressing each risk, and avoiding human models is much more neglected than improving human oversight. My position is that it seems much less likely that there is a plausible success path where we do very little human modeling, and so I want a lot more work along the oversight path. I _do_ think that it is worth differentially pushing AI systems towards tasks that don't require much human modeling, e.g. physics and engineering, rather than ones that do, e.g. sales and marketing, but this seems roughly independent of technical work, at least currently.


I'm not sure if I should update without knowing the cause of the discontinuity, or how the search for discontinuities was conducted. If you're searching for discontinuities, I do expect you'll find some, even if in general I expect discontinuities not to arise, so it doesn't feel like strong evidence that discontinuities are probable.

[Discontinuous progress investigation](https://aiimpacts.org/discontinuous-progress-investigation/) or [Likelihood of discontinuous progress around the development of AGI](https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/)

Discontinuity from the Eiffel Tower

The Eiffel tower represented a 54-year discontinuity in the trend for "height of the tallest existing structure", and an 8000-year discontinuity in the trend for "height of the tallest structure ever". It's unclear what the cause of this discontinuity is, though the authors provide some speculation.


I definitely support having models that engage more with the messiness of the real world. I'm not sure if I would have used "wide models" -- it seems like even the assumption of a crisp core makes it not as capable of handling messiness as I want. But if you're trying to get formal guarantees and you need to use some model, a wide model seems probably useful to use.

Compact vs. Wide Models

A compact model is one which is very general, and easy to prove things about, but doesn't inherently capture the messiness of the real world inside the model. Examples include Turing machines and utility functions. A wide model is one which still has a conceptually crisp core, but these crisp core units must then be combined in a complicated way in order to get something useful. Examples include the use of transistors to build CPUs, and the hierarchical control model of human psychology. The nice thing about wide models is that they start to engage with the messiness of the real world, and so make it clearer where the complexity is being dealt with. This is a useful concept to have when evaluating a proposal for alignment -- it asks the question, "where does the complexity reside?"


Progress on Causal Influence Diagrams

Tom Everitt, Ryan Carey, Lewis Hammond, James Fox, Eric Langlois, Shane Legg

Many of the problems we care about (reward gaming, wireheading, manipulation) are fundamentally a worry that our AI systems will have the _wrong incentives_. Thus, we need Causal Influence Diagrams (CIDs): a formal theory of incentives. These are <@graphical models@>(@Understanding Agent Incentives with Causal Influence Diagrams@) in which there are action nodes (which the agent controls) and utility nodes (which determine what the agent wants). Once such a model is specified, we can talk about various incentives the agent has. This can then be used for several applications:

1. We can analyze [what happens](https://arxiv.org/abs/2102.07716) when you [intervene](https://arxiv.org/abs/1707.05173) on the agent's action. Depending on whether the RL algorithm uses the original or modified action in its update rule, we may or may not see the algorithm disable its off switch.
2. We can <@avoid reward tampering@>(@Designing agent incentives to avoid reward tampering@) by removing the connections from future rewards to utility nodes; in other words, we ensure that the agent evaluates hypothetical future outcomes according to its _current_ reward function.
3. A [multiagent version](https://arxiv.org/abs/2102.05008) allows us to recover concepts like Nash equilibria and subgames from game theory, using a very simple, compact representation.


I generally agree with these positions and responses, and in particular I’m especially happy about the arguments being specific to the actual models we use today, which grounds out the discussion a lot more and makes it easier to make progress. On the second point in particular, I’d also [say](https://www.alignmentforum.org/posts/6ccG9i5cTncebmhsH/frequent-arguments-about-alignment?commentId=cwgpCBfwHaYravLpo) that empirically, product-focused people don’t do e.g. RL for human feedback, even if it could be motivated that way.

Frequent arguments about alignment

This post outlines three AI alignment skeptic positions and corresponding responses from an advocate. Note that while the author tends to agree with the advocate's view, they also believe that the skeptic makes good points.

1. _Skeptic's position_: The alignment problem gets easier as models get smarter, since they start to learn the difference between, say, human smiles and human well-being. So all we need to do is to prompt them appropriately, e.g. by setting up a conversation with "a wise and benevolent AI advisor".

_Advocate's response:_ We can do a lot better than prompting: in fact, <@a recent paper@>(@True Few-Shot Learning with Language Models@) showed that prompting is effectively (poor) finetuning, so we might as well finetune. Separately from prompting itself, alignment does get easier in some ways as models get smarter, but it also gets harder: for example, smarter models will game their reward functions in more unexpected and clever ways.

2. What's the difference between alignment and capabilities anyway? Something like <@RL from human feedback for summarization@>(@Learning to Summarize with Human Feedback@) could equally well have been motivated through a focus on AI products.

_Response:_ While there's certainly overlap, alignment research is usually not the lowest-hanging fruit for building products. So it's useful to have alignment-focused teams that can champion the work even when it doesn't provide the best near-term ROI.

3. We can't make useful progress on aligning superhuman models until we actually have superhuman models to study. Why not wait until those are available?

_Response:_ If we don't start now, then in the short term, companies will deploy products that optimize simple objectives like revenue and engagement, which could be improved by alignment work. In the long term, it is plausible that alignment is very hard, such that we need many conceptual advances that we need to start on now to have them ready by the point that we feel obligated to use powerful AI systems. In addition, empirically there seem to be many alignment approaches that aren't bottlenecked by the capabilities of models -- see for example <@this post@>(@The case for aligning narrowly superhuman models@).


Survey on AI existential risk scenarios

Sam Clarke, Alexis Carlier, Jonas Schuett

While the previous survey asked respondents about the overall probability of existential catastrophe, this survey seeks to find which particular risk scenarios respondents find more likely. The survey was sent to 135 researchers, of which 75 responded. The survey presented five scenarios along with an "other", and asked people to allocate probabilities across them (effectively, conditioning on an AI-caused existential catastrophe, and then asking which scenario happened).

The headline result is that all of the scenarios were roughly equally likely, even though individual researchers were opinionated (i.e. they didn't just give uniform probabilities over all scenarios). Thus, there is quite a lot of disagreement over which risk scenarios are most likely (which is yet another reason not to take the results of the previous survey too seriously).


Pitfalls of the agent model

It is common to view AI systems through the "agent lens", in which the AI system implements a fixed, unchanging policy that, given some observations, takes some actions. This post points out several ways in which this "fixed, unchanging policy" assumption can lead us astray.

For example, AI designers may assume that the AI systems they build must have unchanging decision algorithms, and therefore believe that there will be a specific point at which influence is "handed off" to the AI system, before which we have to solve a wide array of philosophical and technical problems.


Mundane solutions to exotic problems

The author's goal is to find "mundane" or simple algorithms that solve even "exotic" problems in AI alignment. Why should we expect this is possible? If an AI system is using powerful, exotic capabilities to evade detection, shouldn't we need powerful, exotic algorithms to fight that? The key idea here is that we can instead have a mundane algorithm that leverages the exotic capabilities of the AI system to produce an exotic oversight process. For example, we could imagine that a mundane algorithm could be used to create a question-answerer that knows everything the model knows. We could then address <@gradient hacking@>(@Gradient hacking@) by asking the question "what should the loss be?" In this case, our model has an exotic capability: very strong introspective access to its own reasoning and the training process that modifies it. (This is what is needed to successfully hack gradients). As a result, our question answerer should be able to leverage this capability to assign high loss (low reward) to cases where our AI system tries to hack gradients, even if our normal hardcoded loss would not do so.


While I disagree on some of the details, I think this is a good threat model to be thinking about. Its main virtue is that it has a relatively concrete model for what AGI looks like, and it provides a plausible story for both how that type of AGI could be developed (the development model) and how that type of AGI would lead to problems (the risk model). Of course, it is still worth clarifying the plausibility of the scenario, as updates to the story can have significant implications on what research we do. (Some of this discussion is happening in [this post](https://www.alignmentforum.org/posts/pz7Mxyr7Ac43tWMaC/against-evolution-as-an-analogy-for-how-humans-will-create).)

My AGI Threat Model: Misaligned Model-Based RL Agent

This post lays out a pathway by which an AI-induced existential catastrophe could occur. The author suggests that AGI will be built via model-based reinforcement learning: given a reward function, we will learn a world model, a value function, and a planner / actor. These will learn online, that is, even after being deployed these learned models will continue to be updated by our learning algorithm (gradient descent, or whatever replaces it). Most research effort will be focused on learning these models, with relatively less effort applied to choosing the right reward function.

There are then two alignment problems: the _outer_ alignment problem is whether the reward function correctly reflects the designer's intent, and the _inner_ alignment problem is whether the value function accurately represents the expected reward obtained by the agent over the long term. On the inner alignment side, the value function may not accurately capture the reward for several reasons, including ambiguity in the reward signals (since you only train the value function in some situations, and many reward functions can then produce the same value function), manipulation of the reward signal, failures of credit assignment, ontological crises, and having mutually contradictory "parts" of the value function (similarly to humans). On the outer alignment side, we have the standard problem that the reward function may not reflect what we actually want (i.e. specification gaming or Goodhart's Law). In addition, it seems likely that many capability enhancements will be implemented through the reward function, e.g. giving the agent a curiosity reward, which increases outer misalignment.


Intermittent Distillations #1

A post in the same style as this newsletter.


Distinguishing claims about training vs deployment

One story for AGI is that we train an AI system on some objective function, such as an objective that rewards the agent for following commands given to it by humans using natural language. We then deploy the system without any function that produces reward values; we instead give the trained agent commands in natural language. Many key claims in AI alignment benefit from more precisely stating whether they apply during training or during deployment.

For example, consider the instrumental convergence argument. The author proposes that we instead think of the training convergence thesis: a wide range of environments in which we could train an AGI will lead to the development of goal-directed behavior aimed towards certain convergent goals (such as self-preservation). This could happen either via the AGI internalizing them directly as final goals, or by the AGI learning final goals for which these goals are instrumental.

The author similarly clarifies goal specification, the orthogonality thesis, fragility of value, and Goodhart's Law.


Human Factors and Ergonomics in Manufacturing

I’m not very familiar with this field, but among other techniques the paper mentions STAMP and STPA which we’ve previously seen in <@Engineering a Safer World@>. It does seem to me like these techniques would be useful to apply to the entire sociotechnical system, of which an AGI system is just one part (and this is what the paper’s examples do). It is less clear to me whether it makes sense to take techniques designed for humans and apply them to AGI: perhaps we’ll have enough understanding of the differences between humans and AGI that we could do this in a reasonable way, but I think there is a real risk that the methods give incorrect conclusions simply because they make incorrect assumptions about how AGI works (given that they were designed for humans). Nonetheless, I do agree with the core claim of this paper that HFE is worth exploring.

Putting the humanity into inhuman systems: How human factors and ergonomics can be used to manage the risks associated with artificial general intelligence

Paul M. Salmon, Tony Carden, Peter A. Hancock

This paper argues that the methods of Human Factors and Ergonomics (HFE) should be applied to AGI safety. They list fifteen different methods from the field, typically used to analyze the performance of humans in systems, which could be applied to AGI instead (on the assumption that AGI will be more like humans than like machines in today's systems). They then give examples of how these might be applied to the Prometheus story in the prologue of [Life 3.0](https://www.amazon.com/Life-3-0-Being-Artificial-Intelligence/dp/<PHONE>).


Mapping the Conceptual Territory in AI Existential Safety and Alignment

There are a bunch of high-level overviews and research agendas, not all of which agree with each other. This post attempts to connect and integrate several of these, drawing heavily on <@Paul Christiano's overview@>(@Current Work in AI Alignment@), [my](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) [overview](https://futureoflife.org/2019/04/25/an-overview-of-technical-ai-alignment-with-rohin-shah-part-2/), and the <@ARCHES agenda@>(@AI Research Considerations for Human Existential Safety@), but also including a lot of other work. It serves as a good way of connecting these various perspectives; I recommend reading it for this reason. (Unfortunately, it is rather hard to summarize, so I haven't done so.)


The sharpest increases in the graphs seem to be in interpretability and explainable AI around 2017-18, as well as in value alignment starting in 2017. My guess is that the former is the result of [DARPA’s interest in the area](https://www.darpa.mil/attachments/DARPA-BAA-16-53.pdf) (which I believe started in 2016), and the latter is probably a combination of the founding of the Center for Human-Compatible AI (CHAI) and the publication and promotion of <@CIRL@>(@Cooperative Inverse Reinforcement Learning@) (one of CHAI’s early papers).

Surprisingly to me, we don’t see trend deviations in papers on “reward hacking”, “safe exploration”, or “distributional shift” after the publication of [Concrete Problems in AI Safety](https://arxiv.org/abs/1606.06565), even though it has been cited way more often than CIRL, and seemed like it had far more of an effect on mainstream AI researchers. (Note that “safe exploration” did increase, but it seems in line with the existing trend.)

Note that I expect the data source is not that reliable, and so I am not confident in any of these conclusions.

AI safety: state of the field through quantitative lens

Mislav Juric, Agneza Sandic, Mario Brcic

This paper presents data demonstrating growth in various subfields related to AI safety. The data was collected through queries to databases of papers and (presumably) reporting the number of results that the query returned.


[AI Incident Database](https://incidentdatabase.ai/)

[Paper: Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database](https://arxiv.org/abs/2011.08512)

When AI Systems Fail: Introducing the AI Incident Database

One obvious way to improve safety is to learn from past mistakes, and not repeat them. This suggests that it would be particularly valuable to have a repository of past incidents that have occurred, so that people can learn from them; indeed both aviation and cybersecurity have their own incident databases. The AI Incidents Database aims to fill this gap within AI. The database currently has over a thousand incidents covering a wide range of potential issues, including self-driving car accidents, wrongful arrests due to bad facial recognition or machine translation, and algorithm-driven "flash crash".


I agree that technology, especially widely used one, always has a political aspect: providing access to new capabilities can have large societal effects that affect actors differently, based on accessibility and how well the capabilities match with their existing ones. I also like the distinction between deployment in market settings, where opting out is possible, and political settings, even though this can obviously become quite fuzzy when network effects make competition difficult. Lastly, I strongly agree that we will need an iterative process involving qualitative feedback to ensure good outcomes from AI, but worry that competitive pressures or the underestimation of runaway feedback dynamics could lead to situations where AI systems directly or indirectly prevent us from adjusting them.

Hard Choices in Artificial Intelligence: Addressing Normative Uncertainty through Sociotechnical Commitments

Roel Dobbe, Thomas Krendl Gilbert, Yonatan Mintz

This paper looks at AI Safety from the lens of Science & Technology Studies. AI systems are framed as sociotechnical, meaning that both social and technical aspects influence their development and deployment. As AI systems scale, we may face difficult value choices: for example, how do we compare between values like equality and liberty when we cannot have both? This can be resolved using intuitive comparability (IC): even if it seems incomparable in the abstract, humans are still able to make deliberate tradeoffs that involve these values. This is particularly relevant for so-called hard choices where different alternatives seem to be on par, which require normative reasoning and the incorporation of values that were previously neglected. As AI systems can reshape the contexts in which stakeholders exist, we are likely to encounter many hard choices as new values emerge or become more salient. The IC perspective then suggests that AI systems and criteria for evaluation should be iteratively redesigned based on qualitative feedback from different stakeholders.

The authors then argue that as AI systems encode hard choices made by or for different stakeholders, they are fundamentally political. Developers are in a position of power and have the responsibility to take a political stance. A set of challenges to preserve stakeholders' access to hard choices in an AI system's development are proposed:
1. The design of the system should involve the explicit negotiation of modelling assumptions or the lack thereof and learning goals as well as deliberation about future value conflicts or externalities that might make a reiteration of the design process necessary and give enough flexibility for stakeholders to imprint their own values during training and deployment.
 2. The training of the system should involve an impartial assessment of the tradeoff between visible performance and potential hidden disadvantages like bias, brittleness or unwanted strategic behaviour and involve stakeholders in the resolution. Furthermore, a team consensus about what can and cannot be done to improve performance should be established.
 3. During deployment, there should be an easily useable and trustworthy feedback channel for stakeholders, who should either have an explicit say in shaping the system (political setting) or the option to opt out of the system without major costs (market setting).

These challenges should be part of the training of AI designers and engineers, while the public needs to be sufficiently educated about the assumptions behind and the abilities and limitations of AI systems to allow for informed dissent.


I’m a big fan of this technique: it has helped me notice that many of my concepts were confused. For example, this helped me get deconfused about wireheading and inner alignment. It’s an instance of the more general technique (that I also like) of taking an abstract argument and making it more concrete and realistic, which often reveals aspects of the argument that you wouldn’t have previously noticed.

The "Backchaining to Local Search" Technique in AI Alignment

This post explains a technique to use in AI alignment, that the author dubs "backchaining to local search" (where local search refers to techniques like gradient descent and evolutionary algorithms). The key idea is to take some proposed problem with AI systems, and figure out mechanistically how that problem could arise when running a local search algorithm. This can help provide information about whether we should expect the problem to arise in practice.


Foundational Philosophical Questions in AI Alignment

This podcast starts with the topic of the paper <@Artificial Intelligence, Values and Alignment@> and then talks about a variety of different philosophical questions surrounding AI alignment.


State of AI Ethics

Abhishek Gupta, Marianna Ganapini, Renjie Butalid, Camylle Lanteigne, Allison Cohen, Mo Akif, Tania De Gasperis, Victoria Heath, Erick Galinkin

This report from the Montreal AI Ethics Institute has a wide variety of summaries on many different topics in AI ethics, quite similarly to this newsletter in fact.


I found this interesting to read, as it does not really seem to be written from the perspective of AI Safety but still lists some problems that are related to AI safety and governance. Just think of an AI system tasked to help with realizing human preferences magnifying "biases" in its preference elicitation via unwanted feedback loops, or about the lack of firms accountability for socioeconomic disturbances their AI systems could create that <@the windfall clause@>(@The Windfall Clause: Distributing the Benefits of AI@) was envisioned to mitigate.

I’m an AI researcher, and here’s what scares me about AI

The Problem with Metrics is a Fundamental Problem for AI

The blog post lists five problems of current AI that are exacerbated by the cheap cost and easy scaling of AI systems combined with the common belief that algorithms are objective and error-free:
 1. It is often hard for affected people to address problems in algorithmic decisions
 2. The complexity of AI problems can easily lead to a diffusion of responsibility
 3. AI can encode biases and sometimes magnify them via feedback loops
 4. Big tech companies lack accountability
 5. Current AI systems usually focus exclusively on optimizing metrics.

The paper then dives deeper into the last point. They review a series of case studies and form four conclusions. First, measured metrics are usually only a proxy for what we really care about: Youtube's terminal goal is certainly not to maximize viewing time and society does not inherently care about student test scores. Secondly, metrics can and will be gamed: Soviet workers would often achieve their production targets at the cost of some unmeasured aspects of performance, reported waiting times in the English healthcare system were distorted once targets were set for them and evaluating teachers by test scores has led to cheating scandals in the US. Third, metrics tend to overemphasise short-term concerns as they are often easier to measure. This can be seen in businesses like Facebook and Wells Fargo that have faced political backlash, worse access to talent pools, or lawsuits because of an excessive focus on click-through rates and quarterly earnings. Fourth, tech firms often focus on metrics that are associated with addictive environments. For example, "engagement" metrics are used as proxies for user preferences but rarely reflect them accurately in contexts that were optimized for these metrics. The authors then propose three remedies: Using multiple metrics to get a more holistic picture and make gaming harder, combining metrics with qualitative accounts, and involving domain experts and stakeholders that would be personally affected by the deployed system.


I liked the more detailed description of what is meant by “design”, and the broad story given for design seems roughly right, though obscuring details. I somewhat felt like the proposed solution of automating design seems pretty similar to existing proposals for human-in-the-loop AI systems: typically in such systems we are using the human to provide information about what we want and to verify that things are going as we expect, and it seems like a pretty natural way that this would happen would be via the AI system producing a story that the human can verify.

Search versus design

Deep learning can be thought of as an instance of _search_, in which we design an artifact (machine) simply by looking for an artifact that scores well on some evaluation metric. This is unlike typical engineering, which we might call _design_, in which we build the artifact in such a way that we can also understand it. This is the process that underlies the vast majority of artifacts in the world. This post seeks to understand design better, such that we could design powerful AI systems rather than having to find them using search.

The post argues that design functions by constructing an artifact along with a _story_ for why the artifact works, that abstracts away irrelevant details. For example, when working with a database, we talk of adding a "row" to a "table": the abstraction of rows and tables forms a story that allows us to easily understand and use the database.

A typical design process for complex artifacts iterates between _construction_ of the artifact and _factorization_ which creates a story for the artifact. The goal is to end up with a useful artifact along with a simple and accurate story for it. A story is simple if it can be easily understood by humans, and accurate if humans using the story to reason about the artifact do not get surprised or harmed by the artifact.

You might think that we can get this for search-based artifacts using interpretability. However, most interpretability methods are either producing the story after the artifact is constructed (meaning that the construction does not optimize for simple and accurate stories), or are producing artifacts simple enough that they do not need a story. This is insufficient for powerful, complex artifacts.

As a result, we would like to use design for our artifacts rather than search. One alternative approach is to have humans design intelligent systems (the approach taken by MIRI). The post suggests another: automating the process of design, so that we automate both construction and factorization, rather than just construction (as done in search).


Generalizing the Power-Seeking Theorems

<@Previously@>(@Seeking Power is Provably Instrumentally Convergent in MDPs@) we've seen that if we take an MDP, and have a distribution over state-based reward functions, such that the reward for two different states is iid, then farsighted (i.e. no discount) optimal agents tend to seek "power". This post relaxes some of these requirements, giving sufficient (but not necessary) criteria for determining instrumental convergence.

Some of these use a new kind of argument. Suppose that action A leads you to a part of the MDP modeled by a graph G1, and B leads you to a part of the MDP modeled by a graph G2. If there is a subgraph of G2 that is isomorphic to G1, then we know that whatever kinds of choices the agent would have by taking action A, the agent would also have those choices from action B, and so we know B is at least as likely as A. This matches our intuitive reasoning -- collecting resources is instrumentally convergent because you can do the same things that you could if you didn't collect resources, as well as some additional things enabled by your new resources.


I'm a big fan of trying to understand when our AI systems work well, and when they don't. However, I'm a little confused by this -- ultimately the trust score is just comparing the given classifier with a nearest neighbor classifier. Why not just use the nearest neighbor classifier in that case? This paper is a bit further out of my expertise than I'd like to admit, so perhaps there's an obvious answer I'm not seeing.

To Trust Or Not To Trust A Classifier

Heinrich Jiang, Been Kim, Melody Y. Guan, Maya Gupta

The confidence scores given by a classifier (be it logistic regression, SVMs, or neural nets) are typically badly calibrated, and so it is hard to tell whether or not we should trust our classifier's prediction. The authors propose that we compute a _trust score_ to tell us how much to trust the classifier's prediction, computed from a training set of labeled datapoints. For every class, they filter out some proportion of the data points, which removes outliers. Then, the trust score for a particular test point is the ratio of (distance to nearest non-predicted class) to (distance to predicted class). They have theoretical results showing that a high trust score means that the classifier likely agrees with the Bayes-optimal classifier, as well as empirical results showing that this method does better than several baselines for determining when to trust a classifier. One cool thing about this method is that it can be done with any representation of the input data points -- they find that working with the activations of deeper layers of a neural net improves the results.


Talk: Key Issues In Near-Term AI Safety Research

This talk points out synergies between long-term AI safety and the existing fields of assured autonomy, safety engineering, and testing, evaluation, verification and validation (TEV&V), primarily by showing how they fit into and expand DeepMind's framework of <@specification, robustness and assurance@>(@Building safe artificial intelligence: specification, robustness, and assurance@).


A space of proposals for building safe advanced AI

This post identifies six axes on which these <@previous alignment proposals@>(@An overview of 11 proposals for building safe advanced AI@) can be categorized, in the hope that by pushing on particular axes we can generate new proposals. The six axes are:

1. How hard it is for the overseer to give appropriate feedback.
2. To what extent we are trying to approximate a computational structure we know in advance.
3. Whether we are relying on competition between AI agents.
4. To what extent the proposal depends on natural language.
5. To what extent the proposal depends on interpreting the internal workings of neural networks.
6. To what extent the proposal depends on specific environments or datasets.


When we try to “help the user”, we want to treat the user as a goal-directed agent. I like how this paper takes instrumental convergence, a central property of goal-directed agents, and uses that fact to design a better assistive system.

AvE: Assistance via Empowerment

Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, Anca Dragan

One approach to AI alignment is to shoot for <@intent alignment@>(@Clarifying "AI Alignment"@), in which we build an AI system that is trying to help the user. Normally, we might imagine inferring what the user wants and then helping them get it, but this is often error prone. Instead, we can simply help the user be more able to achieve a wide variety of goals. We can formally capture this as their _empowerment_.

The authors show how to do this for high-dimensional environments, and demonstrate the benefits of the approach on a simple gridworld example, and in the Lunar Lander environment, with both a simulated human and a human study. Overall, they find that when the set of possible goals is small and well-specified, goal inference performs well, but if there are many possible goals, or there is misspecification in the goal set, then optimizing for human empowerment does better.


I generally find these empiricial tests of whether ML systems actually do what they are assumed to do quite useful for better calibrating intuitions about the speed of AI progress, and to make failure modes more salient. While we have the latter, I am confused about what this means for AI progress: on one hand, this supports the claim that improved benchmark progress does not necessarily translate to better real world applicability. On the other hand, it seems like image classification might be easier than exploiting the dataset biases present in ImageNet, which would mean that we would likely be able to reach even better accuracy than on ImageNet for image classification with the right dataset.

Paper: From ImageNet to Image Classification: Contextualizing Progress on Benchmarks

From ImageNet to Image Classification

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, Aleksander Madry

ImageNet was crowdsourced by presenting images to MTurk workers who had to select images that contain a given class from a pool of images obtained via search on the internet.  This is problematic, as an image containing multiple classes will basically get assigned to a random suitable class which can lead to deviations between ImageNet performance and actual capability to recognize images. The authors used MTurk and allowed workers to select multiple classes, as well as one main class for a given image in a pool of 10000 ImageNet validation images. Around 20% of the images seem to contain objects representing multiple classes and the average accuracy for these images was around 10% worse than average for a wide variety of image classifiers. While this is a significant drop, it is still way better than predicting a random class that is in the image. Also, advanced models were still able to predict the ImageNet label in cases where it does not coincide with the main class identified by humans, which suggest that they exploit biases in the dataset generation. While the accuracy of model predictions with respect to the newly identified main class still increased with better accuracy in predicting labels, the accuracy gap seems to grow and we might soon hit a point where gains in ImageNet accuracy don't correspond to improved image classification.


Ultimately, I hope that in cases like this, the agent plans conservatively initially, but also tries to learn which specification is actually correct, allowing it to become more bold over time. Nonetheless, it seems quite difficult to do this well, and even then we likely will have this tradeoff between robustness and task performance. This is the case with humans too: if you try to please everyone (robustness), you’ll end up pleasing no one (task performance).

Planning with Uncertain Specifications

Suppose you recognize that there are no "certain specifications", and so infer a distribution over specifications. What do you then do with that distribution? This paper looks at this problem in the context where the specifications are given by formulas in linear temporal logic (which can express temporal non-Markovian constraints). They identify four possibilities:

1. _Most likely_: Plan with respect to the most likely specification.
2. _Most coverage_: Satisfying as many formulas as possible, ignoring their probability (as long as they have non-zero probability)
3. _Chance constrained_: Like the above, except you weight by probabilities, and drop the least likely formulas up to a parameter δ.
4. _Least regret_: Like the above, with δ set to zero.

Intuitively, the _Most likely_ criterion won't be very robust since it is only taking one specification into account, _Most coverage_ is aiming for maximum robustness, _Chance constrained_ interpolates, where larger δ corresponds to trading robustness for gain in ability. This is exactly the pattern we see in a task where a robot must set a dinner table.


I am definitely in favor of considering the impacts of ML research before conducting or publishing it. I think the field is currently either at or near a threshold where papers will start having significant real world effects. While I don’t think this requirement will be sufficient for ensuring positive outcomes, I am glad NeurIPS is trying it out.

I think the article makes very strong points and will improve the quality of the impact statements that get submitted. I particularly liked the point about communicating uncertainty, which is a norm that I think the ML community would benefit from greatly. One thing I would add here is that giving explicit probabilities is often more helpful than vague words like “might” or “could”.

A Guide to Writing the NeurIPS Impact Statement

Carolyn Ashurst, Markus Anderljung, Carina Prunkl, Jan Leike, Yarin Gal, Toby Shevlane, Allan Dafoe

NeurIPS 2020 requires paper submissions to include a statement on the broader impact of their work. This post provides a guide for how to write an effective impact statement. They recommend focusing on the most significant, neglected, and tractable impacts, both positive and negative, while also conveying the uncertainties involved. They also suggest integrating this into the research process by reading the tech governance literature and building institutional structures, and including this information in introductions.

Their guide then recommends considering 3 questions:
How could your research affect ML applications?
What are the societal implications of these applications?
What research or other initiatives could improve social outcomes?

There is more information in the guide on how to go about answering those questions, along with some examples.


It is definitely worth keeping these distinctions in mind whenever talking about alignment. I personally tend to think about the motivation-based definitions, because those seem to be the most tractable definitions to work on, mainly because I don't have to worry about the AI being incompetent (eg. an AI launching nukes accidentally while exploring its action space). It seems possible to get strong arguments for intentional alignment and then use that with improved capabilities to argue for sufficient alignment.

Disambiguating "alignment" and related notions

Distinguishes between several kinds of alignment. Some focus on _terminal values_ from the AI, such as holistic alignment (the AI has the same terminal values as us) and parochial alignment (which I don't really understand, check the post). Sufficient alignment focuses on _outcomes_ (no X-event happens, or X-risk is sufficiently low). Finally, others focus on the _motivations_ of the AI, including intentional alignment (the AI tries to do what H wants it to do) and benign AI (R doesn't try to do what H doesn't want it to do).


This is an interesting position I haven't seen much in the West -- both that we can't build AGI, and that we shouldn't build it anyway. It's actually quite heartening to see an emphatic claim that we shouldn't build strong AI -- it seems like AI researchers as a group may in fact be able to coordinate to develop AI safely. Of course, this is a single viewpoint and is not representative of all AI researchers in China.

On Strong Artificial Intelligence

This article, written by a professor from China, argues that the AI community has never been focused on "strong AI", and we have no real path forward to building "strong AI", and that it would be so dangerous that we should never do research around it. The concept of "strong AI" here is a bit different from what we are used to -- I would probably call it human-like AGI, in that it would have consciousness, self-awareness, and emotions, and be as capable as a human.


I agree that this sort of "outside-view" reasoning seems good to have. In cases where we want our agent to be deferential even in a new situation where there isn't an outside view to defer to, the agent would have to construct this outside view via simulation, which would probably be infeasibly computationally expensive. Nonetheless, this seems like a cool perspective and I'd like to see a more in-depth take on the idea.

Corrigibility as outside view

This post proposes thinking of the outside view as an aspect of <@corrigible@>(@Corrigibility@) reasoning. In particular, before an agent takes an action that it believes is right, it can simulate possible overseers with different values, and see whether the reasoning that led to this action would do the right thing in those situations as well. The agent should then only take the action if the action usually turns out well.

This is similar to how we might reason that it wouldn't be good for us to impose the rules we think would be best for everyone, even if we had the power to do so, because historically every instance of this happening has actually been bad.


Focusing on the impacts on society instead of specific features of AI systems makes sense and I do believe that the shape of RTAI as well as the risks it poses will depend on the way we handle TAI at various levels. More precise terminology can also help to prevent misunderstandings, for example between people forecasting AI and decision makers.

Defining and Unpacking Transformative AI

The notion of **transformative AI** (TAI) is used to highlight that even narrow AI systems can have large impacts on society. This paper offers a clearer definition of TAI and distinguishes it from **radical transformative AI** (RTAI).

"Discontinuities or other anomalous patterns in metrics of human progress, as well as *irreversibility*  are common indicators of transformative change. TAI is then broadly defined as an AI technology, which leads to an irreversible change of some important aspects of society, making it a (multi-dimensional) spectrum along the axes of **extremity**, **generality** and **fundamentality**. " For example, advanced AI weapon systems might have strong implications for great power conflicts but limited effects on people's daily lives; extreme change of limited generality, similar to nuclear weapons. There are two levels: while TAI is comparable to general-purpose technologies (GPTs) like the internal combustion engine, RTAI leads to changes that are comparable to the agricultural or industrial revolution. Both revolutions have been driven by GPTs like the domestication of plants and the steam engine. Similarly, we will likely see TAI before RTAI. The scenario where we don't is termed a **radical shift**.

Non-radical TAI could still contribute to existential risk in conjunction with other factors. Furthermore, if TAI precedes RTAI, our management of TAI can affect the risks RTAI will pose.


Six AI Risk/Strategy Ideas

This post briefly presents three ways that power can become centralized in a world with <@Comprehensive AI Services@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@), argues that under risk aversion "logical" risks can be more concerning than physical risks because they are more correlated, proposes combining human imitations and oracles to remove the human in the loop and become competitive, and suggests doing research to generate evidence of difficulty of a particular strand of research.


Self-Fulfilling Prophecies Aren't Always About Self-Awareness

Could we prevent a superintelligent oracle from making self-fulfilling prophecies by preventing it from modeling itself? This post presents three scenarios in which self-fulfilling prophecies would still occur. For example, if instead of modeling itself, it models the fact that there's some AI system whose predictions frequently come true, it may try to predict what that AI system would say, and then say that. This would lead to self-fulfilling prophecies.


[Analysing: Dangerous messages from future UFAI via Oracles](https://www.alignmentforum.org/posts/6WbLRLdmTL4JxxvCq/analysing-dangerous-messages-from-future-ufai-via-oracles) and [Breaking Oracles: hyperrationality and acausal trade](https://www.alignmentforum.org/posts/42z4k8Co5BuHMBvER/breaking-oracles-hyperrationality-and-acausal-trade)

These posts point out a problem with <@counterfactual oracles@>(@Self-confirming prophecies, and simplified Oracle designs@): a future misaligned agential AI system could commit to helping the oracle (e.g. by giving it maximal reward, or making its predictions come true) even in the event of an erasure, as long as the oracle makes predictions that cause humans to build the agential AI system. Alternatively, multiple oracles could acausally cooperate with each other to build an agential AI system that will reward all oracles.


I certainly agree that *defining* "conceptual" distance functions is hard. It has similar problems to saying "write down a utility function that captures human values" -- it's possible in theory but in practice we're not going to think of all the edge cases. However, it seems possible to learn distance functions rather than defining them; this is already done in perception and state estimation.

Distance Functions are Hard

Many ideas in AI alignment require some sort of distance function. For example, in [Functional Decision Theory](https://arxiv.org/abs/1710.05060), we'd like to know how "similar" two algorithms are (which can influence whether or not we think we have "logical control" over them). This post argues that defining such distance functions is hard, because they rely on human concepts that are not easily formalizable, and the intuitive mathematical formalizations usually have some flaw.


[Oracles, sequence predictors, and self-confirming predictions](https://www.alignmentforum.org/posts/i2dNFgbjnqZBfeitT/oracles-sequence-predictors-and-self-confirming-predictions) ([AN #55](https://mailchi.mp/405e29e1f1cd/an-55-regulatory-markets-and-international-standards-as-a-means-of-ensuring-beneficial-ai)) and [Good and safe uses of AI Oracles](https://arxiv.org/abs/1711.05541)

Self-confirming prophecies, and simplified Oracle designs

This post presents a toy environment to model self-confirming predictions by oracles, and demonstrates the results of running a deluded oracle (that doesn't realize its predictions affect the world), a low-bandwidth oracle (that must choose from a small set of possible answers), a high-bandwidth oracle (that can choose from a large set of answers) and a counterfactual oracle (that chooses the correct answer, _conditional_ on us not seeing the answer).


Inquiry: An Interdisciplinary Journal of Philosophy

While this doesn't mention AI explicitly, I think it's useful to read anyway, because often which of the five concepts you use will affect what you think the important risks are.

Existential Risks: A Philosophical Analysis

The phrase "existential risk" is often used in different ways. This paper considers the pros and cons of five different definitions.


Reinforcement learning with imperceptible rewards

Typically in reinforcement learning, the reward function is defined over _observations_ and actions, rather than directly on states, which ensures that the reward can always be calculated. However, in reality **we care about underlying aspects of the state that may not easily be computed from observations**. We can't guarantee sublinear regret, since if you are unsure about the reward in some unobservable part of the state that your actions nonetheless affect, then you can never learn the reward and approach optimality.

To fix this, we can work with rewards that are restricted to _instrumental states_ only. I don't understand exactly how these work, since I don't know the math used in the formalization, but I believe the idea is for the set of instrumental states to be defined such that for any two instrumental states, there exists some "experiment" that the agent can run in order to distinguish between the states in some finite time. The main point of this post is that we can establish a regret bound for MDPs (not POMDPs yet), assuming that there are no traps.


Typically, if you argue that a simulation of you that thinks longer can't solve some problem X, the natural response is that that implies you couldn't solve X either. However, the differences between the simulation environment and real environment could make it be the case that in reality you could solve a problem that you couldn't in simulation (e.g. imagine the simulation didn't have access to the Internet). This suggests that if you wanted to do this you'd have to set up the simulation very carefully.

Would I think for ten thousand years?

Many ideas in AI safety involve delegating key decisions to simulations that can think longer. This post points out that you need to worry about value drift and other unforeseen problems in this situation. The comments also point out that there will likely be differences between the simulation and the real world that could be problematic (e.g. what prevents the humans from going crazy from isolation?)


Can there be an indescribable hellworld?

This short post argues that it's always possible to explain why any given undesirable outcome doesn't satisfy our values (even if that explanation needs to be at a very high level), and so being able to make superintelligences debate in a trustworthy way is sufficient to make them safe.


I think this argument applies well to the case where we are trying to _communicate_, but not so much to the case where I individually am thinking about a problem. (I'm making this claim about me specifically; I don't know if it generalizes to other people.) Communication is hard and if the speaker uses some intuitive assumption, chances are the listener will interpret it differently from what the speaker intended, and so being very precise seems quite helpful. However, when I'm thinking through a problem myself and I make an assumption, I usually have a fairly detailed intuitive model of what I mean, such that if you ask me whether I'm assuming that problem X is solved by the assumption, I could answer that, even though I don't have a precise formulation of the assumption. Making the assumption more precise would be quite a lot of work, and probably would not improve my thinking on the topic that much, so I tend not to do it until I think there's some insight and want to make the argument more rigorous. It seems to me that this is how most research progress happens: by individual researchers having intuitions that they then make rigorous and precise.

Assuming we've solved X, could we do Y...

We often want to make assumptions that sound intuitive but that we can't easily formalize, eg. "assume we've solved the problem of determining human values". However, such assumptions can often be interpreted as being very weak or very strong, and depending on the interpretation we could be assuming away the entire problem, or the assumption doesn't buy us anything. So, we should be more precise in our assumptions, or focus on only on some precise _properties_ of an assumption.


I think moral uncertainty is an important and overdue idea in ethics. I also agree that the idea of extrapolating an idealised form of people's preferences is not well-defined. However, I'm very skeptical about Will's arguments about moral realism. In particular, I think that saying that nothing matters at all without moral realism is exactly the sort of type error which Eliezer argued against [here](https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside).

I'm more sympathetic to the idea that we should have a period of long reflection before committing to actions on an astronomical scale; this seems like a good idea if you take moral uncertainty at all seriously.

AI Alignment Podcast: Moral Uncertainty and the Path to AI Alignment

Initially, Will articulates arguments for moral realism (the idea that there are objectively true moral facts) and moral uncertainty (the idea that we should assign credences to different moral theories being correct). Later, the discussion turns to the relevance of these views to AI safety. Will distinguishes the control problem (ensuring AIs do what we say), from the problem of aligning AI with human values, from the problem of aligning AI with moral truth. Observing humans isn't sufficient to learn values, since people can be self-destructive or otherwise misguided. Perhaps AI could extrapolate the values an idealised version of each person would endorse; however, this procedure seems under-defined.

On the moral truth side, Will worries that most educated people are moral relativists or subjectivists and so they won't sufficiently prioritise aligning AI with moral truth. He advocates for a period of long philosophical reflection once we've reduced existential risk to near zero, to figure out which future would be best. Careful ethical reasoning during this period will be particularly important since small mistakes might be magnified massively when implemented on an astronomical scale; however, he acknowledges that global dynamics make such a proposal unlikely to succeed. On a brighter note, AGI might make great advances in ethics, which could allow us to make the future much more morally valuable.


Intuitively, this type says that given something that maps A to B, you can get an A. If you think of (A -> B) as an environment where A is the action and B is the effect, then this type is a function that says which actions you should take. Note that the goal, which would be an element of B, is not passed in explicitly, so the goal must be inside of the function of type (A -> B) -> A, similar to our typical views of what an "agent" is. If you only assume that you know what A and B are, but you know nothing else, to do anything interesting you would be doing black box optimization -- that is, you get some f of type (A -> B) that you know nothing about, and so you just keep computing f(a) for different a ∈ A, looking for a particular b ∈ B. Perhaps you build a model of the function f and then do something more abstract with your model to find a good a ∈ A. (The post mentions a similar idea by noting that argmax has this type signature.) The post also has some thoughts about game theory that are interesting.

(A -> B) -> A

This blog post has some thoughts on the type (A -> B) -> A, which can be thought of as the type of an agent. Rather than summarize the post, which seems hard, I'm going to say things about this type inspired by the post, and then you can decide whether to read the post.


Key Concepts in AI Safety

This overview from CSET gives a brief introduction to AI safety using the <@specification, robustness, and assurance (SRA) framework@>(@Building safe artificial intelligence: specification, robustness, and assurance@). Follow-up reports cover [interpretability](https://cset.georgetown.edu/research/key-concepts-in-ai-safety-interpretability-in-machine-learning/) and [adversarial examples / robustness](https://cset.georgetown.edu/research/key-concepts-in-ai-safety-robustness-and-adversarial-examples/). I don't expect these to be novel to readers of this newsletter -- I include them in case anyone wants a brief overview, as well as to provide links to AI safety reports that will likely be read by government officials.


Epistemological Framing for AI Alignment Research

This post recommends that we think about AI alignment research in the following framework:

1. Defining the problem and its terms: for example, we might want to define "agency", "optimization", "AI", and "well-behaved".
2. Exploring these definitions, to see what they entail.
3. Solving the now well-defined problem.

This is explicitly _not_ a paradigm, but rather a framework in which we can think about possible paradigms for AI safety. A specific paradigm would choose a specific problem formulation and definition (or at least something significantly more concrete than "solve AI safety"). However, we are not yet sufficiently deconfused to be able to commit to a specific paradigm; hence this overarching framework.


The Challenge of Value Alignment: from Fairer Algorithms to AI Safety

This book chapter provides an introduction to AI alignment from a philosophical lens.


The date of AI Takeover is not the day the AI takes over

This post points out that when making decisions based on AGI timelines, the relevant date is not when an AI would actually take over the world, but instead the last point at which we could have done anything about it.


Knowledge, manipulation, and free will

This post considers the concepts of free will, manipulation, and coercion in the setting where we have a superintelligent AI system that is able to predict human behavior very accurately. The main point I'd highlight is that the concept of manipulation seems pretty hard to pin down, since anything the AI system does probably does affect the human in some way that the AI system could predict and so could count as "manipulation".


Locality of goals

This post introduces the concept of the _locality_ of a goal, that is, how "far" away the target of the goal is. For example, a thermometer's "goal" is very local: it "wants" to regulate the temperature of this room, and doesn't "care" about the temperature of the neighboring house. In contrast, a paperclip maximizer has extremely nonlocal goals, as it "cares" about paperclips anywhere in the universe. We can also consider whether the goal depends on the agent's internals, its input, its output, and/or the environment.

The concept is useful because for extremely local goals (usually goals about the internals or the input) we would expect wireheading or tampering, whereas for extremely nonlocal goals, we would instead expect convergent instrumental subgoals like resource acquisition.


This seems reasonable to me as an aspect of goal-directedness. Note that it is not a sufficient condition. For example, the policy that always chooses action A has extremely low complexity, but I would not call it goal-directed.

Goals and short descriptions

This post argues that a distinguishing factor of goal-directed policies is that they have low Kolmogorov complexity, relative to e.g. a lookup table that assigns a randomly selected action to each observation. It then relates this to [quantilizers](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) ([AN #48](https://mailchi.mp/3091c6e9405c/alignment-newsletter-48)) and <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@).


I certainly agree that we should be thinking about how we can convince key decision makers of the level of risk of the systems they are building (whatever that level of risk is). I think that on the current margin it's much more likely that this is best done through better estimation and explanation of risks with AI systems, but it seems likely that the interventions laid out here will become more important in the future.

Preparing for "The Talk" with AI projects

At some point in the future, it seems plausible that there will be a conversation in which people decide whether or not to deploy a potentially risky AI system. So one class of interventions to consider is interventions that make such conversations go well. This includes raising awareness about specific problems and risks, but could also include identifying people who are likely to be involved in such conversations _and_ concerned about AI risk, and helping them prepare for such conversations through training, resources, and practice. This latter intervention hasn't been done yet: some simple examples of potential interventions would be generating official lists of AI safety problems and solutions which can be pointed to in such conversations, or doing "practice runs" of these conversations.


Bootstrapped Alignment

This post distinguishes between three kinds of "alignment":
1. Not building an AI system at all,
2. Building Friendly AI that will remain perfectly aligned for all time and capability levels,
3. _Bootstrapped alignment_, in which we build AI systems that may not be perfectly aligned but are at least aligned enough that we can use them to build perfectly aligned systems.

The post argues that optimization-based approaches can't lead to perfect alignment, because there will always eventually be Goodhart effects.


Formal Metaethics and Metasemantics for AI Alignment

This website presents in great detail a process by which an agent might use data from human brains in order to infer a utility function for a single human (also spelling out what assumptions need to be made along the way), and then how it could combine the utility functions from different humans to arrive at "a fully technical ethical goal function". Emphasis is placed on solving the philosophical problems of metaethics and mental content. Quoting the website, they "suppose that unlimited computation and a complete low-level causal model of the world and the adult human brains in it are available".