I found this interesting to read, as it does not really seem to be written from the perspective of AI Safety but still lists some problems that are related to AI safety and governance. Just think of an AI system tasked to help with realizing human preferences magnifying "biases" in its preference elicitation via unwanted feedback loops, or about the lack of firms accountability for socioeconomic disturbances their AI systems could create that <@the windfall clause@>(@The Windfall Clause: Distributing the Benefits of AI@) was envisioned to mitigate.

I’m an AI researcher, and here’s what scares me about AI

The Problem with Metrics is a Fundamental Problem for AI

The blog post lists five problems of current AI that are exacerbated by the cheap cost and easy scaling of AI systems combined with the common belief that algorithms are objective and error-free:
 1. It is often hard for affected people to address problems in algorithmic decisions
 2. The complexity of AI problems can easily lead to a diffusion of responsibility
 3. AI can encode biases and sometimes magnify them via feedback loops
 4. Big tech companies lack accountability
 5. Current AI systems usually focus exclusively on optimizing metrics.

The paper then dives deeper into the last point. They review a series of case studies and form four conclusions. First, measured metrics are usually only a proxy for what we really care about: Youtube's terminal goal is certainly not to maximize viewing time and society does not inherently care about student test scores. Secondly, metrics can and will be gamed: Soviet workers would often achieve their production targets at the cost of some unmeasured aspects of performance, reported waiting times in the English healthcare system were distorted once targets were set for them and evaluating teachers by test scores has led to cheating scandals in the US. Third, metrics tend to overemphasise short-term concerns as they are often easier to measure. This can be seen in businesses like Facebook and Wells Fargo that have faced political backlash, worse access to talent pools, or lawsuits because of an excessive focus on click-through rates and quarterly earnings. Fourth, tech firms often focus on metrics that are associated with addictive environments. For example, "engagement" metrics are used as proxies for user preferences but rarely reflect them accurately in contexts that were optimized for these metrics. The authors then propose three remedies: Using multiple metrics to get a more holistic picture and make gaming harder, combining metrics with qualitative accounts, and involving domain experts and stakeholders that would be personally affected by the deployed system.


I liked the more detailed description of what is meant by “design”, and the broad story given for design seems roughly right, though obscuring details. I somewhat felt like the proposed solution of automating design seems pretty similar to existing proposals for human-in-the-loop AI systems: typically in such systems we are using the human to provide information about what we want and to verify that things are going as we expect, and it seems like a pretty natural way that this would happen would be via the AI system producing a story that the human can verify.

Search versus design

Deep learning can be thought of as an instance of _search_, in which we design an artifact (machine) simply by looking for an artifact that scores well on some evaluation metric. This is unlike typical engineering, which we might call _design_, in which we build the artifact in such a way that we can also understand it. This is the process that underlies the vast majority of artifacts in the world. This post seeks to understand design better, such that we could design powerful AI systems rather than having to find them using search.

The post argues that design functions by constructing an artifact along with a _story_ for why the artifact works, that abstracts away irrelevant details. For example, when working with a database, we talk of adding a “row” to a “table”: the abstraction of rows and tables forms a story that allows us to easily understand and use the database.

A typical design process for complex artifacts iterates between _construction_ of the artifact and _factorization_ which creates a story for the artifact. The goal is to end up with a useful artifact along with a simple and accurate story for it. A story is simple if it can be easily understood by humans, and accurate if humans using the story to reason about the artifact do not get surprised or harmed by the artifact.

You might think that we can get this for search-based artifacts using interpretability. However, most interpretability methods are either producing the story after the artifact is constructed (meaning that the construction does not optimize for simple and accurate stories), or are producing artifacts simple enough that they do not need a story. This is insufficient for powerful, complex artifacts.

As a result, we would like to use design for our artifacts rather than search. One alternative approach is to have humans design intelligent systems (the approach taken by MIRI). The post suggests another: automating the process of design, so that we automate both construction and factorization, rather than just construction (as done in search).


Generalizing the Power-Seeking Theorems

<@Previously@>(@Seeking Power is Provably Instrumentally Convergent in MDPs@) we’ve seen that if we take an MDP, and have a distribution over state-based reward functions, such that the reward for two different states is iid, then farsighted (i.e. no discount) optimal agents tend to seek “power”. This post relaxes some of these requirements, giving sufficient (but not necessary) criteria for determining instrumental convergence.

Some of these use a new kind of argument. Suppose that action A leads you to a part of the MDP modeled by a graph G1, and B leads you to a part of the MDP modeled by a graph G2. If there is a subgraph of G2 that is isomorphic to G1, then we know that whatever kinds of choices the agent would have by taking action A, the agent would also have those choices from action B, and so we know B is at least as likely as A. This matches our intuitive reasoning -- collecting resources is instrumentally convergent because you can do the same things that you could if you didn’t collect resources, as well as some additional things enabled by your new resources.


I'm a big fan of trying to understand when our AI systems work well, and when they don't. However, I'm a little confused by this -- ultimately the trust score is just comparing the given classifier with a nearest neighbor classifier. Why not just use the nearest neighbor classifier in that case? This paper is a bit further out of my expertise than I'd like to admit, so perhaps there's an obvious answer I'm not seeing.

To Trust Or Not To Trust A Classifier

Heinrich Jiang, Been Kim, Melody Y. Guan, Maya Gupta

The confidence scores given by a classifier (be it logistic regression, SVMs, or neural nets) are typically badly calibrated, and so it is hard to tell whether or not we should trust our classifier's prediction. The authors propose that we compute a _trust score_ to tell us how much to trust the classifier's prediction, computed from a training set of labeled datapoints. For every class, they filter out some proportion of the data points, which removes outliers. Then, the trust score for a particular test point is the ratio of (distance to nearest non-predicted class) to (distance to predicted class). They have theoretical results showing that a high trust score means that the classifier likely agrees with the Bayes-optimal classifier, as well as empirical results showing that this method does better than several baselines for determining when to trust a classifier. One cool thing about this method is that it can be done with any representation of the input data points -- they find that working with the activations of deeper layers of a neural net improves the results.


Podcast: Astronomical Future Suffering and Superintelligence with Kaj Sotala


Talk: Key Issues In Near-Term AI Safety Research

This talk points out synergies between long-term AI safety and the existing fields of assured autonomy, safety engineering, and testing, evaluation, verification and validation (TEV&V), primarily by showing how they fit into and expand DeepMind's framework of <@specification, robustness and assurance@>(@Building safe artificial intelligence: specification, robustness, and assurance@).


A space of proposals for building safe advanced AI

This post identifies six axes on which these <@previous alignment proposals@>(@An overview of 11 proposals for building safe advanced AI@) can be categorized, in the hope that by pushing on particular axes we can generate new proposals. The six axes are:

1. How hard it is for the overseer to give appropriate feedback.
2. To what extent we are trying to approximate a computational structure we know in advance.
3. Whether we are relying on competition between AI agents.
4. To what extent the proposal depends on natural language.
5. To what extent the proposal depends on interpreting the internal workings of neural networks.
6. To what extent the proposal depends on specific environments or datasets.


When we try to “help the user”, we want to treat the user as a goal-directed agent. I like how this paper takes instrumental convergence, a central property of goal-directed agents, and uses that fact to design a better assistive system.

AvE: Assistance via Empowerment

Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, Anca Dragan

One approach to AI alignment is to shoot for <@intent alignment@>(@Clarifying "AI Alignment"@), in which we build an AI system that is trying to help the user. Normally, we might imagine inferring what the user wants and then helping them get it, but this is often error prone. Instead, we can simply help the user be more able to achieve a wide variety of goals. We can formally capture this as their _empowerment_.

The authors show how to do this for high-dimensional environments, and demonstrate the benefits of the approach on a simple gridworld example, and in the Lunar Lander environment, with both a simulated human and a human study. Overall, they find that when the set of possible goals is small and well-specified, goal inference performs well, but if there are many possible goals, or there is misspecification in the goal set, then optimizing for human empowerment does better.


I generally find these empiricial tests of whether ML systems actually do what they are assumed to do quite useful for better calibrating intuitions about the speed of AI progress, and to make failure modes more salient. While we have the latter, I am confused about what this means for AI progress: on one hand, this supports the claim that improved benchmark progress does not necessarily translate to better real world applicability. On the other hand, it seems like image classification might be easier than exploiting the dataset biases present in ImageNet, which would mean that we would likely be able to reach even better accuracy than on ImageNet for image classification with the right dataset.

Paper: From ImageNet to Image Classification: Contextualizing Progress on Benchmarks

From ImageNet to Image Classification

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, Aleksander Madry

ImageNet was crowdsourced by presenting images to MTurk workers who had to select images that contain a given class from a pool of images obtained via search on the internet.  This is problematic, as an image containing multiple classes will basically get assigned to a random suitable class which can lead to deviations between ImageNet performance and actual capability to recognize images. The authors used MTurk and allowed workers to select multiple classes, as well as one main class for a given image in a pool of 10000 ImageNet validation images. Around 20% of the images seem to contain objects representing multiple classes and the average accuracy for these images was around 10% worse than average for a wide variety of image classifiers. While this is a significant drop, it is still way better than predicting a random class that is in the image. Also, advanced models were still able to predict the ImageNet label in cases where it does not coincide with the main class identified by humans, which suggest that they exploit biases in the dataset generation. While the accuracy of model predictions with respect to the newly identified main class still increased with better accuracy in predicting labels, the accuracy gap seems to grow and we might soon hit a point where gains in ImageNet accuracy don't correspond to improved image classification.


Ultimately, I hope that in cases like this, the agent plans conservatively initially, but also tries to learn which specification is actually correct, allowing it to become more bold over time. Nonetheless, it seems quite difficult to do this well, and even then we likely will have this tradeoff between robustness and task performance. This is the case with humans too: if you try to please everyone (robustness), you’ll end up pleasing no one (task performance).

Planning with Uncertain Specifications

Suppose you recognize that there are no “certain specifications”, and so infer a distribution over specifications. What do you then do with that distribution? This paper looks at this problem in the context where the specifications are given by formulas in linear temporal logic (which can express temporal non-Markovian constraints). They identify four possibilities:

1. _Most likely_: Plan with respect to the most likely specification.
2. _Most coverage_: Satisfying as many formulas as possible, ignoring their probability (as long as they have non-zero probability)
3. _Chance constrained_: Like the above, except you weight by probabilities, and drop the least likely formulas up to a parameter δ.
4. _Least regret_: Like the above, with δ set to zero.

Intuitively, the _Most likely_ criterion won’t be very robust since it is only taking one specification into account, _Most coverage_ is aiming for maximum robustness, _Chance constrained_ interpolates, where larger δ corresponds to trading robustness for gain in ability. This is exactly the pattern we see in a task where a robot must set a dinner table.


I am definitely in favor of considering the impacts of ML research before conducting or publishing it. I think the field is currently either at or near a threshold where papers will start having significant real world effects. While I don’t think this requirement will be sufficient for ensuring positive outcomes, I am glad NeurIPS is trying it out.

I think the article makes very strong points and will improve the quality of the impact statements that get submitted. I particularly liked the point about communicating uncertainty, which is a norm that I think the ML community would benefit from greatly. One thing I would add here is that giving explicit probabilities is often more helpful than vague words like “might” or “could”.

A Guide to Writing the NeurIPS Impact Statement

Carolyn Ashurst, Markus Anderljung, Carina Prunkl, Jan Leike, Yarin Gal, Toby Shevlane, Allan Dafoe

NeurIPS 2020 requires paper submissions to include a statement on the broader impact of their work. This post provides a guide for how to write an effective impact statement. They recommend focusing on the most significant, neglected, and tractable impacts, both positive and negative, while also conveying the uncertainties involved. They also suggest integrating this into the research process by reading the tech governance literature and building institutional structures, and including this information in introductions.

Their guide then recommends considering 3 questions:
How could your research affect ML applications?
What are the societal implications of these applications?
What research or other initiatives could improve social outcomes?

There is more information in the guide on how to go about answering those questions, along with some examples.


It is definitely worth keeping these distinctions in mind whenever talking about alignment. I personally tend to think about the motivation-based definitions, because those seem to be the most tractable definitions to work on, mainly because I don't have to worry about the AI being incompetent (eg. an AI launching nukes accidentally while exploring its action space). It seems possible to get strong arguments for intentional alignment and then use that with improved capabilities to argue for sufficient alignment.

Disambiguating "alignment" and related notions

Distinguishes between several kinds of alignment. Some focus on _terminal values_ from the AI, such as holistic alignment (the AI has the same terminal values as us) and parochial alignment (which I don't really understand, check the post). Sufficient alignment focuses on _outcomes_ (no X-event happens, or X-risk is sufficiently low). Finally, others focus on the _motivations_ of the AI, including intentional alignment (the AI tries to do what H wants it to do) and benign AI (R doesn't try to do what H doesn't want it to do).


This is an interesting position I haven't seen much in the West -- both that we can't build AGI, and that we shouldn't build it anyway. It's actually quite heartening to see an emphatic claim that we shouldn't build strong AI -- it seems like AI researchers as a group may in fact be able to coordinate to develop AI safely. Of course, this is a single viewpoint and is not representative of all AI researchers in China.

On Strong Artificial Intelligence

This article, written by a professor from China, argues that the AI community has never been focused on "strong AI", and we have no real path forward to building "strong AI", and that it would be so dangerous that we should never do research around it. The concept of "strong AI" here is a bit different from what we are used to -- I would probably call it human-like AGI, in that it would have consciousness, self-awareness, and emotions, and be as capable as a human.


I agree that this sort of "outside-view" reasoning seems good to have. In cases where we want our agent to be deferential even in a new situation where there isn't an outside view to defer to, the agent would have to construct this outside view via simulation, which would probably be infeasibly computationally expensive. Nonetheless, this seems like a cool perspective and I'd like to see a more in-depth take on the idea.

Corrigibility as outside view

This post proposes thinking of the outside view as an aspect of <@corrigible@>(@Corrigibility@) reasoning. In particular, before an agent takes an action that it believes is right, it can simulate possible overseers with different values, and see whether the reasoning that led to this action would do the right thing in those situations as well. The agent should then only take the action if the action usually turns out well.

This is similar to how we might reason that it wouldn't be good for us to impose the rules we think would be best for everyone, even if we had the power to do so, because historically every instance of this happening has actually been bad.


Clarifying The Malignity of the Universal Prior: The Lexical Update


Focusing on the impacts on society instead of specific features of AI systems makes sense and I do believe that the shape of RTAI as well as the risks it poses will depend on the way we handle TAI at various levels. More precise terminology can also help to prevent misunderstandings, for example between people forecasting AI and decision makers.

Defining and Unpacking Transformative AI

The notion of **transformative AI** (TAI) is used to highlight that even narrow AI systems can have large impacts on society. This paper offers a clearer definition of TAI and distinguishes it from **radical transformative AI** (RTAI).

"Discontinuities or other anomalous patterns in metrics of human progress, as well as *irreversibility*  are common indicators of transformative change. TAI is then broadly defined as an AI technology, which leads to an irreversible change of some important aspects of society, making it a (multi-dimensional) spectrum along the axes of **extremity**, **generality** and **fundamentality**. " For example, advanced AI weapon systems might have strong implications for great power conflicts but limited effects on people's daily lives; extreme change of limited generality, similar to nuclear weapons. There are two levels: while TAI is comparable to general-purpose technologies (GPTs) like the internal combustion engine, RTAI leads to changes that are comparable to the agricultural or industrial revolution. Both revolutions have been driven by GPTs like the domestication of plants and the steam engine. Similarly, we will likely see TAI before RTAI. The scenario where we don't is termed a **radical shift**.

Non-radical TAI could still contribute to existential risk in conjunction with other factors. Furthermore, if TAI precedes RTAI, our management of TAI can affect the risks RTAI will pose.


Six AI Risk/Strategy Ideas

This post briefly presents three ways that power can become centralized in a world with <@Comprehensive AI Services@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@), argues that under risk aversion "logical" risks can be more concerning than physical risks because they are more correlated, proposes combining human imitations and oracles to remove the human in the loop and become competitive, and suggests doing research to generate evidence of difficulty of a particular strand of research.


Self-Fulfilling Prophecies Aren't Always About Self-Awareness

Could we prevent a superintelligent oracle from making self-fulfilling prophecies by preventing it from modeling itself? This post presents three scenarios in which self-fulfilling prophecies would still occur. For example, if instead of modeling itself, it models the fact that there's some AI system whose predictions frequently come true, it may try to predict what that AI system would say, and then say that. This would lead to self-fulfilling prophecies.


[Analysing: Dangerous messages from future UFAI via Oracles](https://www.alignmentforum.org/posts/6WbLRLdmTL4JxxvCq/analysing-dangerous-messages-from-future-ufai-via-oracles) and [Breaking Oracles: hyperrationality and acausal trade](https://www.alignmentforum.org/posts/42z4k8Co5BuHMBvER/breaking-oracles-hyperrationality-and-acausal-trade)

These posts point out a problem with <@counterfactual oracles@>(@Self-confirming prophecies, and simplified Oracle designs@): a future misaligned agential AI system could commit to helping the oracle (e.g. by giving it maximal reward, or making its predictions come true) even in the event of an erasure, as long as the oracle makes predictions that cause humans to build the agential AI system. Alternatively, multiple oracles could acausally cooperate with each other to build an agential AI system that will reward all oracles.


I certainly agree that *defining* "conceptual" distance functions is hard. It has similar problems to saying "write down a utility function that captures human values" -- it's possible in theory but in practice we're not going to think of all the edge cases. However, it seems possible to learn distance functions rather than defining them; this is already done in perception and state estimation.

Distance Functions are Hard

Many ideas in AI alignment require some sort of distance function. For example, in [Functional Decision Theory](https://arxiv.org/abs/1710.05060), we'd like to know how "similar" two algorithms are (which can influence whether or not we think we have "logical control" over them). This post argues that defining such distance functions is hard, because they rely on human concepts that are not easily formalizable, and the intuitive mathematical formalizations usually have some flaw.


AI Alignment Podcast: On Consciousness, Qualia, and Meaning

Lucas Perry, Mike Johnson and Andrés Gómez Emilsson


[Oracles, sequence predictors, and self-confirming predictions](https://www.alignmentforum.org/posts/i2dNFgbjnqZBfeitT/oracles-sequence-predictors-and-self-confirming-predictions) ([AN #55](https://mailchi.mp/405e29e1f1cd/an-55-regulatory-markets-and-international-standards-as-a-means-of-ensuring-beneficial-ai)) and [Good and safe uses of AI Oracles](https://arxiv.org/abs/1711.05541)

Self-confirming prophecies, and simplified Oracle designs

This post presents a toy environment to model self-confirming predictions by oracles, and demonstrates the results of running a deluded oracle (that doesn't realize its predictions affect the world), a low-bandwidth oracle (that must choose from a small set of possible answers), a high-bandwidth oracle (that can choose from a large set of answers) and a counterfactual oracle (that chooses the correct answer, _conditional_ on us not seeing the answer).


Inquiry: An Interdisciplinary Journal of Philosophy

While this doesn't mention AI explicitly, I think it's useful to read anyway, because often which of the five concepts you use will affect what you think the important risks are.

Existential Risks: A Philosophical Analysis

The phrase "existential risk" is often used in different ways. This paper considers the pros and cons of five different definitions.


To first order, moral realism and moral anti-realism are the same thing


Reinforcement learning with imperceptible rewards

Typically in reinforcement learning, the reward function is defined over _observations_ and actions, rather than directly on states, which ensures that the reward can always be calculated. However, in reality **we care about underlying aspects of the state that may not easily be computed from observations**. We can't guarantee sublinear regret, since if you are unsure about the reward in some unobservable part of the state that your actions nonetheless affect, then you can never learn the reward and approach optimality.

To fix this, we can work with rewards that are restricted to _instrumental states_ only. I don't understand exactly how these work, since I don't know the math used in the formalization, but I believe the idea is for the set of instrumental states to be defined such that for any two instrumental states, there exists some "experiment" that the agent can run in order to distinguish between the states in some finite time. The main point of this post is that we can establish a regret bound for MDPs (not POMDPs yet), assuming that there are no traps.


AAMAS '18 Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems

Human-Centered Artificial Intelligence and Machine Learning


Typically, if you argue that a simulation of you that thinks longer can't solve some problem X, the natural response is that that implies you couldn't solve X either. However, the differences between the simulation environment and real environment could make it be the case that in reality you could solve a problem that you couldn't in simulation (e.g. imagine the simulation didn't have access to the Internet). This suggests that if you wanted to do this you'd have to set up the simulation very carefully.

Would I think for ten thousand years?

Many ideas in AI safety involve delegating key decisions to simulations that can think longer. This post points out that you need to worry about value drift and other unforeseen problems in this situation. The comments also point out that there will likely be differences between the simulation and the real world that could be problematic (e.g. what prevents the humans from going crazy from isolation?)


Can there be an indescribable hellworld?

This short post argues that it's always possible to explain why any given undesirable outcome doesn't satisfy our values (even if that explanation needs to be at a very high level), and so being able to make superintelligences debate in a trustworthy way is sufficient to make them safe.


I think this argument applies well to the case where we are trying to _communicate_, but not so much to the case where I individually am thinking about a problem. (I'm making this claim about me specifically; I don't know if it generalizes to other people.) Communication is hard and if the speaker uses some intuitive assumption, chances are the listener will interpret it differently from what the speaker intended, and so being very precise seems quite helpful. However, when I'm thinking through a problem myself and I make an assumption, I usually have a fairly detailed intuitive model of what I mean, such that if you ask me whether I'm assuming that problem X is solved by the assumption, I could answer that, even though I don't have a precise formulation of the assumption. Making the assumption more precise would be quite a lot of work, and probably would not improve my thinking on the topic that much, so I tend not to do it until I think there's some insight and want to make the argument more rigorous. It seems to me that this is how most research progress happens: by individual researchers having intuitions that they then make rigorous and precise.

Assuming we've solved X, could we do Y...

We often want to make assumptions that sound intuitive but that we can't easily formalize, eg. "assume we've solved the problem of determining human values". However, such assumptions can often be interpreted as being very weak or very strong, and depending on the interpretation we could be assuming away the entire problem, or the assumption doesn't buy us anything. So, we should be more precise in our assumptions, or focus on only on some precise _properties_ of an assumption.


I think moral uncertainty is an important and overdue idea in ethics. I also agree that the idea of extrapolating an idealised form of people's preferences is not well-defined. However, I'm very skeptical about Will's arguments about moral realism. In particular, I think that saying that nothing matters at all without moral realism is exactly the sort of type error which Eliezer argued against [here](https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside).

I'm more sympathetic to the idea that we should have a period of long reflection before committing to actions on an astronomical scale; this seems like a good idea if you take moral uncertainty at all seriously.

AI Alignment Podcast: Moral Uncertainty and the Path to AI Alignment

Initially, Will articulates arguments for moral realism (the idea that there are objectively true moral facts) and moral uncertainty (the idea that we should assign credences to different moral theories being correct). Later, the discussion turns to the relevance of these views to AI safety. Will distinguishes the control problem (ensuring AIs do what we say), from the problem of aligning AI with human values, from the problem of aligning AI with moral truth. Observing humans isn't sufficient to learn values, since people can be self-destructive or otherwise misguided. Perhaps AI could extrapolate the values an idealised version of each person would endorse; however, this procedure seems under-defined.

On the moral truth side, Will worries that most educated people are moral relativists or subjectivists and so they won't sufficiently prioritise aligning AI with moral truth. He advocates for a period of long philosophical reflection once we've reduced existential risk to near zero, to figure out which future would be best. Careful ethical reasoning during this period will be particularly important since small mistakes might be magnified massively when implemented on an astronomical scale; however, he acknowledges that global dynamics make such a proposal unlikely to succeed. On a brighter note, AGI might make great advances in ethics, which could allow us to make the future much more morally valuable.


EuroSys '18 Proceedings of the Thirteenth EuroSys Conference, Article No. 18

Does Your Model Know the Digit 6 Is Not a Cat? A Less Biased Evaluation of "Outlier" Detectors

Alireza Shafaei, Mark Schmidt, James J. Little


Intuitively, this type says that given something that maps A to B, you can get an A. If you think of (A -> B) as an environment where A is the action and B is the effect, then this type is a function that says which actions you should take. Note that the goal, which would be an element of B, is not passed in explicitly, so the goal must be inside of the function of type (A -> B) -> A, similar to our typical views of what an "agent" is. If you only assume that you know what A and B are, but you know nothing else, to do anything interesting you would be doing black box optimization -- that is, you get some f of type (A -> B) that you know nothing about, and so you just keep computing f(a) for different a ∈ A, looking for a particular b ∈ B. Perhaps you build a model of the function f and then do something more abstract with your model to find a good a ∈ A. (The post mentions a similar idea by noting that argmax has this type signature.) The post also has some thoughts about game theory that are interesting.

(A -> B) -> A

This blog post has some thoughts on the type (A -> B) -> A, which can be thought of as the type of an agent. Rather than summarize the post, which seems hard, I'm going to say things about this type inspired by the post, and then you can decide whether to read the post.


Key Concepts in AI Safety

This overview from CSET gives a brief introduction to AI safety using the <@specification, robustness, and assurance (SRA) framework@>(@Building safe artificial intelligence: specification, robustness, and assurance@). Follow-up reports cover [interpretability](https://cset.georgetown.edu/research/key-concepts-in-ai-safety-interpretability-in-machine-learning/) and [adversarial examples / robustness](https://cset.georgetown.edu/research/key-concepts-in-ai-safety-robustness-and-adversarial-examples/). I don’t expect these to be novel to readers of this newsletter -- I include them in case anyone wants a brief overview, as well as to provide links to AI safety reports that will likely be read by government officials.


Epistemological Framing for AI Alignment Research

This post recommends that we think about AI alignment research in the following framework:

1. Defining the problem and its terms: for example, we might want to define “agency”, “optimization”, “AI”, and “well-behaved”.
2. Exploring these definitions, to see what they entail.
3. Solving the now well-defined problem.

This is explicitly _not_ a paradigm, but rather a framework in which we can think about possible paradigms for AI safety. A specific paradigm would choose a specific problem formulation and definition (or at least something significantly more concrete than “solve AI safety”). However, we are not yet sufficiently deconfused to be able to commit to a specific paradigm; hence this overarching framework.


The Challenge of Value Alignment: from Fairer Algorithms to AI Safety

This book chapter provides an introduction to AI alignment from a philosophical lens.


The date of AI Takeover is not the day the AI takes over

This post points out that when making decisions based on AGI timelines, the relevant date is not when an AI would actually take over the world, but instead the last point at which we could have done anything about it.


Knowledge, manipulation, and free will

This post considers the concepts of free will, manipulation, and coercion in the setting where we have a superintelligent AI system that is able to predict human behavior very accurately. The main point I’d highlight is that the concept of manipulation seems pretty hard to pin down, since anything the AI system does probably does affect the human in some way that the AI system could predict and so could count as “manipulation”.


Locality of goals

This post introduces the concept of the _locality_ of a goal, that is, how “far” away the target of the goal is. For example, a thermometer’s “goal” is very local: it “wants” to regulate the temperature of this room, and doesn’t “care” about the temperature of the neighboring house. In contrast, a paperclip maximizer has extremely nonlocal goals, as it “cares” about paperclips anywhere in the universe. We can also consider whether the goal depends on the agent’s internals, its input, its output, and/or the environment.

The concept is useful because for extremely local goals (usually goals about the internals or the input) we would expect wireheading or tampering, whereas for extremely nonlocal goals, we would instead expect convergent instrumental subgoals like resource acquisition.


This seems reasonable to me as an aspect of goal-directedness. Note that it is not a sufficient condition. For example, the policy that always chooses action A has extremely low complexity, but I would not call it goal-directed.

Goals and short descriptions

This post argues that a distinguishing factor of goal-directed policies is that they have low Kolmogorov complexity, relative to e.g. a lookup table that assigns a randomly selected action to each observation. It then relates this to [quantilizers](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) ([AN #48](https://mailchi.mp/3091c6e9405c/alignment-newsletter-48)) and <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@).


I certainly agree that we should be thinking about how we can convince key decision makers of the level of risk of the systems they are building (whatever that level of risk is). I think that on the current margin it's much more likely that this is best done through better estimation and explanation of risks with AI systems, but it seems likely that the interventions laid out here will become more important in the future.

Preparing for "The Talk" with AI projects

At some point in the future, it seems plausible that there will be a conversation in which people decide whether or not to deploy a potentially risky AI system. So one class of interventions to consider is interventions that make such conversations go well. This includes raising awareness about specific problems and risks, but could also include identifying people who are likely to be involved in such conversations _and_ concerned about AI risk, and helping them prepare for such conversations through training, resources, and practice. This latter intervention hasn't been done yet: some simple examples of potential interventions would be generating official lists of AI safety problems and solutions which can be pointed to in such conversations, or doing "practice runs" of these conversations.


Bootstrapped Alignment

This post distinguishes between three kinds of “alignment”:
1. Not building an AI system at all,
2. Building Friendly AI that will remain perfectly aligned for all time and capability levels,
3. _Bootstrapped alignment_, in which we build AI systems that may not be perfectly aligned but are at least aligned enough that we can use them to build perfectly aligned systems.

The post argues that optimization-based approaches can’t lead to perfect alignment, because there will always eventually be Goodhart effects.


Against the Linear Utility Hypothesis and the Leverage Penalty

More on the Linear Utility Hypothesis and the Leverage Prior


Formal Metaethics and Metasemantics for AI Alignment

This website presents in great detail a process by which an agent might use data from human brains in order to infer a utility function for a single human (also spelling out what assumptions need to be made along the way), and then how it could combine the utility functions from different humans to arrive at "a fully technical ethical goal function". Emphasis is placed on solving the philosophical problems of metaethics and mental content. Quoting the website, they "suppose that unlimited computation and a complete low-level causal model of the world and the adult human brains in it are available".


Approaches to Deploying a Safe Artificial Moral Agent

This post investigates which of the current moral theories would most reduce existential risk if we programmed it into an AI system, and settles on Aristotelian virtue ethics (over utilitarianism and Kant's categorical imperative).


International Conference on Computer Safety, Reliability, and Security 2018

The analogy is interesting but I'm not sure what to take away from the paper, and I think there are also big disanalogies. The biggest one is that we have to communicate our goals to an AI, whereas humans come equipped with some goals from birth (though arguably most of our goals come from the environment we grow up in). I'd be interested in seeing future work from this agenda, since I don't know how I could do work on the agenda laid out in this paper.

A Psychopathological Approach to Safety Engineering in AI and AGI

Vahid Behzadan, Arslan Munir, Roman V. Yampolskiy

Since AGI research aims for cognitive functions that are similar to humans, they will be vulnerable to similar psychological issues. Some problems can be recast in this light -- for example, wireheading can be thought of as delusional or addictive behavior. This framework suggests new solutions to AI safety issues -- for example, analogous to behavioral therapy, we can retrain a malfunctioning agent in controlled environments to remove the negative effects of earlier experiences.


[The Unavoidable Problem of Self-Improvement in AI](https://futureoflife.org/2019/03/19/the-unavoidable-problem-of-self-improvement-in-ai-an-interview-with-ramana-kumar-part-1/) and [The Problem of Self-Referential Reasoning in Self-Improving AI](https://futureoflife.org/2019/03/21/the-problem-of-self-referential-reasoning-in-self-improving-ai-an-interview-with-ramana-kumar-part-2/)

These articles introduce the thinking around AI self-improvement, and the problem of how to ensure that future, more intelligent versions of an AI system are just as safe as the original system. This cannot be easily done in the case of proof-based systems, due to Godel's incompleteness theorem. Some existing work on the problem: [Botworld](https://intelligence.org/files/Botworld.pdf), [Vingean reflection](https://intelligence.org/files/VingeanReflection.pdf), and [Logical induction](https://intelligence.org/files/LogicalInduction.pdf).


These criteria are theoretically quite nice, but I'm not sure how they relate to the broader picture. Is the hope that we will be able to elicit the causal influence diagram an AI system is using, or something like it? Or perhaps that we will be able to create a causal influence diagram of the environment, and these criteria can tell us which nodes we should be particularly interested in? Maybe the goal was simply to understand agent incentives better, with the expectation that more knowledge would help in some as-yet-unknown way? None of these seem very compelling to me, but the authors might have something in mind I haven't thought of.

Understanding Agent Incentives with Causal Influence Diagrams

This post and associated paper model an agent's decision process using a causal influence diagram -- think of a Bayes net, and then imagine that you add nodes corresponding to actions and utilities. A major benefit of Bayes nets is that the criterion of d-separation can be used to determine whether two nodes are conditionally independent. Once we add actions and utilities, we can also analyze whether observing or intervening on nodes would lead the agent to achieve higher expected utility. The authors derive criteria resembling d-separation for identifying each of these cases, which they call observation incentives (for nodes whose value the agent would like to know) and intervention incentives (for nodes whose value the agent would like to change). They use observation incentives to show how to analyze whether a particular decision is fair or not (that is, whether it depended on a sensitive feature that should not be used, like gender). Intervention incentives are used to establish the security of [counterfactual oracles](https://arxiv.org/abs/1711.05541) more simply and rigorously.


Beliefs at different timescales


Misbehaving AIs can't always be easily stopped!


I've written enough opinions about this version of corrigibility, I think I'd just be repeating myself. You can look through Stuart's recent posts and find my comments there if you really want (eg. [here](https://www.lesswrong.com/posts/T5ZyNq3fzN59aQG5y/the-limits-of-corrigibility)).

Petrov corrigibility

There can be situations where a human asks an AI for guidance, and the AI's action then determines what the human's preferences are. In such a situation, taking either action is incorrigible and so the AI should say "there is no corrigible action to take". But what if saying that would itself predictably change the human's decision? In that case, even that would not be corrigible.

There is also a comment chain discussing whether how this notion of corrigibility/alignment differs from the notion Paul Christiano talks about [here](https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6).


I take a very different approach to AI safety, so it was nice to read a summary of what other people are thinking about. Based on the summaries, it sounds like most of the essays that focused on AGI were anthropomorphizing AGI more than I would like (though of course I haven't actually read the book).

Book Review: AI Safety and Security

A review of the new AI Safety and Security book. It goes through each of the papers, giving a short summary of each and some comments (similar to this newsletter).


It's not obvious to me, even on the "optimizing an abstract computational process" model, why an AI would not want get more compute -- it can use this compute for itself, without changing the abstract computational process it is optimizing, and it will probably do better. It seems that if you want to get this to work, you need to have the AI want to compute the result of running _itself_ without any modification or extra compute on the virtual world. This feels very hard to me. Separately, I also find it hard to imagine us building a virtual world that is similar enough to the real world that we are able to transfer solutions between the two, even with some finetuning in the real world.

Safely and usefully spectating on AIs optimizing over toy worlds

One way to achieve safety would be to build an AI that optimizes in a virtual world running on a computer, and doesn't care about the physical world. Even if it realizes that it can break out and eg. get more compute, these sorts of changes to the physical world would not be helpful for the purpose of optimizing the abstract computational object that is the virtual world. However, if we take the results of the AI and build them in the real world, that causes a distributional shift from the toy world to the real world that could be catastrophic. For example, if the AI created another agent in the toy world that did reasonable things in the toy world, when we bring it to the real world it may realize that it can instead manipulate humans in order to do things.


Exploring AI Safety in Degrees: Generality, Capability and Control

This paper argues that we should decompose the notion of “intelligence” in order to talk more precisely about AI risk, and in particular suggests focusing on _generality_, _capability_, and _control_. We can think of capability as the expected performance of the system across a wide variety of tasks. For a fixed level of capability, generality can be thought of as how well the capability is distributed across different tasks. Finally, control refers to the degree to which the system is reliable and deliberate in its actions. The paper qualitatively discusses how these characteristics could interact with risk, and shows an example quantitative definition for a simple toy environment.


Guide Me: Interacting with Deep Networks

Christian Rupprecht, Iro Laina, Nassir Navab, Gregory D. Hager, Federico Tombari


Wireheading is in the eye of the beholder


Three ways that "Sufficiently optimized agents appear coherent" can be false

This post talks about three ways that agents could not appear coherent, where here "coherent" means "optimizing for a _reasonable_ goal". First, if due to distributional shift the agent is put into situations it has never encountered before, it may not act coherently. Second, we may want to "force" the agent to pretend as though compute is very expensive, even if this is not the case, in order to keep them bounded. Finally, we may explicitly try to keep the agent incoherent -- for example, population ethics has impossibility results that show that any coherent agent must bite some bullet that we don't want to bite, and so we may instead elect to keep the agent incoherent instead. (See <@Impossibility and Uncertainty Theorems in AI Value Alignment@>(@Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)@).)


I'm not convinced that we should be thinking a lot more about moral philosophy. While I agree that locking in a set of values would likely be quite bad, I think this means that researchers should not hardcode a set of values, or create an AI that infers some values and then can never change them. It's not clear to me why studying more moral philosophy helps us with this goal. For the other points, it seems not too important to get preference aggregation or particular strategic approaches exactly perfect as long as we don't lock in values -- as an analogy, we typically don't argue that politicians should be experts on moral philosophy, even though they aggregate preferences and have large impacts on society.

AI Alignment Podcast: On Becoming a Moral Realist

There's a fair amount of complexity in this podcast, and I'm not an expert on moral philosophy, but here's an *oversimplified* summary anyway. First, in the same way that we can reach mathematical truths through reason, we can also arrive at moral truths through reason, which suggests that they are true facts about the universe (a moral realist view). Second, preference utilitarianism has the problem of figuring out which preferences you want to respect, which isn't a problem with hedonic utilitarianism. Before and after the interview, Lucas argues that moral philosophy is important for AI alignment. Any strategic research "smuggles" in some values, and many technical safety problems, such as preference aggregation, would benefit from a knowledge of moral philosophy. Most importantly, given our current lack of consensus on moral philosophy, we should be very wary of locking in our values when we build powerful AI.


Building Safer AGI by introducing Artificial Stupidity


Sandboxing by Physical Simulation?


AAAI: AI and Society: Ethics, Safety and Trustworthiness in Intelligent Agents

Papers from AI and Society: Ethics, Safety and Trustworthiness in Intelligent Agents


The Blessings of Multiple Causes


How artificial intelligence is changing science

AI is being used in many different projects across many different fields at Stanford. This post has a list of a whole bunch of scientific projects that AI is helping with.


Talk to Books


While this isn't very relevant to technical alignment, it does seem important to have more rigor in ML, since ML researchers are likely to be the ones building advanced AI.

Winner's Curse?

D. Sculley, Jasper Snoek, Ali Rahimi, Alex Wiltschko

A short paper arguing that we need more empirical rigor in ML, identifying some structural incentives that push against this and suggesting solutions.


This is a fairly complicated paper, but the authors do a reasonable job of organizing the presentation of results. In particular, the analysis of agent behavior and their neural representations is well done. At a higher level, I found it interesting that the authors partially reject the idea of evaluating agents with just expected returns. I broadly agree with the authors that the evaluation of agents across multi-player tasks is an open problem without an immediate solution. With respect to agent capability, I found the section on experimentation to be most interesting. In particular, I look forward to seeing more research on how attention mechanisms catalyze such behavior.

**Rohin's opinion:** One of my models about deep learning is “Diversity is all you need”. Suppose you’re training for some task for which there’s a relevant feature F (such as the color of the goal pyramid). If F only ever takes on a single value in your training data (you only ever go to yellow pyramids), then the learned model can be specialized to that particular value of F, rather than learning a more general computation that works for arbitrary values of F. Instead, you need F to vary a lot during training (consider pyramids that are yellow, blue, green, red, orange, black, etc) if you want your model to generalize to new values of F at test time. That is, your model will be zero-shot robust to changes in a feature F if and only if your training data was diverse along the axis of feature F.  (To be clear, this isn’t literally true, it is more like a first-order main effect.)

Some evidence supporting this model:
- The approach in this paper explicitly has diversity in the objective and the world, and so the resulting model works zero-shot on new objectives of a similar type and can be finetuned quickly.
- In contrast, the similar <@hide and seek project@>(@Emergent Tool Use from Multi-Agent Interaction@) did not have diversity in the objective, had distinctly less diversity in the world, and instead got diversity from emergent strategies for multiagent interaction (but there were fewer than 10 such strategies). Correspondingly, the resulting agents could not be quickly finetuned.
- My understanding is that in image recognition, models trained on larger, more diverse datasets become significantly more robust.

Based on this model, I would make the following predictions about agents in XLand:
- They will not generalize to objectives that can’t be expressed in the predicate language used at training time, such as “move all the pyramids near each other”. (In some sense this is obvious, since the agents have never seen the word “all” and so can’t know what it means.)
- They will not work in any environment outside of XLand (unless that environment looks very very similar to XLand).
In particular, I reject the idea that these agents have learned “general strategies for problem solving” or something like that, such that we should expect them to work in other contexts as well, perhaps with a little finetuning. I think they have learned general strategies for solving a specific class of games in XLand.

You might get the impression that I don’t like this research. That’s not the case at all — it is interesting and impressive, and it suggests that we could take the same techniques and apply them in broader, more realistic domains where the resulting agents could be economically useful. Rather, I expect my readership to overupdate on this result and think that we’ve now reached agents that can do “general planning” or some such, and I want to push against that.

Generally capable agents emerge from open-ended play

Open-Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, Wojciech Marian Czarnecki

Artificial intelligence agents have become successful at games when trained for each game separately. However, it has proven challenging to build agents that can play _previously unseen_ games. This paper makes progress on this challenge in three primary areas: creating rich simulated environments and tasks, training agents with attention mechanisms over internal states, and evaluating agents over a variety of games. The authors show that agents trained with goal-based attention in their proposed environment (XLand) succeed at a range of novel, unseen tasks with no additional training required. Moreover, such agents appear to use general tactics such as decision-making, tool use, and experimentation during game-play episodes.

The authors argue that training-data generation is a central challenge to training general RL agents (an argument we’ve seen before with <@POET@>(@Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions@) and <@PAIRED@>(@Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design@)). They propose the training environment XLand to address this. XLand includes many multiplayer games within consistent, human-relatable 3D worlds and allows for dynamic agent learning through the procedural generation of tasks which are split into three components: world, agents, and goals. The inclusion of other agents makes this a partially observable environment. Goals are defined with Boolean formulas. Each goal is a combination of options and every option is a combination of atomic predicates. For example, in hide-and-seek one player has the goal see(me,opponent) and the other player not(see(opponent,me)). The space of worlds and games are shown to be both vast and smooth, which supports training.

The agents themselves are trained using deep-RL combined with a goal-attention module (GOAT). The per-timestep observations of the agent are ego-centric RGB images, proprioception values indicating forces, and the goal of the agent. The GOAT works by processing this information with a recurrent network and then using a goal-attention module to select hidden states that are most relevant to achieving a high return. This is determined by estimating the expected return if the agent focused on an option until the end of the episode.

As with many other major deep-RL projects, it is important to have a good curriculum, where more and more challenging tasks are introduced over time. The obvious method of choosing tasks with the lowest reward doesn’t work, because the returns from different games are non-comparable. To address this, an iterative notion of improvement is proposed, and scores are given as percentiles relative to a population. This is similar in spirit to the <@AlphaStar League@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@). Following this, game-theoretic notions such as Pareto dominance can be used to compare agents and to determine how challenging a task is, which can then be used to create a curriculum.

Five generations of agents are trained, each of which is used in the next generation to create opponents and relative comparisons for defining the curriculum. Early in training, the authors find that adding an intrinsic reward based on self-play is important to achieve good performance. This encourages agents to achieve non-zero rewards in as many games as possible, which the authors call “participation”. The authors also conduct an ablation study and find that dynamic task generation, population-based methods, and the GOAT module have a significant positive impact on performance.

The agents produced during training have desirable generalization capability. They can compete in games that were not seen before in training. Moreover, fine-tuning dramatically improves the performance of agents in tasks where training from scratch completely fails. A number of case studies are also presented to explore emergent agent behavior. In one experiment, an agent is asked to match a colored shape and another environment feature such as a shape or floor panel. At the start of the episode, the agent decides to carry a black pyramid to an orange floor, but then after seeing a yellow sphere changes options and places the two shapes together. This shows that the agent has robust option evaluation capability. In other experiments, the agents show the capacity to create ramps to move to higher levels in the world environment. Additionally, agents seem capable of experimentation. In one instance, the agent is tasked with producing a specific configuration of differently colored cube objects. The agent demonstrates trial-and-error and goes through several different configurations until it finds one it evaluates highly.

There are limitations to the agent capabilities. While agents can use ramps in certain situations they fail to use ramps more generally. For example, they frequently fail to use ramps to cross gaps. Additionally, agents generally fail to create more than a single ramp. Agents also struggle to play cooperative games involving following not seen during training. This suggests that experimentation does not extend to co-player behavior. More broadly, whether or not co-player agents decide to cooperate is dependent on the population the agents interacted with during training. In general, the authors find that agents are more likely to cooperate when both agents have roughly equal performance or capability.


You might wonder why Melting Pot is just used for evaluation, rather than as a training suite: given the diversity of scenarios in Melting Pot, shouldn’t you get similar benefits as in the highlighted post? The answer is that there isn’t nearly enough diversity, as there are only ~100 scenarios across ~20 substrates. For comparison, <@ProcGen@>(@Procgen Benchmark@) requires 500-10,000 levels to get decent generalization performance. Both ProcGen and XLand are more like a single substrate for which we can procedurally generate an unlimited number of scenarios. Both have _more_ diversity than Melting Pot, in a _narrower_ domain; this is why training on XLand or ProcGen can lead to good generalization but you wouldn’t expect the same to occur from training on Melting Pot.

Given that you can’t get the generalization from simply training on something similar to Melting Pot, the generalization capability will instead have to come from some algorithmic insight or by finding some other way of pretraining agents on a wide diversity of substrates and scenarios. For example, you might figure out a way to procedurally generate a large, diverse set of possible background agents for each of the substrates.

(If you’re wondering why this argument doesn’t apply to supervised learning, it’s because in supervised learning the training set has many thousands or even millions of examples, sampled from the same distribution as the test set, and so you have the necessary diversity for generalization to work.)

Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot

Joel Z. Leibo*, Edgar Duéñez-Guzmán*, Alexander Sasha Vezhnevets*, John P. Agapiou*, Peter Sunehag, Raphael Koster, Jayd Matyas, Charles Beattie, Igor Mordatch, Thore Graepel

In supervised learning, the test dataset is different from the training dataset, and thus evaluates how well the learned model generalizes (within distribution). So far, we mostly haven't done this with reinforcement learning: the test environment is typically identical to the training environment. This is because it would be very challenging -- you would have to design a large number of environments and then split them into a train and test set; each environment would take a very long time to create (unlike in, say, image classification, where it takes a few seconds to label an image).

The core insight of this paper is that when evaluating a multiagent RL algorithm, you can get a “force multiplier” by taking a single multiagent environment (called a “substrate”) and “filling in” some of the agent slots with agents that are automatically created using RL to create a “scenario” for evaluation. For example, in the <@Capture the Flag substrate@>(@Capture the Flag: the emergence of complex cooperative agents@), in one scenario we fill in all but one of the agents using agents trained by A3C, which means that the remaining agent (to be supplied by the algorithm being evaluated) must cooperate with previously-unseen agents on its team, to play against previously-unseen opponents. Scenarios can fall in three main categories:

1. **Resident mode:** The agents created by the multiagent RL algorithm under evaluation outnumber the background “filled-in” agents. This primarily tests whether the agents created by the multiagent RL algorithm can cooperate with each other, even in the presence of perturbations by a small number of background agents.
2. **Visitor mode:** The background agents outnumber the agents created by the algorithm under evaluation. This often tests whether the new agents can follow existing norms in the background population.
3. **Universalization mode:** A _single_ agent is sampled from the algorithm and used to fill _all_ the slots in the substrate, effectively evaluating whether the policy is universalizable.

The authors use this approach to create Melting Pot, a benchmark for evaluating multiagent RL algorithms that can produce populations of agents (i.e. most multiagent RL algorithms). Crucially, the algorithm being evaluated is _not_ permitted to see the agents in any specific scenario in advance; this is thus a test of generalization to new opponents. (It is allowed unlimited access to the substrate.) They use ~20 different substrates and create ~5 scenarios for each substrate, giving a total of ~100 scenarios on which the multiagent RL algorithm can be evaluated. (If you exclude the universalization mode, which doesn’t involve background agents and so may not be a test of generalization, then there are ~80 scenarios.) These cover both competitive, collaborative, and mixed-motive scenarios.


Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design

Michael Dennis*, Natasha Jaques*, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, Sergey Levine

One argument for AI risk is that we have to specify some aspects of the training procedure, and if these are poorly specified, then bad outcomes may result. Typically we think of bad specification of the reward function as the risk, but this can also apply to environments: if we train a system in a simulated environment, then it may fail if the simulation is insufficiently similar to the real environment.

A typical approach would be _domain randomization_: we randomly vary some parameters that control the behavior of the environment. Unfortunately, this can often create environments that are too easy: in a maze environment, this approach often doesn’t have enough walls. Another approach could be to choose the environment adversarially, so that the agent learns the skills needed for hard environments. Unfortunately, this can often make the environment unsolvable: in the maze environment, the goal may be unreachable from the initial position.

The key idea of this paper is a method to create environments that are just on the edge of the agent’s abilities, by finding an environment that maximizes the _agent’s regret_: how poorly the agent performs, relative to how well _it could have done_. To operationalize how well the agent “could have done”, we also train an _antagonist agent_, and we then choose an environment that the antagonist performs well on but the protagonist performs poorly on. This results in environments that are solvable but challenging for the protagonist.


Call for contributors to the Alignment Newsletter

I'm looking for content creators and a publisher for this newsletter! Apply by September 6.


I'm excited for this forum, and will be collating the value learning sequence for its launch. Since these sequences are meant to teach some of the key ideas in AI alignment, I would probably end up highlighting every single post. Instead of that, I'm going to create new categories for each sequence and summarize them each week within the category, but *you should treat them as if I had highlighted them*.

Introducing the AI Alignment Forum (FAQ)

The Alignment Forum has officially launched! It aims to be the single online hub for researchers to have conversations about all the ideas in the field, while also helping new researchers get up to speed. While posting is restricted to members, all content is cross-posted to LessWrong, where anyone can engage with it. In addition, for the next few weeks there will be a daily post from one of three new sequences on embedded agency, iterated amplification, and value learning.


Announcement: AI alignment prize round 2 winners and next round

The winners of the second round of the AI alignment prize have been announced! All of the winners have already been sent out in this newsletter, except for the first place winner, "[The Alignment Problem for History-Based Bayesian Reinforcement Learners](http://www.tomeveritt.se/papers/alignment.pdf)". The deadline for the next iteration of the AI alignment prize is June 30, 2018.


Overall, I like these four directions and am excited to see what comes out of them! I'll comment on specific directions below.

Request for proposals for projects in AI alignment that work with deep learning systems

Open Philanthropy is seeking proposals for AI safety work in four major areas related to deep learning, each of which I summarize below. Proposals are due January 10, and can seek up to $1M covering up to 2 years. Grantees may later be invited to apply for larger and longer grants.


I'm very excited to see this newsletter: this is a category of papers that I want to know about and that are relevant to safety, but I don't have the time to read all of these papers given all the other alignment work I read, especially since I don't personally work in these areas and so often find it hard to summarize them or place them in the appropriate context. Dan on the other hand has written many such papers himself and generally knows the area, and so will likely do a much better job than I would. I recommend you subscribe, especially since I'm not going to send a link to each MLSN in this newsletter.

[MLSN #1]: ICLR Safety Paper Roundup

This is the first issue of the ML Safety Newsletter, which is "a monthly safety newsletter which is designed to cover empirical safety research and be palatable to the broader machine learning research community".


AI Safety Papers

AI Safety Papers (announced [here](https://www.alignmentforum.org/posts/GgusnG2tiPEa4aYFS/ai-safety-papers-an-app-for-the-tai-safety-database)) is an app to interactively explore a previously collected <@database of AI safety work@>(@TAI Safety Bibliographic Database@). I believe it contains every article in this newsletter (at least up to a certain date; it doesn’t automatically update) along with their summaries, so you may prefer to use that to search past issues of the newsletter instead of the [spreadsheet I maintain](https://docs.google.com/spreadsheets/d/1PwWbWZ6FPqAgZWOoOcXM8N_tUCuxpEyMbN1NYYC02aM/edit#gid=0).


DeepMind hiring Research Scientist, Safety

Career opportunity!


Announcement: AI alignment prize round 3 winners and next round

The winners of the third round of the [AI Alignment Prize](https://www.lesswrong.com/posts/SSEyiHaACSYDHcYZz/announcement-ai-alignment-prize-round-2-winners-and-next) have been announced! Vadim Kosoy wins the first prize of $7500 for [The Learning-Theoretic AI Alignment Research Agenda](https://agentfoundations.org/item?id=1816), and Alexander Turner wins the second prize of $2500 for [Worrying About the Vase: Whitelisting](https://www.lesswrong.com/posts/H7KB44oKoSjSCkpzL/worrying-about-the-vase-whitelisting) and [Overcoming Clinginess in Impact Measures](https://www.lesswrong.com/posts/DvmhXysefEyEvXuXS/overcoming-clinginess-in-impact-measures). The next round has started and will last until December 31, and each participant has been asked to submit a single entry (possibly in parts).


BERI seeking new university collaborators

[BERI](http://existence.org/faq) is expanding its offerings to provide free services to a wider set of university-affiliated groups and projects, and they’re now accepting applications from groups and individuals interested in receiving their support. If you’re a member of a research group, or an individual researcher, working on long-termist projects, you can [apply here](http://existence.org/apply).


FHI Summer Research Fellowship

This six week summer fellowship allows fellows to take the lead on a project relevant to the long-term future, working with an FHI Research Scholar. Application deadline is March 22.


Microsoft invests in and partners with OpenAI to support us building beneficial AGI

After moving to a <@capped-profit investment model@>(@OpenAI LP@), Microsoft has invested $1 billion in OpenAI. This allows OpenAI to keep their focus on developing and sharing beneficial AGI: instead of having to create a product to cover costs, they can license their pre-AGI technologies, likely through Microsoft.


Funding for Study and Training Related to AI Policy Careers

The Open Philanthropy Project has launched an AI policy scholarships program; the deadline for the first round is October 15.


SafeML Workshop: Accepted Papers

The camera-ready papers from the SafeML workshop are now available! There are a lot of good papers on robustness, adversarial examples, and more that will likely never make it into this newsletter (there's only so much I can read and summarize), so I encourage you to browse through it yourself.


Why the world’s leading AI charity decided to take billions from investors


Given the high cost of salaries and compute for machine learning research, I don't find this a particularly surprising development. I'd also note that, in the context of investing in a startup, a 100x return over a timeframe of decades is not actually that high.

OpenAI LP

OpenAI is transitioning to a new structure, consisting of a capped-profit company (OpenAI LP) controlled by the original OpenAI nonprofit organisation. The nonprofit is still dedicated to its charter, which OpenAI LP has a legal duty to prioritise. All investors must agree that generating profits for them is a secondary goal, and that their overall returns will be capped at 100x their investment (with any excess going back to the nonprofit).


It's great to see a center for AI policy that's run by a person who has wanted to consume AI policy analysis in the past (Jason Matheny was previously the director of IARPA). It's interesting to see the areas he focuses on in this Q&A -- it's not what I would have expected given my very little knowledge of AI policy.

Q&A with Jason Matheny, Founding Director of CSET

The [Center for Security and Emerging Technology](https://cset.georgetown.edu/) has been announced, with a [$55 million grant from the Open Philanthropy Project](https://www.openphilanthropy.org/giving/grants/georgetown-university-center-security-and-emerging-technology), and is [hiring](https://cset.georgetown.edu/careers/). While the center will work on emerging technologies generally, it will initially focus on AI, since demand for AI policy analysis has far outpaced supply.

One area of focus is the implications of AI on national and international security. Current AI systems are brittle and can easily be fooled, implying several safety and security challenges. What are these challenges, and how important are they? How can we make systems that are more robust and mitigate these problems?

Another area is how to enable effective competition on AI in a global environment, while also cooperating on issues of safety, security and ethics? This will likely require measurement of investment flows, publications, data and hardware across countries, as well as management of talent and knowledge workflows.

See also [Import AI](https://jack-clark.net/2019/03/04/import-ai-136-what-machine-learning-power-infrastructure-means-for-humanity-new-gca-benchmarkdataset-challenges-image-captioning-systems-and-google-uses-frankenrl-to-create-more-mobile-robot/).


Governance of AI Fellowship

The Center for the Governance of AI is looking for a few fellows to work for around 3 months on AI governance research. They expect that fellows will be at the level of PhD students or postdocs, though there are no strict requirements. The first round application deadline is Feb 28, and the second round application deadline is Mar 28.


SafeML ICLR 2019 Call for Papers

The SafeML workshop has a paper submission deadline of Feb 22, and is looking for papers on specification, robustness and assurance (based on [Building safe artificial intelligence: specification, robustness, and assurance](https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1) ([AN #26](https://mailchi.mp/1ecd1b775703/alignment-newsletter-26))).


Olsson to Join the Open Philanthropy Project

Catherine Olsson, a researcher at Google Brain who was previously at OpenAI, will be joining the Open Philanthropy Project to focus on grant making for reducing x-risk from advanced AI. Given her first-hand research experience, she has knowledge of the dynamics of research groups and a nuanced understanding of various safety subproblems. Congratulations to both her and OpenPhil.


GovAI Summer 2022 Fellowships

Applications are now open for the GovAI 2022 Summer Fellowship! This is an opportunity for early-career individuals to spend three months working on an AI governance research project, learning about the field, and making connections with other researchers and practitioners. Application deadline is Jan 1.


Foundations of Cooperative AI Lab

This new lab at CMU aims to create foundations of game theory appropriate for advanced, autonomous AI agents -- think of work on agent foundations and <@cooperative AI@>(@Open Problems in Cooperative AI@). Apply for a PhD [here](https://csd.cmu.edu/academics/doctoral/admissions) (deadline Dec 9) or for a postdoc [here](https://apply.interfolio.com/98450).


OpenAI hiring Software Engineer, Alignment

Exactly what it sounds like: OpenAI is hiring a software engineer to work with the Alignment team.


CHAI Internships 2022

CHAI internships are open once again! Typically, an intern will execute on an AI safety research project proposed by their mentor, resulting in a first-author publication at a workshop. The early deadline is November 23rd and the regular deadline is December 13th.


Announcing the Vitalik Buterin Fellowships in AI Existential Safety

FLI is launching a fellowship for incoming PhD students and postdocs who are focused on AI existential safety. The application deadline is October 29 for the PhD fellowship, and November 5 for the postdoc fellowship.


The Open Phil AI Fellowship (Year 5)

Applications are now open for the fifth cohort of the <@Open Phil AI Fellowship@>! They are also due October 29.


Research Scientist, Long-term Strategy & Governance

DeepMind (my employer) is hiring for several Research Scientist positions on the Long-term Strategy and Governance Team, across a wide range of backgrounds and skills. (Though note that you do need a PhD, or equivalent experience.) See also this [EA Forum post](https://forum.effectivealtruism.org/posts/atbonGDAFegfeDbTF/deepmind-is-hiring-long-term-strategy-and-governance).


Cooperative AI Workshop 2021

The <@Cooperative AI@>(@Open Problems in Cooperative AI@) <@NeurIPS workshop@>(@Cooperative AI Workshop@) is running again this year! The paper submission deadline is September 25.


You can now apply to EA Funds anytime! (LTFF & EAIF only)

The Long-Term Future Fund (LTFF) has funding available for people working on AI alignment. I’m told that the LTFF is constrained by high-quality applications, and that applying only takes a few hours, so it is probably best to err on the side of applying. The LTFF has removed its previous round-based system and now accepts applications anytime.


Stanford Existential Risks Conference

This conference on existential risks will run April 17-18. Applications to attend close April 12. There will be no charge to attend the conference.


Research Engineer, Safety (OpenAI)

The Applied Safety team at OpenAI is looking to hire a research engineer, and explicitly states that the job is about safety of general-purpose AI systems (as opposed to narrow AI systems like autonomous vehicles).


Chinese translation of Human Compatible

The Chinese translation of <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@) came out in October and the first chapter is [here](https://cread.jd.com/read/startRead.action?bookId=30675029&readType=1).


DPhil Scholarships Applications Open

FHI will be awarding up to six scholarships for the 2021/22 academic year for DPhil students starting at the University of Oxford whose research aims to answer crucial questions for improving the long-term prospects of humanity. Applications are due Feb 14.


CHAI Internship

<@CHAI internships@>(@CHAI 2020 Internships@) are open once again! The deadline for applications is December 13.


Hiring engineers and researchers to help align GPT-3

The Reflection team at OpenAI is hiring ML engineers and ML researchers to push forward work on aligning GPT-3. Their most recent results are described in <@Learning to Summarize with Human Feedback@>.


The Open Phil AI Fellowship

We’re now at the fourth cohort of the <@Open Phil AI Fellowship@>! Applications are due October 22.


Navigating the Broader Impacts of AI Research

This is a workshop at NeurIPS; the title tells you exactly what it's about. The deadline to submit is October 12.


FHI is hiring Researchers, Research Fellows, and Senior Research Fellows

FHI is hiring for researchers across a wide variety of topics, including technical AI safety research and AI governance. The application deadline is October 19.


Cooperative AI Workshop

This NeurIPS workshop has the goal of improving the _cooperation_ skills of AI systems (whether with humans or other machines), which encompasses a _very_ wide range of research topics. The deadline to submit is September 18.


Senior Systems Safety Engineer

OpenAI is hiring for a senior systems safety engineer. From my read of the job description, it seems like the goal is to apply the principles from <@Engineering a Safer World@> to AI development.


Early-career funding for individuals interested in improving the long-term future

This Open Philanthropy program aims to provide support for people who want to focus on improving the long-term future. The primary form of support would be funding for graduate school, though other one-off activities that build career capital also count. They explicitly say that people interested in working on AI policy or risks from transformative AI should apply to this program (possibly in addition to their <@AI fellowship@>(@Open Phil AI Fellowship@)). The stage 1 deadline is January 1, but if you submit earlier they aim to respond within 10 working days.


Looking for adversarial collaborators to test our Debate protocol

OpenAI is looking for people to help test their <@debate@>(@Writeup: Progress on AI Safety via Debate@) protocol, to find weaknesses that allow a dishonest strategy to win such debates.


This is notable since this is (to my knowledge) OpenAI’s first commercial application.

OpenAI API

OpenAI has released a commercial API that gives access to natural language completions via <@GPT-3@>(@Language Models are Few-Shot Learners@), allowing users to specify tasks in English that GPT-3 can then (hopefully) solve.


Visible Thoughts Project and Bounty Announcement

MIRI would like to test whether language models can be made more understandable by training them to produce visible thoughts. As part of this project, they need a dataset of thought-annotated dungeon runs. They are offering $200,000 in prizes for building the first fragments of the dataset, plus an additional $1M prize/budget for anyone who demonstrates the ability to build a larger dataset at scale.


I think this is a particularly good contest to try to test your fit with (a certain kind of) theoretical alignment research: even if you don't have much background, you can plausibly get up to speed in tens of hours. I will also try to summarize ELK next week, but no promises.

Prizes for ELK proposals

The Alignment Research Center (ARC) recently published a technical report on Eliciting Latent Knowledge (ELK). They are offering prizes of $5,000 to $50,000 for proposed strategies that tackle ELK. The deadline is the end of January.


FLI launches Worldbuilding Contest with $100,000 in prizes

Worldbuilding Contest

FLI invites individuals and teams to compete for a prize purse worth $100,000+ by designing visions of a plausible, aspirational future including artificial general intelligence. The deadline for submissions is April 15.


WAISE 2018


CLR Open Positions: Researchers and Summer Research Fellows

The Center on Long-Term Risk is looking for researchers and summer research fellows to work on high-quality research relevant to s-risks, including on (among other areas) multiagent systems. The application deadline is May 13.


Announcing Web-TAISU, May 13-17

The <@Technical AI Safety Unconference@>(@TAISU - Technical AI Safety Unconference@) will be held online from May 13-17.


You might be wondering why I've included these jobs in the newsletter, given that I don't do very many promotions. I think that it is reasonably likely that the US government (and the military in particular) will be a key player in the future of AI, and that there could be a lot to learn from their testing, evaluation, validation & verification (TEV&V) framework (which often seems more risk-averse to me than many alignment schemes are). As a result, I would be excited if readers of this newsletter interested in how the military thinks about AI filled these positions: it seems great to have a flow of ideas between the two communities (so that the government learns about alignment concerns, and so that we learn about TEV&V).

Careers at the Joint AI Center

The Joint AI Center is searching for ML experts for a variety of roles.


TAISU - Technical AI Safety Unconference

This unconference on technical AI safety will be held May 14th-17th; application deadline is February 23.


AI Alignment Visiting Fellowship

This fellowship would support 2-3 applicants to visit FHI for three or more months to work on human-aligned AI. The application deadline is Feb 28.


AI Safety Unconference 2019

David Krueger, Orpheus Lummis, and Gretchen Krueger

Like last year, there will be an AI safety unconference alongside NeurIPS, on Monday Dec 9 from 10am to 6pm. While the website suggests a registration deadline of Nov 25, the organizers have told me it's a soft deadline, but you probably should [register](https://docs.google.com/forms/d/e/1FAIpQLSfOXyo2P0Wv6bxyNyzzdMnzSL8_wGa4pMIDTh1tQwYeZmMebw/viewform) now to secure a place.


CHAI 2020 Internships

CHAI (the lab where I work) is currently accepting applications for its 2020 internship program. The deadline to apply is **Dec 15**.


FHI DPhil Scholarships

The Future of Humanity Institute will be awarding up to two DPhil scholarships for the 2020/21 academic year, open to students beginning a DPhil at the University of Oxford whose research aims to answer crucial questions for improving the long-term prospects of humanity. Applications will open around January or February, and decisions will be made in April.


Open Phil AI Fellowship

The Open Phil AI Fellowship is seeking applications for its third cohort. Applications are due by October 25. The fellowship is open to current and incoming PhD students, including those with pre-existing funding sources. It provides up to 5 years of support with a stipend of $40,000 and a travel allocation of $10,000.


Ought: why it matters and ways to help

This post discusses the work that Ought is doing, and makes a case that it is important for AI alignment (see the summary for [Delegating open-ended cognitive work](https://ought.org/presentations/delegating-cognitive-work-2019-06) above). Readers can help Ought by applying for their web developer role, by participating in their experiments, and by donating.


Join our rapidly growing research teams

The Future of Humanity Institute is hiring researchers across a wide range of topics, including AI safety and strategy. The deadline to apply is midday August 16.


Contest: $1,000 for good questions to ask to an Oracle AI


Offer of collaboration and/or mentorship

This is exactly what it sounds like. You can find out more about Vanessa's research agenda from <@The Learning-Theoretic AI Alignment Research Agenda@>, and I've summarized two of her recent posts in this newsletter.


Human-aligned AI Summer School

Jan Kulveit, Tomáš Gavenčiak, Jan Romportl

The second Human-aligned AI Summer School will be held in Prague from July 25-28, with a focus on "optimization and decision-making". Applications are due June 15.


AI Safety workshop at IJCAI 2019

Huáscar Espinoza, Han Yu, Xiaowei Huang, Freddy Lecue, Cynthia Chen, José Hernández-Orallo,Seán Ó hÉigeartaigh, and Richard Mallah

There will be a workshop on AI safety at IJCAI 2019 in Macao, China; the paper submission deadline is April 12. In addition to the standard submissions (technical papers, proposals for technical talks, and position papers), they are seeking papers for their "AI safety landscape" initiative, which aims to build a single document identifying the core knowledge and needs of the AI safety community.


MIRI Summer Fellows Program

CFAR and MIRI are running the MIRI Summer Fellows Program from August 9-24. Applications are due March 31.


RAISE is launching their MVP

The Road to AI Safety Excellence will begin publishing lessons on inverse reinforcement learning and iterated amplification on Monday. They are looking for volunteers for their testing panel, who will study the material for about one full day per week, with guidance from RAISE, and provide feedback on the material and in particular on any sources of confusion.


FHI DPhil Scholarships

The Future of Humanity Institute is accepting applications for scholarships for candidates beginning a DPhil programme.


PAI Fellowship Program Call For Applications

The Partnership on AI is opening applications for Research Fellows who will "conduct groundbreaking multi-disciplinary research".


Announcement: AI alignment prize round 4 winners

The last iteration of the AI alignment prize has concluded, with awards of $7500 each to [Penalizing Impact via Attainable Utility Preservation](https://www.alignmentforum.org/posts/mDTded2Dn7BKRBEPX/penalizing-impact-via-attainable-utility-preservation) ([AN #39](https://mailchi.mp/036ba834bcaf/alignment-newsletter-39)) and [Embedded Agency](https://www.alignmentforum.org/s/Rm6oQRJJmhGCcLvxh) ([AN #31](https://mailchi.mp/7d0e3916e3d9/alignment-newsletter-31), [AN #32](https://mailchi.mp/8f5d302499be/alignment-newsletter-32)), and $2500 each to [Addressing three problems with counterfactual corrigibility](https://www.alignmentforum.org/posts/owdBiF8pj6Lpwwdup/addressing-three-problems-with-counterfactual-corrigibility) ([AN #30](https://mailchi.mp/c1f376f3a12e/alignment-newsletter-30)) and [Three AI Safety Related Ideas](https://www.alignmentforum.org/posts/vbtvgNXkufFRSrx4j/three-ai-safety-related-ideas)/[Two Neglected Problems in Human-AI Safety](https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety) ([AN #38](https://mailchi.mp/588354e4b91d/alignment-newsletter-38)).


Summit on Machine Learning meets Formal Methods

This is a one-day summit on July 13 that is part of the Federated Logic Conference. This seems like an unusually good venue to think about how to apply formal methods to AI systems -- in particular I'm impressed by the list of speakers, which includes a variety of experts in both fields.


It's a good roundup of papers, including several papers that I haven't covered in this newsletter.

2018 AI Alignment Literature Review and Charity Comparison

This post summarizes relevant papers in AI alignment over the last year, and uses them to compare different organizations working on AI alignment in order to choose which one to donate to.


Internships and fellowships for 2019

There are a lot of AI internships and fellowships to apply for now, including the [CHAI summer internship](https://humancompatible.ai/jobs#internship) (focused on safety in particular), the OpenAI [Fellows, Interns](https://blog.openai.com/openai-fellows-interns-2019/) and [Scholars](https://blog.openai.com/openai-scholars-2019/) programs, the [Google AI Residency Program](https://ai.google/research/join-us/ai-residency/) ([highlights](http://ai.googleblog.com/2018/10/highlights-from-google-ai-residency.html)), the [Facebook AI Research Residency Program](https://research.fb.com/programs/facebook-ai-research-residency-program/), the [Microsoft AI Residency Program](https://www.microsoft.com/en-us/research/academic-program/microsoft-ai-residency-program/), and the [Uber AI Residency](https://eng.uber.com/uber-ai-residency/).


I like the breakdown of research topics, though personally I would have made them broader. I think I would want the "reward learning" category to include anything that aims to provide a specification for the AIs behavior, such as natural language instructions (that are mapped directly to policies with no reward in between). The "reliability" section is then about successfully meeting the specification, while the third section would include anything that allows the operator to empirically verify whether and enforce that the previous two sections are working correctly, including "interpretability". Actually, having written this, it's pretty similar to the three categories in the DeepMind blog post covered in the highlights.

The Open Philanthropy Project AI Fellows Program

The second Open Phil AI Fellowship has been announced, open to AI and ML PhD students, and people applying to PhD programs starting in 2019. Even if you aren't looking for a fellowship, you may want to read through their example research topics, which are split into three main categories -- reward learning, reliability, and interpretability.


80,000 Hours Job Board: AI/ML safety research

80,000 Hours recently updated their job board, including the section on technical safety research. The [AI strategy and governance](https://80000hours.org/job-board/ai-strategy-governance/) section is probably also of interest.


[BERI](http://existence.org/jobs/ml-engineer)/[CHAI](https://humancompatible.ai/jobs#engineer) ML engineer

I want to highlight this role in particular -- I expect this to be a position where you can not only have a large impact, but also learn more about technical research, putting you in a better position to do research in the future.


State of California Endorses Asilomar AI Principles


DeepMind job: Science Writer

According to the job listing, the role would involve creating content for the blog, videos, presentations, events, etc. and would require a reasonably technical background and strong writing skills. Vishal Maini at DeepMind notes that this person would likely have a significant impact on how AI research is communicated to various key strategic audiences around the world -- from the technical community to the broader public -- and would spend some of their time engaging with AI alignment research, among other areas.


Internship: The Future Society

An internship which will focus on AI policy research as well as support to organize two large AI governance events. To apply, send a CV and a short letter explaining ‘why you?’ to caroline.jeanmaire@thefuturesociety.org.


MIRI's April 2018 Newsletter

Lots of links to things MIRI has done, and some links to other people's work as well.


AI Strategy Project Manager


Public reports are now optional for EA Funds grantees

This is your regular reminder that you can apply to the Long-Term Future Fund (and the broader EA Funds) for funding for a wide variety of projects. They have now removed the requirement for public reporting of your grant. They encourage you to apply if you have a preference for private funding.


Sydney AI Safety Fellowship

This 7-week fellowship will provide fellows from Australia and New Zealand the opportunity to pursue projects in AI Safety or spend time upskilling. Applications are due December 14.


$2 Million Donated to Keep Artificial General Intelligence Beneficial and Robust

The next round of FLI grants have been announced! There are fewer grants than in their [first round](https://futureoflife.org/first-ai-grant-recipients/) and the topics seem more focused on AGI safety.


AI Safety Needs Great Engineers

If the previous two roles weren't enough to convince you, this post explicitly argues that a lot of AI safety work is bottlenecked on good engineers, and encourages people to apply to such roles.


AI Safety Camp Virtual 2022

Applications are open for this remote research program, where people from various disciplines come together to research an open problem under the mentorship of an established AI-alignment researcher. Deadline to apply is December 1st.


Q&A Panel on Applying for Grad School

In this event run by AI Safety Support on November 7, current PhD students will share their experiences navigating the application process and AI Safety research in academia. RSVP [here](evt.to/dhaimisw).


SafeAI Workshop 2022

The SafeAI workshop at AAAI is now accepting paper submissions, with a deadline of Nov 12.


FLI's $25M Grants Program for Existential Risk Reduction

This podcast talks about FLI's recent grants program for x-risk reduction. I've previously mentioned the <@fellowships@>(@Announcing the Vitalik Buterin Fellowships in AI Existential Safety@) they are running as part of this program. As a reminder, the application deadline is October 29 for the PhD fellowship, and November 5 for the postdoc fellowship.


[Job ad] Research important longtermist topics at Rethink Priorities!

Of particular interest to readers, there are roles available in AI governance and strategy. The application deadline is Oct 24.


2022 IEEE Conference on Assured Autonomy

The ICAA conference seeks contributions on all aspects of AI safety, security, and privacy in autonomous systems. The paper submission deadline is October 18 and the conference itself will take place March 22-24.


CSER Job Posting: Academic Programme Manager

CSER is searching for a candidate for a relatively senior role that combines academic, management and administrative responsibilities. The application deadline is September 20.