Section 1.3 of [this version of the paper](https://mltheory.org/dg_short.pdf)

Distributional Generalization: A New Kind of Generalization

Suppose you train a classifier to distinguish between CIFAR-10 classes, except each airplane has a 30% chance of being mislabeled as a car. If you then train a model to achieve perfect accuracy on this badly labeled dataset, it will get 100% accuracy on the training set, and 97% of those labels will actually be correct (since 3% are mislabeled airplanes). Under the current paradigm, if we say that the model “generalizes”, that means that it will also get 97% accuracy at test time (according to the actually correct labels). However, this doesn’t tell us anything about what mistakes are made at test time -- is it still the case that 30% of airplanes are mislabeled as cars, or does the model also make mistakes on e.g. deer?

_Distributional generalization_ aims to make claims about situations like these. The core idea is to make claims about the full distribution of classifier outputs, rather than just the single metric of test accuracy.

Formally, we assume there is some distribution D, from which we can sample pairs of points (x, y), which generates both our train and test sets. Then, the train (resp. test) distribution of classifier outputs is (x, f(x)), with x coming from the train (resp. test) set. The train and test distributions of classifier outputs are the objects of study in distributional generalization. In particular, given a [0,1]-valued function on distributions (called a _test_ T), we say that the classifier _generalizes w.r.t T_ if T outputs similar values on the train and test distribution. (W.r.t means “with respect to”.) For example, given a distribution, the _accuracy test_ checks how often the classifier’s output is correct in expectation over that distribution. Generalization w.r.t the accuracy test is equivalent to the canonical notion of generalization.

Let’s suppose that the classifier perfectly fits the training set, so that the train distribution of classifier outputs is the same as the original distribution D. Let’s additionally suppose that the classifier generalizes with respect to the accuracy test, so that the classifier has perfect test accuracy. Then, the test distribution of classifier outputs will also be the same as the original distribution D, that is, all distributions are identical and there isn’t much more to say. So, the interesting situations are when one of these two assumptions is false, that is, when either:
1. The classifier does not perfectly fit the training set, or
2. The classifier does not generalize w.r.t accuracy.
This paper primarily focuses on classifiers that _do_ perfectly fit the training set, but don’t generalize w.r.t accuracy. One typical way to get this setting is to inject label noise (as in the mislabeled airplanes case), since this prevents the classifier from getting 100% test accuracy.

Speaking of which, let’s return to our original example in which we add label noise by mislabeling 30% of airplanes as cars. Notice that, since the label noise is completely divorced from the classifier’s input x, the best way for the classifier to minimize _test_ loss would be to always predict the true CIFAR-10 label, and then 3% of the time the true distribution will say “lol, jk, this airplane is actually a car”. However, in practice, classifiers will also label approximately 30% of airplanes as cars in the test set as well! This incurs higher loss, because the 30% of airplanes that the classifier labels as cars must be independent of the 30% of airplanes that the true distribution labels as cars, which implies that the model disagrees with the true distribution 4.2% of the time; this is worse than the 3% it would get if it consistently labeled airplanes as airplanes. **Classifiers trained to interpolation are not Bayes-optimal in the presence of label noise.**

Okay, let’s get back to distributional generalization. We already know the classifier does not generalize w.r.t accuracy. However, the fact that it still labels about 30% of airplanes as cars suggests a different kind of generalization. Recall that the train and test distributions of classifier outputs have the form (x, f(x)). Consider the feature L(x) that says whether x is an airplane or not. Then, if we replace (x, f(x)) with (L(x), f(x)), then this now looks identical between the train and test distributions! Specifically, this distribution places 7% on (“yes airplane”, “airplane”), 3% on (“yes airplane”, “car”), and 10% on (“no airplane”, c) for every class c other than “airplane”. An alternative way of stating this is that the classifier generalizes w.r.t all tests whose dependence on x factors through the feature L. (In other words, the test can only depend on whether x is an airplane or not, and cannot depend on any other information about x.)

The authors make a more general version of this claim they call _feature calibration_: for every feature L that _could_ be learned by the classifier, the classifier generalizes w.r.t all tests whose dependence on x factors through L. Note that they do not assume that the classifier _actually_ learns L: just that, if you hypothetically trained the classifier on a dataset of (x, L(x)), then it could learn that function near-perfectly.

They then provide evidence for this through a variety of experiments and one theorem:
- If you plug in the constant feature L(x) = 0 into the conjecture, it implies that classifiers should get the right class balance (i.e. if your distribution contains class 1 twice as often as class 0, then you predict class 1 twice as often as class 0 at test time). They demonstrate this on a rebalanced version of CIFAR-10, even for classifiers that generalize poorly w.r.t accuracy.
- When using a WideResNet (for which the true CIFAR-10 labels are learnable), if you add a bunch of structured label noise into CIFAR-10, the test predictions reflect that same structure.
- The same thing is true for decision trees applied to a molecular biology dataset.
- A ResNet-50 trained to predict attractiveness on the CelebA dataset (which does not generalize w.r.t accuracy) does satisfy feature calibration w.r.t “wearing lipstick”, “heavy makeup”, “blond hair”, “male”, and “eye-glasses”. Note there is no label noise in this case.
- AlexNet predicts that the right fraction of dogs are Terriers, even though it mistakes which exact dogs are Terriers.
- The nearest-neighbor classifier provably satisfies feature calibration under relatively mild regularity conditions.

In an appendix, they provide preliminary experiments suggesting this holds _pointwise_. In our mislabeled airplane example, for a _specific_ airplane x from the test set, if you resample a training set (with the 30% mislabeling of airplanes) and retrain a classifier on that set, then there is a roughly 30% chance that that specific x will be misclassified as a car.

The authors then introduce another distributional generalization property: _agreement_. Suppose we have two classifiers f and g trained on independently sampled training sets. The agreement conjecture states that the test accuracy of f is equal to the expected probability that f agrees with g on the test distribution (loosely speaking, this is how often f and g make the same prediction for test inputs). The agreement property can also be framed as an instance of distributional generalization, though I won’t go into the specific test here. The authors perform similar experiments as with feature calibration to demonstrate that the agreement property does seem to hold across a variety of possible classifiers.

Interestingly, these properties are _not_ closed under ensembling. In our mislabeled airplane example, every model will label 30% of airplanes as cars, but _which_ airplanes are mislabeled is independent across models. As a result, the plurality voting used in ensembles reduces the misclassification rate to 22%, which means that you no longer satisfy feature calibration. Consistent with this, the authors observe that neural network ensembles, random forests, and k-nearest neighbors all did not satisfy feature calibration, and tended to be closer to the Bayes-optimal solution (i.e. getting closer to being robust to label noise, in our example).

Summary of the summary: Let’s look at the specific ways in which classifiers make mistakes on the test distribution. This is called distributional generalization. The paper makes two conjectures within this frame. _Feature calibration_ says that for any feature that a classifier could have learned, the distribution of its predictions, conditioned on that feature, will be the same at train and test time, including any mistakes it makes. _Agreement_ says that the test accuracy of a classifier is equal to the probability that, on some randomly chosen test example, the classifier’s prediction matches that of another classifier trained on a freshly generated training set. Interestingly, while these properties hold for a variety of ML models, they do not hold for ensembles, because of the plurality voting mechanism.


While the problem presented in this paper isn’t particularly novel, I appreciated the framing of fragility of distributional shift as being caused by underspecification. I see concerns about <@inner alignment@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) as primarily worries about underspecification, rather than distribution shift more generally, so I’m happy to see a paper that explains it well.

That being said, the experiments with neural networks were not that compelling -- while it is true that the models had higher variance on the metrics testing robustness to distributional shift, on an absolute scale the variance was not high: even a standard deviation of 0.024 (which was an outlier) is not huge, especially given that the distribution is being changed.

Underspecification Presents Challenges for Credibility in Modern Machine Learning

Alexander D'Amour*, Katherine Heller*, Dan Moldovan*, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, D. Sculley

This paper explains one source of fragility to distributional shift, which the authors term _underspecification_. The core idea is that for any given training dataset, there are a large number of possible models that achieve low loss on that training dataset. This means that the model that is actually chosen is effectively arbitrarily chosen from amongst this set of models. While such a model will have good iid (validation) performance, it may have poor inductive biases that result in bad out-of-distribution performance.

The main additional prediction of this framing is that if you vary supposedly “unimportant” aspects of the training procedure, such as the random seed used, then you will get a different model with different inductive biases, which will thus have different out-of-distribution performance. In other words, not only will the out-of-distribution performance be worse, its _variance_ will also be higher.

The authors demonstrate underspecification in a number of simplified theoretical settings, as well as realistic deep learning pipelines. For example, in an SIR model of disease spread, when we only have the current number of infections during the initial growth phase, the data cannot distinguish between the case of having high transmission rate but low durations of infection, vs. a low transmission rate but high durations of infection, even though these make very different predictions about the future trajectory of the disease (the out-of-distribution performance).

In deep learning models, the authors perform experiments where they measure validation performance (which should be relatively precise), and compare it against out-of-distribution performance (which should be lower and have more variance). For image recognition, they show that neural net training has precise validation performance, with 0.001 standard deviation when varying the seed, but less precise performance on <@ImageNet-C@>(@Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations@), with standard deviations in the range of 0.002 to 0.024 on the different corruptions. They do similar experiments with medical imaging and NLP.


While I agree with the general thrust of this article, comparing image misclassification rates to rates of catastrophic failures in aviation seems a bit harsh. I am having difficulties imagining an aviation system that fails due to a single input that has been processed wrongly, even though the correlation between subsequent failures given similar inputs might mean that this is not necessary for locally catastrophic outcomes.

**Rohin's opinion:** My guess is that we’ll need to treat systems based primarily on neural nets similarly to operators. The main reason for this is that the tasks that AI systems will solve are usually not even well-defined enough to have a reliability rate like 0.99999997 (or even a couple of orders of magnitude worse). For example, human performance on image classification datasets is typically under 99%, not because humans are bad at image recognition, but because in many cases what the “true label” should be is ambiguous. For another example, you’d think “predict the next word” would be a nice unambiguous task definition, but then for the question “How many bonks are in a quoit?“, should your answer be [“There are three bonks in a quoit”](https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html) or [“The question is nonsense”](https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/)? (If you’re inclined to say that it’s obviously the latter, consider that many students will do something like the former if they see a question they don’t understand on an exam.)

Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance

Test, Evaluation, Verification, and Validation (TEVV) is an important barrier for AI applications in safety-critical areas. Current TEVV standards have very different rules for certifying _software_ and certifying _human operators_. It is not clear which of these processes should be applied for AI systems.

If we treat AI systems as similar to human operators, we would certify them ensuring that they pass tests of ability. This does not give much of a guarantee of robustness (since only a few situations can be tested), and is only acceptable for humans because humans tend to be more robust to new situations than software. This could be a reasonable assumption for AI systems as well: while systems are certainly vulnerable to adversarial examples, the authors find that AI performance degrades surprisingly smoothly out of distribution in the absence of adversaries, in a plausibly human-like way.

While AI might have some characteristics of operators, there are good reasons to treat it as software. The ability to deploy multiple copies of the same system increases the threat of correlated failures, which is less true of humans. In addition, parallelization can allow for more extensive testing that is typical for software TEVV. For critical applications, a common standard is that of Safety Integrity Levels (SILs), which correspond to approximate failure rates per hour. Current AI systems fail way more often than current SILs for safety-critical applications demand. For example an image recognition system would require an accuracy of 0.99999997 at 10 processed frames per second just to reach the weakest SIL used in aviation.

However, SILs are often used on multiple levels and it is possible to build a system with a strong SIL from weaker components by using redundant components that fail independently or by detecting failures sufficiently early, such that AI modules could still be used safely as parts of a system specifically structured to cope with their failures. For example, we can use out-of-distribution detection to revert to a safe policy in simple applications. However, this is not possible for higher levels of automation where such a policy might not be available.


I think this is an awesome paper which, among other things, points at potential research directions for increasing OOD robustness: training more, training more diversely, and training in self-supervised ways. I think it's pretty noteworthy that larger models don't increase OOD robustness in NLP (all else equal), because it implies that certain guarantees may be constrained entirely by training procedures.

Pretrained Transformers Improve Out-of-Distribution Robustness

Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, Dawn Song

One important metric for the performance of deep learning models is the extent to which they generalize to examples that are _out-of-distribution_ (OOD) from the original distribution on which they were trained. This ability is sometimes called out-of-distribution _robustness_. This paper examines the OOD robustness of several NLP models: a bag-of-words model, word embedding models that use word averages, LSTMs, or ConvNets, and several models that use pretrained bidirectional transformers (BERT).

The paper finds that:
- Pretrained transformers (BERT) are significantly more OOD robust.
- Pretrained transformers (BERT) are significantly better at _detecting_ when they've encountered an OOD example. Previous models do worse than random chance at detection.
- Larger models don't increase OOD robustness in NLP the way they seem to in computer vision.
- Model distillation (using a larger trained neural network to train a smaller neural network) reduces OOD robustness, suggesting that naive in-distribution tests for model distillation methods may mask later failures.
- More diverse data improves OOD robustness.

The paper hypothesizes that these pretrained models may perform better because they were pretrained on particularly diverse data, were trained on a large amount of data, and were trained with self-supervised objectives, which previous work has suggested improves OOD robustness and detection.


While I'm not sure to what extent models learn truly irrelevant information (see <@Adversarial Examples Are Not Bugs, They Are Features@>), it seems good to add an incentive against learning information that won't be useful for a downstream task, and the empirical results (especially of the next paper) suggest that it is providing some benefit.

The Conditional Entropy Bottleneck

While I've categorized this paper under robustness because it can apply to most forms of training, I'll talk about it specifically in the context of unsupervised learning (and in particular its relation to Contrastive Predictive Coding (CPC), summarized in the highlights).

One potential problem with deep learning is that there might be too _much_ information in the input, causing the model to learn spurious correlations that do not actually generalize well (see <@Causal Confusion in Imitation Learning@> as an example). The idea with CEB is to penalize the model for learning irrelevant information, using a form of _information bottleneck_.

We consider a setting where we want to learn a representation **Z** of some input data **X** in order to predict some downstream data **Y**. In CPC, **X** would be the inputs from time 1 to t, **Z** would be the latent representation **z_t**, and **Y** would be the future data **x_{t+k}**. Then, we want **Z** to capture the **minimum necessary information** needed for **Z** to predict **Y** as best as possible. The _necessary_ information is **I(Y; Z)**, that is, the mutual information between **Z** and **Y**: we want to maximize this to maximize our accuracy at predicting **Y**. Since **Y** depends on **X** and **Z** is computed from **X**, any information about **Y** must come through mutual information between **X** and **Z**. Maximizing just this **I(Y; Z)** term gives us Contrastive Predictive Coding.

However, we don't want to capture any extra irrelevant information (the minimality criterion), which means that **Z** shouldn't capture any _more_ information about **X** beyond what it captured to maximize **I(Y; Z)**. In information-theoretic terms, we want to _minimize_ **I(X; Z | Y)**. Thus, we have the CEB objective: minimizing **I(X; Z | Y) - γ I(Y; Z)**, where **γ** is a hyperparameter controlling the tradeoff between the two terms. The authors then use some fairly straightforward math to reduce the objective to simpler terms which can be bounded using variational approximations, leading to an algorithm that can work in practice.

The authors perform experiments on Fashion MNIST and CIFAR10 (where Y corresponds to the labels for the images, so we're in the supervised learning setting). Since the main benefit of CEB is to remove unnecessary information from the model, they evaluate adversarial robustness and out-of-distribution detection in addition to standard performance checks. They find that models trained with CEB perform better than ones trained with a variational information bottleneck, or ones trained with vanilla SGD.


CEB Improves Model Robustness

This empirical paper finds that ImageNet classifiers trained with the CEB objective (summarized above) are already somewhat adversarially robust, without having any decrease in accuracy, and without any adversarial training. Notably, since CEB does not rely on knowing the attack method ahead of time, its adversarial robustness generalizes to multiple kinds of attacks, whereas models that were adversarially trained tend to be fragile in the face of previously unseen attacks.


Call for Papers: ICML 2019 Workshop on Uncertainty and Robustness in Deep Learning

Topics of this workshop include out-of-distribution detection, calibration, robustness to corruptions, robustness to adversaries, etc. Submissions are due April 30th.


The semi-verified model seems highly useful for various safety-related scenarios including learning with [label](https://arxiv.org/abs/1803.09050) [corruption](https://arxiv.org/abs/1802.05300), poisoned input data, and minimal supervision.

Learning from Untrusted Data

Moses Charikar, Jacob Steinhardt, Gregory Valiant

This paper introduces semi-verified learning. Here a model learns from a verified or trusted dataset, and from an untrusted dataset which consists in a mixture of legitimate and arbitrary examples. For the untrusted dataset, it is not known which points are legitimate and which are not. This scenario can occur when data is scraped from the internet, recorded by unreliable devices, or gathered through [crowdsourcing](https://ai.googleblog.com/2017/04/federated-learning-collaborative.html). Concretely if a (possibly small) fraction of the scraped data is hand-labeled, then this could count as the trusted set, and the remaining data could be considered the untrusted set. This differs from semi-supervised learning where there are labeled and unlabeled task-relevant examples. Here there are trusted examples and examples which are untrusted (e.g., labels may be wrong, features may be out-of-distribution, examples may be malicious, and so on). See the full paper for theorems and an algorithm applicable to tasks such as robust density estimation.


Experimentally evaluating whether honesty generalizes

The highlighted post introduced the notion of optimism about generalization. On this view, if we train an AI agent on question-answer pairs (or comparisons) where we are confident in the correctness of the answers (or comparisons), the resulting agent will continue to answer honestly even on questions where we wouldn’t be confident of the answer.

While we can’t test exactly the situation we care about -- whether a superintelligent AI system would continue to answer questions honestly -- we _can_ test an analogous situation with existing large language models. In particular, let’s consider the domain of unsupervised translation: we’re asking a language model trained on both English and French to answer questions about French text, and we (the overseers) only know English.

We could finetune the model on answers to questions about grammar ("Why would it have been a grammatical error to write Tu Vas?") and literal meanings ("What does Defendre mean here?"). Once it performs well in this setting, we could then evaluate whether the model generalizes to answer questions about tone ("Does the speaker seem angry or sad about the topic they are discussing?"). On the optimism about generalization view, it seems like this should work. It is intentional here that we only finetune on two categories rather than thousands, since that seems more representative of the case we’ll actually face.

There are lots of variants which differ in the type of generalization they are asking for: for example, we could finetune a model on all questions about French text and German text, and then see whether it generalizes to answering questions about Spanish text.

While the experiments as currently suggested probably won’t show good generalization, a variant that could support it would be one in which we train for _plausibility_. In our original example, we finetune on correct answers for grammar and literal meanings, and then we _also_ finetune to have the model give _plausible_ answers to tone (i.e. when asked about tone, instead of saying “en colère means 'angry'”, the model says “the author is angry, as we can see from the use of ‘en colère’”). It seems possible that this combination leads to the model giving actually correct answers about tone, just because “honestly report your best guess” seems like the simplest policy that meets all of these criteria.


Evaluating the Robustness of Collaborative Agents

Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, A. D. Dragan, Rohin Shah

Assuming a well-specified reward function, we would like to evaluate robustness of an agent by looking at the average reward it obtains on a wide scenario of plausible test time inputs that it might get. However, the key challenge of robustness is that it is hard to specify the test distribution in advance, and we must work with the training distribution instead.

This paper (on which I am an author) proposes _measuring_ robustness using a suite of hand-designed _unit tests_. Just as a function is tested by having the programmer write down potential edge cases and checking for the expected behavior, AI developers can come up with a set of potential “edge case” situations (especially ones not likely to arise during training) and check whether the agent’s behavior on these situations works well or not. Intuitively, since these unit tests are created separately from the training process, they may not have the same spurious correlations that could be present in the training data. Thus, they can serve as an evaluation of the robustness of the agent.

The authors built a test suite for <@Overcooked@>(@Collaborating with Humans Requires Understanding Them@), and use it to evaluate several techniques aimed to improve the robustness of agents trained to collaborate with humans.

For example, one technique is to start each episode from a state sampled randomly from a dataset of human-human gameplay, so that the agents learn how to handle a broader diversity of states. This technique _decreases_ the average _validation_ reward, and if that’s all we look at, we would conclude that it did not work. However, the technique also _increases_ performance on the unit test suite, suggesting that in reality the technique does increase robustness, though it comes at the cost of reduced performance when playing with the particular set of partners that make up the validation distribution.


I think it is plausible that the final stages of AGI deployment are carried out by the US military, similar to how the Manhattan Project took on the final stages of nuclear bomb development. If this happens, it will be important for the DoD to recognize and address issues of AI safety very carefully. If the TEVV process does get updated in a meaningful way soon, this could be a great opportunity for the AI safety community to try to translate some of its research into practice as well as building credibility and trust with the DoD.

Separately from long-term AGI deployment, I think that the DoD could be a good partner for AI safety research, particularly regarding areas like adversarial training and robustness. My main concern is that confidentiality might lead to safety research being hidden and thus less useful than if it was carried out in the open. I don’t know enough about the details of working with the DoD to know how likely that is to be an issue, however.

Building Trust through Testing

Michèle A. Flournoy, Avril Haines, Gabrielle Chefitz

In order to deploy AI systems, especially in the military, we will need to have good reason to _trust_ them. However, under the current machine learning paradigm, there are several challenges to building such trust. These challenges are broken down into technological and bureaucratic challenges. Technological challenges include lack of robustness, lack of representative test sets, lack of interpretability, and complexity once ML is integrated into larger systems. Bureaucratic challenges include lack of coordination, recruiting talent, and coordination between DoD, the private sector, and academia.

To address these challenges, the authors suggest that DoD updates its testing, evaluation, verification, and validation (TEVV) process to handle AI systems. They make 11 recommendations. A few that I found particularly interesting are:
- Create an OSD coordinating body to lead on AI/ML TEVV and incentivize strong cooperation with the services.
- Develop industry / U.S. government TEVV standards and promote them internationally.
- Test, train, and certify human-machine teams through wargaming, simulation, and experimentation.
- Increase resources for and attention on adversarial testing and red-teaming.


I was surprised that the accuracies the paper presented after adversarially training on TextFooler-produced sentences still weren't very high-- BERT's after-attack accuracy on one dataset went from 11.5% to 18.7%, and on another went from 4.0% to 8.3%. The paper didn't give a detailed description of its retraining procedure, so this may just be because they didn't adversarially train as much as they could have.

**Rohin's opinion:** This is an instance of the general trend across domains where if you search in a black-box way around training or test inputs, you can relatively easily uncover examples where your model performs poorly. We've seen this with adversarial examples in image classification, and with <@adversarial@>(@Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures@) <@policies@>(@Adversarial Policies: Attacking Deep Reinforcement Learning@) in deep reinforcement learning.

Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment

Di Jin*, Zhijing Jin*, Joey Tianyi Zhou, Peter Szolovits

This paper presents TextFooler, an algorithm for generating adversarial text for natural language tasks with only black-box access to models. TextFooler tries to generate sentences that are grammatical and semantically similar to original input sentences but produce incorrect labels. It does this by identifying a small set of most important words in the original sentence, generating candidate synonyms for those words, and gradually replacing the important words in the sentence by testing which synonyms cause the model to mispredict or report the least confidence score.

TextFooler is tested on three state-of-the-art NLP models-- WordCNN, WordLSTM, and BERT, all trained to ~80 - 90% test accuracy. On a variety of text classification datasets, TextFooler reduces accuracy to below ~15% with less than ~20% of the words perturbed. Humans evaluating the generated sentences say they are approximately as grammatical as the original, have the same label as the original in ~90% of cases, and have a sentence similarity score to the original sentence of 0.9 on a 0 to 1 scale. The paper finds that generally, models with higher original accuracy have higher after-attack acuracy.

The authors retrain BERT from scratch using data produced by TextFooler and then attack it using TextFooler again. They find that the after-attack accuracy is higher and that attacks require more perturbed words.


Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks

Mingchen Li, Mahdi Soltanolkotabi and
Samet Oymak

Previous [empirical](https://arxiv.org/pdf/1901.09960.pdf#page=2&zoom=100,0,81) [papers](https://papers.nips.cc/paper/8094-generalized-cross-entropy-loss-for-training-deep-neural-networks-with-noisy-labels.pdf) have shown that finding ways to decrease training time greatly improves robustness to label corruptions, but to my knowledge this is the first theoretical treatment.


The biggest challenge I see with this area of research, at least in its application to powerful and general AI systems, is how you get the specification in the first place, so I'm glad to see "learning specifications" as one of the areas of interest.

If I take the view from this post, it seems to me that techniques like domain randomization, and more generally training on a larger distribution of data, would count as an example of the second type of research: it is a change to the training procedure that allows us to meet the specification "the agent should achieve high reward in a broad variety of environments". Of course, this doesn't give us any provable guarantees, so I'm not sure if the authors of the post would include it in this category.

Towards Robust and Verified AI: Specification Testing, Robust Training, and Formal Verification

Pushmeet Kohli, Krishnamurthy (Dj) Dvijotham, Jonathan Uesato, Sven Gowal, and the Robust & Verified Deep Learning group

This post highlights three areas of current research towards making robust AI systems. First, we need better evaluation metrics: rather than just evaluating RL systems on the environments they were trained on, we need to actively search for situations in which they fail. Second, given a specification or constraint that we would like to ensure, we can develop new training techniques that can ensure that the specifications hold. Finally, given a specification, we can use formal verification techniques to ensure that the model obeys the specification on all possible inputs. The authors also list four areas of future research that they are excited about: leveraging AI capabilities for evaluation and verification, developing publicly available tools for evaluation and verification, broadening the scope of adversarial examples beyond the L-infinity norm ball, and learning specifications.


Adversarial Examples Are a Natural Consequence of Test Error in Noise

Nicolas Ford, Justin Gilmer, Nicolas Carlini, Dogus Cubuk

While this was previously summarized in [AN #32](https://mailchi.mp/8f5d302499be/alignment-newsletter-32), this draft is much more readable.


Trustworthy Deep Learning Course

Jacob Steinhardt, Dawn Song, Trevor Darrell

This underway course covers topics in AI Safety topics for current deep learning systems. The course includes slides and videos.


AI Alignment Podcast: The Byzantine Generals’ Problem, Poisoning, and Distributed Machine Learning

Byzantine resilience is the ability of a system to operate successfully when some of its components have been corrupted, even if it's unclear which ones they are. In the context of machine learning, this is relevant to poisoning attacks in which some training data is altered to affect the batch gradient (one example being the activity of fake accounts on social media sites). El Mahdi explains that when data is very high-dimensional, it is easy to push a neural network into a bad local minimum by altering only a small fraction of the data. He argues that his work on mitigating this is relevant to AI safety: even superintelligent AGI will be vulnerable to data poisoning due to time constraints on computation, and the fact that data poisoning is easier than resilient learning.


ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, Wieland Brendel

This paper empirically demonstrates the outsized influence of textures in classification. To address this, they apply style transfer to ImageNet images and train with this dataset. Although training networks on a specific corruption tends to provide robustness only to that specific corruption, stylized ImageNet images supposedly lead to generalization to new corruption types such as uniform noise and high-pass filters (but not blurs).


Iterative Learning with Open-set Noisy Labels

Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, Shu-Tao Xia

Much previous research on corrupted learning signals deals with label corruption, but this CVPR 2018 paper considers learning with corrupted or irrelevant inputs. For example, they train a CIFAR-10 classifier on CIFAR-10 data mixed with out-of-class CIFAR-100 data; such a scenario can occur with flawed data curation or data scraping. They use a traditional anomaly detection technique based on the local outlier factor to weight training examples; the more out-of-distribution an example is, the less weight the example has in the training loss. This approach apparently helps the classifier cope with irrelevant inputs and recover accuracy.


Bayesian Policy Optimization for Model Uncertainty

Gilwoo Lee, Brian Hou, Aditya Mandalika, Jeongseok Lee, Sanjiban Choudhury, Siddhartha S. Srinivasa


Reinforcement Learning with Perturbed Rewards


Teaching ML to answer questions honestly instead of predicting human answers

This post presents an algorithm that aims to solve the second problem from the highlighted post. As a reminder, the second problem is that an AI system that already has to make predictions about humans might learn a policy that is just “say what humans would say”, since that is simpler than learning another translation that maps its knowledge to human language (so that it can answer honestly to the best of its knowledge).

The core idea is to have a “simple” labeling process and a “complex” labeling process, where the complex process can catch errors from the simple process. We’ll also assume we have a simple / complex dataset (labeled by the simple / complex processes respectively), where we are confident that all of the answers in the dataset are correct. This means that the complex dataset has more challenging questions than the simple one. The _simple / complex instrumental policy_ is the policy that predicts what the simple / complex labeling process would do.

As an inefficient first algorithm, we can train our model on answers from the simple process where we are confident in their correctness, and then simply check whether the model performs well on new, harder questions where the complex and simple processes disagree. If the model fails this check, we restart from scratch. Intuitively, when the model eventually passes the check, it has probably learned the intended policy, as that’s the one which would generalize zero-shot to complex questions as well (and in particular, the simple instrumental policy would _not_ generalize in this way).

Of course, we need a more efficient algorithm than that. Instead of creating an algorithm where the intended policy would work but the instrumental policy wouldn’t, could we instead change the inductive biases so that the intended policy is _likely_ while the instrumental policy is not? Our original worry was that the intended policy has to bake in two implementations of language -- one in its world model, and one when translating answers into human-understandable concepts. So we could instead try to train a model that learns language from the simple instrumental policy, but is also trained on the complex dataset. The hope would be that the intended policy can learn the second implementation of language “for free” from the simple instrumental policy, while still working on the complex dataset. The actual details are quite complex and I’m not going to go into them here.

[This post](https://www.alignmentforum.org/posts/gEw8ig38mCGjia7dj/answering-questions-honestly-instead-of-predicting-human) by Evan Hubinger points out some problems and potential solutions with the approach.


AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

Dan Hendrycks*, Norman Mu*, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, Balaji Lakshminarayanan

This paper introduces a data augmentation technique to improve robustness and uncertainty estimates. The idea is to take various random augmentations such as random rotations, produce several augmented versions of an image with compositions of random augmentations, and then pool the augmented images into a single image by way of an elementwise convex combination. Said another way, the image is augmented with various traditional augmentations, and these augmented images are “averaged” together. This produces highly diverse augmentations that have similarity to the original image. Unlike techniques such as AutoAugment, this augmentation technique uses typical resources, not 15,000 GPU hours. It also greatly improves generalization to unforeseen corruptions, and it makes models more stable under small perturbations. Most importantly, even as the distribution shifts and accuracy decreases, this technique produces models that can [remain calibrated under distributional shift](https://openreview.net/pdf?id=S1gmrxHFvB#page=8&zoom=100,144,298).


An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods

Sanghyuk Chun, Seong Joon Oh, Sangdoo Yun, Dongyoon Han, Junsuk Choe, Youngjoon Yoo

There are several small tricks to improve classification performance such as label smoothing, dropout-like regularization, mixup, and so on. However, this paper shows that many of these techniques have mixed and often negative effects on various notions of robustness and uncertainty estimates.


Improving Robustness of Machine Translation with Synthetic Noise

Vaibhav Vaibhav, Sumeet Singh, Craig Stewart, Graham Neubig

By injecting noise (such as typos, word omission, slang) into the training set of a machine translation model, the authors are able to improve performance on naturally occurring data. While this trick usually does not work for computer vision models, it can work for NLP models.


Push the Student to Learn Right: Progressive Gradient Correcting by Meta-learner on Corrupted Labels

Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, Deyu Meng


Learning Robust Representations by Projecting Superficial Statistics Out

Haohan Wang, Zexue He, Zachary C. Lipton, Eric P. Xing


[Open Category Detection with PAC Guarantees](http://proceedings.mlr.press/v80/liu18e/liu18e.pdf) is the corresponding paper.

Making AI Safe in an Unpredictable World: An Interview with Thomas G. Dietterich

Thomas G. Dietterich and Jolene Creighton


Redwood Research’s current project

This post introduces Redwood Research’s current alignment project: to ensure that a language model finetuned on fanfiction never describes someone getting injured, while maintaining the quality of the generations of that model. Their approach is to train a classifier that determines whether a given generation has a description of someone getting injured, and then to use that classifier as a reward function to train the policy to generate non-injurious completions. Their hope is to learn a general method for enforcing such constraints on models, such that they could then quickly train the model to, say, never mention anything about food.


I was pleasantly surprised at how understandable the paper was, and they do a good job of looking at exactly what their method is doing and how it accomplishes the attack in different ways under different settings.

Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks

Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, Tom Goldstein

Demonstrates a data poisoning attack in which the adversary gets to choose a poison input to add to the training set, but does _not_ get to choose its label. The goal is to misclassify a single test instance as a specific base class. They achieve this by creating a poison input that looks like the base class in pixel space but looks like the test instance in feature space (i.e. the activations in the penultimate layer). The poison input will be labeled by humans as the base class, and then when the network is retrained with the original dataset and the new poisoned input(s), it will classify the poison input as the base class, and with it the test instance as well (since they have very similar features).


When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks


Adversarial Attacks Against Medical Deep Learning Systems

Samuel G. Finlayson, Hyung Won Chung, Isaac S. Kohane, Andrew L. Beam


39th IEEE Symposium on Security and Privacy

Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning

Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, Bo Li


Learning Robust Rewards with Adversarial Inverse Reinforcement Learning

GAIL and Guided Cost Learning (GCL) both have the idea of learning a policy and a reward function simultaneously, in a GAN-like way (where the generator is the policy, and the discriminator has to distinguish between trajectories from the policy and the expert trajectories, which can then be interpreted as a reward function). In GAIL, the discriminator can’t be interpreted as a reward function, and you only get a policy as output (which is why it is called imitation learning). However, if you enforce that the discriminator has to be of the form:
D(τ) = exp(f(τ)) / (exp(f(τ)) + π(τ))
Then you can show that if π is trained to maximize
R(τ) = log(1 - D(τ)) - log(D(τ))
Then R and π converge to the optimal reward and policy respectively.

This trajectory-centric formulation is called GAN-GCL (the GAN version of guided cost learning). It’s main issue is that it works with trajectories and so is hard to optimize -- the gradients are very high variance. So, instead, we can work with individual state-action pairs instead of trajectories, just replacing every τ in the equations above with (s, a). This makes it more sample-efficient, and in this case f converges to the advantage function of the optimal policy. However, the advantage function induces a heavily entangled reward function, which rewards the RL agent for doing the action that the expert would have taken, without actually understanding the goal that the expert had. We would like to learn a disentangled reward, which they define as a reward function that leads to the optimal policy according to the true reward function even if the transition dynamics change.

Intuitively, since entanglement happens by rewarding the agent for taking the same action as the expert, we can do better by enforcing that the reward function only be a function of the state, so that it is forced to learn the actual goal rather than memorizing the actions that are good. They prove two theorems under the condition that the true reward is only a function of the state. First, the learned optimal reward function is fully disentangled if it is a function of only the state, assuming that the transition dynamics are “decomposable”. Second, the reward function must be a function of only the state if it is fully disentangled.

Now the discriminator in the formulation above is either looking at the trajectory as a whole, or looking at the current action in order to see whether or not you are matching the expert demonstrations. Clearly we can’t just make the discriminator a function of only the current state -- there’s no way that could distinguish between policies, since it has no access to information about the actions that the policies took. However, we can instead separate the discriminator’s f function into the reward term and a shaping term, and enforce that the shaping term does not change the optimal policy:
f(s, a, s’) = g(s) + γh(s’) − h(s)
(It happens to be the case that for any function h, adding γh(s’) − h(s) to the reward function does not change the optimal policy.) Now, the discriminator gets information about the action taken by the policy by seeing the next state s’ that resulted. Since γh(s’) − h(s) does not change the optimal policy, g(s) should converge to an optimal reward function, while h(s) must then be the value function V(s) in order to have f(s, a, s’) be the advantage function.

They run a bunch of experiments with recovering a reward function and then transferring it to a situation with different dynamics, and show that it works much better than any other algorithm. They also show that for direct imitation (no transfer required), it does about as well as GAIL.


Generative Adversarial Imitation Learning

To do imitation learning, you could try to learn the state-action correspondence that defines the expert policy via behavioral cloning (simply learn a model of action given state using supervised learning), but that’s brittle. Specifically, this is learning the policy over single timesteps, so any errors in one timestep lead to the next timestep being slightly out of distribution, leading to worse errors, giving compounding errors. So, you can instead use IRL to find the cost function, and use RL to then find a good policy. But IRL is slow. They propose a fast method that gets directly to the policy, but it’s as if you did IRL followed by RL.

They write down IRL as a maximization of a minimization that gives you a cost function, and then RL as a minimization over policies of the expected cost. Specifically, maxent RL(c) computes a good policy π given a cost function c by minimizing expected cost E[c] and maximizing entropy H(π), where H(π) is computed according to causal entropy, not regular entropy (see MaxCausalEnt from [AN #12](https://mailchi.mp/bcb2c6f1d507/alignment-newsletter-12)). IRL(π) finds a cost function c such that the expert policy is maximally better than the RL policy.

Let’s assume that we’re working with all cost functions c : S → A → Reals. This is a huge space, so let’s throw in a convex regularizer ψ for the IRL problem. Then, they prove that RL(IRL(π)) can be expressed as a single minimization involving ψ*, the convex conjugate of ψ. We can make different choices of ψ give us different imitation learning algorithms that minimize this objective.

Let ρ(s, a) be the occupancy measure of a policy π for a state-action pair (s, a). (Occupancy measure is the sum of the discounted probabilities of taking the given state-action pair.) Then, the expected cost of a trajectory is the sum of ρ(s, a) c(s, a) over all s, a. In the special case where you don’t do any regularization, the occupancy measure of the solution is guaranteed to be equal to the occupancy measure of the expert policy. (In the language of convex optimization, IRL and maxent RL are dual to each other, and strong duality applies.)

Of course, this all assumes that you have access to the full expert policy, whereas in practice you only have access to some expert demonstrations. So now, the question is how to choose a good ψ that deals with that issue. If you don’t regularize at all, you get something that matches the estimated occupancy measure exactly -- which is not great, since there are likely many areas of state space where there are not enough samples to accurately match the occupancy measure, at least in the case with stochastic dynamics. On the other hand, previous algorithms are basically using a ψ that is infinity on cost functions outside of a particular class, and constant elsewhere, which means that they only work if the cost function is actually in the class of learnable functions.

So, they instead propose a ψ that is dependent on the expert data. It requires that the cost function be negative everywhere, and puts a high penalty on any cost function that puts high cost (i.e. close to zero) on the expert data. Note that this can represent any (bounded) reward function, so it has very good expressive power. They chose the particular form they did because when you work through the convex conjugate, you end up choosing π to minimize the Jensen-Shannon divergence between the inferred policy and the expert policy, which can be interpreted as the optimal log loss of a classifier that distinguishes between the two policiees. This is very GAN-like, where we have a discriminator trying to distinguish between two policies, and a generator (the learned policy) that’s trying to fool the discriminator. Their proposed algorithm, GAIL, follows this quite precisely, alternating between an Adam gradient step to update the discriminator, and a TRPO step to update the policy. (Both the discriminator and the policy are neural nets.)

In the experiments, they imitate classic control and MuJoCo tasks that have been trained with TRPO. GAIL consistently achieves the expert performance, even with not much data, though this comes at a cost of lots of environment interaction (which eg. behavioral cloning does not require).


Maximum Entropy Inverse Reinforcement Learning

Brian D. Ziebart, Andrew Maas, J.Andrew Bagnell, and Anind K. Dey

While matching empirical feature counts helps to deal with the ambiguity of the reward functions, exactly matching featuer counts will typically require policies to be stochastic, in which case there are many stochastic policies that get the right feature counts. How do you pick among these policies? We should choose the distribution using the [principle of maximum entropy](https://en.wikipedia.org/wiki/Principle_of_maximum_entropy), which says to pick the stochastic policy (or alternatively, a probability distribution over trajectories) that has maximum entropy (and so the least amount of information). Formally, we’re trying to find a function P(ζ) that maximizes H(P), subject to E[features(ζ)] = empirical feature counts, and that P(ζ) is a probability distribution (sums to 1 and is non-negative for all trajectories). For the moment, we’re assuming deterministic dynamics.

We solve this constrained optimization problem using the method of Lagrange multipliers. With simply analytical methods, we can get to the standard MaxEnt distribution, where P(ζ | θ) is proportional to exp(θ f(ζ)). But where did θ come from? It is the Lagrange multiplier for constraint on expected feature counts. So we’re actually not done with the optimization yet, but this intermediate form is interesting in and of itself, because we can identify the Lagrange multiplier θ as the reward weights. Unfortunately, we can’t finish the optimization analytically -- however, we can compute the gradient for θ, which we can then use in a gradient descent algorithm. This gives the full MaxEnt IRL algorithm for deterministic environments. When you have (known) stochastic dynamics, we simply tack on the probability of the observed transitions to the model P(ζ | θ) and optimize from there, but this is not as theoretically compelling.

One warning -- when people say they are using MaxEnt IRL, they are usually actually talking about MaxCausalEnt IRL, which we'll discuss next.


Modeling Interaction via the Principle of Maximum Causal Entropy

Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey

When we have stochastic dynamics, MaxEnt IRL does weird things. It is basically trying to maximize the entropy H(A1, A2, ... | S1, S2, ...), subject to matching the feature expectations. However, when you choose the action A1, you don’t know what the future states are going to look like. What you really want to do is maximize the causal entropy, that is, you want to maximize H(A1 | S1) + H(A2 | S1, S2) + ..., so that each action’s entropy is only conditioned on the previous states, and not future states. You can then run through the same machinery as for MaxEnt IRL to get the MaxCausalEnt IRL algorithm.


A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress

This is a comprehensive survey of IRL that should be useful to researchers, or students looking to perform a deep dive into IRL. It's particularly useful because it can compare and contrast across many different IRL algorithms, whereas each individual IRL paper only talks about their method and a few particular weaknesses of other methods. If you want to learn a lot about IRL, I would start with the previous readings, then read this one, and perhaps after that read individual papers that interest you.


Learning from humans: what is inverse reinforcement learning?

This article introduces and summarizes the first few influential papers on inverse reinforcement learning. [Algorithms for IRL](http://ai.stanford.edu/~ang/papers/icml00-irl.pdf) attacked the problem by formulating it as a linear program, assuming that the given policy or demonstrations is optimal. However, there are many possible solutions to this problem -- for example, the zero reward makes any policy or demonstration optimal. [Apprenticeship Learning via IRL](http://people.eecs.berkeley.edu/~russell/classes/cs294/s11/readings/Abbeel+Ng:2004.pdf) lets you learn from an expert policy that is near-optimal. It assumes that the reward function is a weighted linear combination of _features_ of the state. In this case, given some demonstrations, we only need to match the feature expectations of the demonstrations in order to achieve the same performance as the demonstrations (since the reward is linear in the features). So, they do not need to infer the underlying reward function (which may be ambiguous).


AI Alignment Podcast: An Overview of Technical AI Alignment in 2018 and 2019

Lucas Perry, Buck Shlegeris and Rohin Shah

This podcast with Buck and me is loosely structured around the [review I wrote](https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review) ([AN #84](https://mailchi.mp/1af38085edc5/an-84-reviewing-ai-alignment-work-in-2018-19)), but with a lot more debate and delving into specific points of pessimism and optimism. I suspect that every reader will have some section they're interested in. Since much of the discussion was itself meant to be a summary, I'm not going to try and summarize even further. Here's the list of topics covered:

Our optimism and pessimism about different approaches to aligned AI
Traditional arguments for AI as an x-risk
Modeling agents as expected utility maximizers
Ambitious value learning and specification learning/narrow value learning
Agency and optimization
Robustness
Scaling to superhuman abilities
Universality
Impact regularization
Causal models, oracles, and decision theory
Discontinuous and continuous takeoff scenarios
Probability of AI-induced existential risk
Timelines for AGI
Information hazards


I really like this decomposition. I already laid out most of my thoughts back when I summarized just the <@main slide@>(@AI alignment landscape@); I still endorse them.

Current Work in AI Alignment

In this talk (whose main slide we covered <@before@>(@AI alignment landscape@)), Paul Christiano explains how he decomposes the problem of beneficial AI:

1. At the top level, "make AI go well" is decomposed into making AI competent, making AI aligned, and coping with the impacts of AI. Paul focuses on the alignment part, which he defines as building AI systems that are _trying_ to do what we want. See <@Clarifying "AI Alignment"@> and [my comment on it](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment#3ECKoYzFNW2ZqS6km). Paul considers many problems of competence as separate from alignment, including understanding humans well, and most reliability / robustness work.

2. Within alignment, we can consider the concept of an "alignment tax": the cost incurred by insisting that we only deploy aligned AI. One approach is to help pay the alignment tax, for example, by convincing important actors that they should care about alignment, or by adopting agreements that make it easier to coordinate to pay the tax, as with the <@OpenAI Charter@>). Technical AI safety research on the other hand can help _reduce_ the alignment tax, by creating better aligned AI systems (which consequently incur less cost than before).

3. With alignment tax reduction, we could either try to advance current alignable algorithms (making them more competent, and so reducing their tax), or make existing algorithms alignable. It would be particularly nice to take some general class of algorithms (such as deep reinforcement learning) and figure out how to transform them to make them alignable, such that improvements to the algorithms automatically translate to improvements in the alignable version. This is what Paul works on.

4. The next layer is simply a decomposition of possible algorithms we could try to align, e.g. planning, deduction, and learning. Paul focuses on learning.

5. Within aligned learning, we can distinguish between outer alignment (finding an objective that incentivizes aligned behavior) and inner alignment (ensuring that the trained agent robustly pursues the aligned objective). Paul works primarily on outer alignment, but has <@written about inner alignment@>(@Worst-case guarantees (Revisited)@).

6. Within outer alignment, we could either consider algorithms that learn from a teacher, such as imitiation learning or preference inference, or we could find algorithms that perform better than the teacher (as would be needed for superhuman performance). Paul focuses on the latter case.

7. To go beyond the teacher, you could extrapolate beyond what you've seen (i.e. generalization), do some sort of <@ambitious value learning@>(@What is ambitious value learning?@), or build a better teacher. Paul focuses on the last case, and thinks of amplification as a way to achieve this.


I've become more optimistic about recursive reward modeling since the <@original paper@>(@Scalable agent alignment via reward modeling@), primarily (I think) because I now see more value in approaches that can be used to perform specific tasks (relative to approaches that try to infer "human values").

I also appreciated the recommendations for the AI safety community, and agree with them quite a lot. Relative to Jan, I see more value in conceptual work described using fuzzy intuitions, but I do think that more effort should be put into exposition of that kind of work.

AI Alignment Podcast: On DeepMind, AI Safety, and Recursive Reward Modeling

While Jan originally worked on theory (specifically AIXI), DQN, AlphaZero and others demonstrated that deep RL was a plausible path to AGI, and so now Jan works on more empirical approaches. In particular, when selecting research directions, he looks for techniques that are deeply integrated with the current paradigm, that could scale to AGI and beyond. He also wants the technique to work for agents in general, rather than just question answering systems, since people will want to build agents that can act, at least in the digital world (e.g. composing emails). This has led him to work on <@recursive reward modeling@>(@Scalable agent alignment via reward modeling@), which tries to solve the specification problem in the <@SRA framework@>(@Building safe artificial intelligence: specification, robustness, and assurance@).

Reward functions are useful because they allow the AI to find novel solutions that we wouldn't think of (e.g. AlphaGo's move 37), but often are incorrectly specified, leading to reward hacking. This suggests that we should do _reward modeling_, where we learn a model of the reward function from human feedback. Of course, such a model is still likely to have errors leading to reward hacking, and so to avoid this, the reward model needs to be updated online. As long as it is **easier to evaluate behavior than to produce behavior**, reward modeling should allow AIs to find novel solutions that we wouldn't think of.

However, we would eventually like to apply reward modeling to tasks where evaluation is also hard. In this case, we can decompose the evaluation task into smaller tasks, and recursively apply reward modeling to train AI systems that can perform those small helper tasks. Then, assisted by these helpers, the human should be able to evaluate the original task. This is essentially forming a "tree" of reward modeling agents that are all building up to the reward model for the original, hard task. While currently the decomposition would be done by a human, you could in principle also use recursive reward modeling to automate the decomposition. Assuming that we can get regular reward modeling working robustly, we then need to make sure that the tree of reward models doesn't introduce new problems. In particular, it might be the case that as you go up the tree, the errors compound: errors in the reward model at the leaves lead to slightly worse helper agents, which lead to worse evaluations for the second layer, and so on.

He recommends that rather than spending a lot of time figuring out the theoretically optimal way to address a problem, AI safety researchers should alternate between conceptual thinking and trying to make something work. The ML community errs on the other side, where they try out lots of techniques, but don't think as much about how their systems will be deployed in the real world. Jan also wants the community to focus more on clear, concrete technical explanations, rather than vague blog posts that are difficult to critique and reason about. This would allow us to more easily build on past work, rather than reasoning from first principles and reinventing the wheel many times.

More broadly, DeepMind is taking a portfolio approach to AI safety: they are trying many different lines of attack, and hoping that some of them will pan out. Currently, there are teams for agent alignment (primarily recursive reward modeling), incentive theory, trained agent analysis, policy, and ethics. They have also spent some time thinking about AI safety benchmarks, as in [AI Safety Gridworlds](https://arxiv.org/abs/1711.09883), since progress in machine learning is driven by benchmarks, though Jan does think it is quite hard to create a well-made benchmark.


Here are a few points about this decomposition that were particularly salient or interesting to me.

First, at the top level, the problem is decomposed into alignment, competence, and coping with the impacts of AI. The "alignment tax" (extra technical cost for safety) is only applied to alignment, and not competence. While there isn't a tax in the "coping" section, I expect that is simply due to a lack of space; I expect that extra work will be needed for this, though it may not be technical. I broadly agree with this perspective: to me, it seems like the major technical problem which _differentially_ increases long-term safety is to figure out how to get powerful AI systems that are _trying_ to do what we want, i.e. they have the right [motivation](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment#3ECKoYzFNW2ZqS6km) ([AN #33](https://mailchi.mp/b6dc636f6a1b/alignment-newsletter-33)). Such AI systems will hopefully make sure to check with us before taking unusual irreversible actions, making e.g. robustness and reliability less important. Note that <@techniques like verification, transparency, and adversarial training@>(@Techniques for optimizing worst-case performance@) may still be needed to ensure that the _alignment_ itself is robust and reliable (see the inner alignment box); the claim is just that robustness and reliability of the AI's _capabilities_ is less important.

Second, strategy and policy work here is divided into two categories: improving our ability to pay technical taxes (extra work that needs to be done to make AI systems better), and improving our ability to handle impacts of AI. Often, generically improving coordination can help with both categories: for example, the <@publishing concerns around GPT-2@>(@Better Language Models and Their Implications@) have allowed researchers to develop synthetic text detection (the first category) as well as to coordinate on when not to release models (the second category).

Third, the categorization is relatively agnostic to the details of the AI systems we develop -- these only show up in level 4, where Paul specifies that he is mostly thinking about aligning learning, and not planning and deduction. It's not clear to me to what extent the upper levels of the decomposition make as much sense if considering other types of AI systems: I wouldn't be surprised if I thought the decomposition was not as good for risks from e.g. powerful deductive algorithms, but it would depend on the details of how deductive algorithms become so powerful. I'd be particularly excited to see more work presenting more concrete models of powerful AGI systems, and reasoning about risks in those models, as was done in <@Risks from Learned Optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@).

AI alignment landscape

This post presents the following decomposition of how to make AI go well:

IMAGE HERE


This is a welcome update to _Concrete Problems_ since it is slightly more concrete, current, and discusses improving safety in both deep learning and RL rather than mostly RL. While the document mentions many problems, the set of problems retains precision and fortunately does not include every capabilities concern that may possibly one day impact safety. A takeaway is that value learning and model transparency still need groundwork, but fortunately other problems including out-of-distribution robustness are more concretized and mostly need time and continued effort.

**Rohin's opinion:** One thing I particularly like about this agenda is that the connection to AI _alignment_ is significantly clearer than in _Concrete Problems_.

AI Alignment Research Overview

It has been over three years since _[Concrete Problems in AI Safety](https://arxiv.org/abs/1606.06565)_. Since that time we have learned more about the structure of the safety problem. This document represents an updated taxonomy of problems relevant for AI alignment. Jacob Steinhardt decomposes the remaining technical work into “technical alignment (the overcoming of conceptual or engineering issues needed to create aligned AI), detecting failures (the development of tools for proactively assessing the safety/alignment of a system or approach), methodological understanding (best practices backed up by experience), and system-building (how to tie together the three preceding categories in the context of many engineers working on a large system).”

The first topic under “technical alignment” is “Out-of-Distribution Robustness,” which receives more emphasis than it did in Concrete Problems. Out-of-Distribution Robustness is in part motivated by the fact that transformative AI will lead to substantial changes to the real world, and we should like our systems to perform well even under these large and possibly rapid data shifts. Specific subproblems include _some_ work on adversarial examples and out-of-distribution detection. Next, the problem of Reward Learning is described. For this, there are challenges including learning human values and ensuring those lossily represented human values can remain aligned under extreme optimization. While we have attained more conceptual clarity about reward learning since _Concrete Problems_, reward learning still remains largely “uncharted,” and it is still not clear “how approach the problem.” The next section on Scalable Reward Generation points out that, in the future, labeling meaning or providing human oversight will prove increasingly difficult. Next, he proposes that we ought to study how to make systems “act conservatively,” such as endowing systems with the ability to activate a conservative fallback routine when they are uncertain. The final topic under technical alignment is Counterfactual Reasoning. Here one possible direction is generating a family of simulated environments to generate counterfactuals.

The “technical alignment” section is the majority of this document. Later sections such as “Detecting Failures in Advance” highlight the importance of deep neural network visualization and recent model stress-test datasets. “Methodological Understanding” suggests that we are more likely to build aligned AI systems if we improve our best practices for building and evaluating models, and “System Building” speculates about how to do this for future multi-faceted ML systems.


International Joint Conference on Artificial Intelligence (IJCAI), 2018

This is great as a way to find references. I do wish there was more comparison between papers and/or approaches, but that's probably asking too much.

AGI Safety Literature Review

Self-explanatory. It's more of a list of approaches and references within each approach than an integrated whole, but I still expect it to be useful.


I'm glad this has finally been published -- it's been informing my views for a long time now. I broadly buy the general view put forward here, with a few nitpicks that you can see in [the post](https://www.alignmentforum.org/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services). I really do recommend you read at least the post -- that's just the _summary_ of the report, so it's full of insights, and it should be interesting to technical safety and strategy researchers alike.

I'm still not sure how this should affect what research we do -- techniques like preference learning and recursive reward modeling seem applicable to CAIS as well, since they allow us to more accurately specify what we want each individual service to do.

Reframing Superintelligence: Comprehensive AI Services as General Intelligence

This is a huge document; rather than summarize it all in this newsletter, I wrote up my summary in [this post](https://www.alignmentforum.org/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services). For this newsletter, I've copied over the description of the model, but left out all of the implications and critiques.

The core idea is to look at the pathway by which we will develop general intelligence, rather than assuming that at some point we will get a superintelligent AGI agent. To predict how AI will progress in the future, we can look at how AI progresses currently -- through research and development (R&D) processes. AI researchers consider a problem, define a search space, formulate an objective, and use an optimization technique in order to obtain an AI system, called a service, that performs the task.

A service is an AI system that delivers bounded results for some task using bounded resources in bounded time. Superintelligent language translation would count as a service, even though it requires a very detailed understanding of the world, including engineering, history, science, etc. Episodic RL agents also count as services.

While each of the AI R&D subtasks is currently performed by a human, as AI progresses we should expect that we will automate these tasks as well. At that point, we will have automated R&D, leading to recursive technological improvement. This is not recursive self-improvement, because the improvement comes from R&D services creating improvements in basic AI building blocks, and those improvements feed back into the R&D services. All of this should happen before we get any powerful AGI agents that can do arbitrary general reasoning.


AI Alignment Podcast: Inverse Reinforcement Learning and the State of AI Alignment

Lucas interviewed me and we talked about a bunch of different topics. Some quick highlights, without the supporting arguments:
- If we want to use inverse reinforcement learning (IRL) to infer a utility function that we then optimize, we would have to account for systematic biases, and this is hard, and subject to an impossibility result.
- Humans do seem to be good at inferring goals of other humans, probably because we model them as planning in a similar way that we ourselves plan. It's reasonable to think that IRL could replicate this. However, humans have very different ideas on how the future should go, so this seems not enough to get a utility function that can then be optimized over the long term.
- Another issue with having a utility function that is optimized over the long term is that it would have to somehow solve a whole lot of very difficult problems like the nature of identity and population ethics and metaphilosophy.
- Since human preferences seem to change as the environment changes, we could try to build AI systems whose goals are constantly changing by continuously running IRL. This sort of approach is promising but we don't know how to get it working yet.
- IRL, agency and optimization all seem to require a notion of counterfactuals.
- One view of agency is that it is about how a search process thinks of itself, or about other search processes. This gives it a feeling of "choice", even though the output of the search process is determined by physics. This can explain the debates over whether evolution is an optimization process -- on the one hand, it can be viewed as a search process, but on the other, we understand it well enough to think of it as a "deterministic" procedure.
- One way to view the AI alignment problem is to view it as a human-AI interaction problem, so that we get an AI that evolves over time along with us.
- Rather than building a function maximizer, we could aim to build an AI system that is corrigible, or one that follows norms.- Both iterated amplification and debate operate on an exponential deliberation tree, though in different ways, using reasoning learned from humans. If a human would have some desirable property (such as good epistemics), so too should their amplification.- Both iterated amplification and debate are based on _explicit_ human reasoning, as opposed to intuitive reasoning.
- Value drift in the literal sense can be both positive and negative -- I certainly expect and want my stated preferences to change as I become more knowledgeable in the future.
- We only want the combined human-AI system to have a goal, which allows for a space of possibilities where the AI is not optimizing a goal.
- One of the problems that seems most troubling is the issue of inner optimizers, which will hopefully be described in a sequence soon.


I'm excited to see what feels like quite a strong connection to an existing field of research. I especially liked the section about building in "common sense" (Section 5).

Incomplete Contracting and AI Alignment

This paper explores an analogy between AI alignment and incomplete contracting. In human society, we often encounter principal-agent problems, where we want to align the incentives of the agent with those of the principal. In theory, we can do this with a "complete" contract, that is an enforceable contract that fully specifies the optimal behavior in every possible situation. Obviously in practice we cannot write such contracts, and so we end up using incomplete contracts instead. Similarly, in AI alignment, in theory we could perfectly align an AI with humans by imbuing it with the true human utility function, but in practice this is impossible -- we cannot consider every possible situation that could come up. The difference between the behavior implied by the reward function we write down and the utility function we actually want leads to misalignment. The paper then talks about several ideas from incomplete contracting and their analogues in AI alignment. The main conclusion is that our AI systems will have to learn and use a "common sense" understanding of what society will and will not sanction, since that is what enables humans to solve principal-agent problems (to the extent that we can).


This is _definitely_ worth reading or listening if you haven't engaged much with Paul's work before, it will probably be my go-to reference to introduce someone to the approach. Even if you have, this podcast will probably help tie things together in a unified whole (at least, it felt that way to me). A lot of the specific things mentioned have been in the newsletter before, if you want to dig up my opinions on them.

80K podcast with Paul Christiano

This is a mammoth 4-hour interview that covers a lot of ground. I'll try to state the main points without the supporting arguments in roughly chronological order, listen to the podcast for more.
- The problem of AI safety is that we don't know how to build AI that does what we want it to do. It arises primarily because each actor faces a tradeoff between AI systems being maximally effective at its task, and being robustly beneficial.
- AI safety has had much more attention in the last few years.
- Everyone agrees that we don't know how to build AI that does what we want, but disagrees on how hard the problem is, or how it should be framed.
- The best arguments against working on alignment are opportunity cost (eg. working on biosecurity instead) and that the problem might be very easy or impossible, but even then it seems like work would be valuable for getting information about how hard the problem actually is.
- It's not very important for the best AI safety team to work with the best ML team for the purpose of pursuing alignment research, but it is important for actually building powerful aligned AI.
- The variance in outcomes from AGI come primarily from uncertainty in how hard the technical problem is, how people behave about AGI, and then how good we are at technical safety research. The last one is easiest to push on.
- It seems useful to build organizations that can make commitments that are credible to outsiders. This would allow the top AI actors to jointly commit that they meet a particular bar for safety, though this would also require monitoring and enforcing to be effective, which is hard to do without leaking information.
- We should expect [slow takeoff](https://sideways-view.com/2018/02/24/takeoff-speeds/), as Paul defines it. (I'm ignoring a lot of detail here.)
- We should focus on short timelines because we have more leverage over them, but the analogous argument for focusing on fast takeoff is not as compelling.
- Paul places 15% probability on human labor being obselete in 10 years, and 35% on 20 years, but doesn't think he has done enough analysis that people should defer to him.
- Comparing current AI systems to humans seems like the wrong way to measure progress in AI. Instead, we should consider what we'd be able to do now _if_ AI becomes comparable to humans in 10-20 years, and compare to that.
- We can decompose alignment into the problem of training an AI given a smarter overseer, and the problem of creating a sufficiently smart overseer. These roughly correspond to distillation and amplification respectively in IDA. (There's more discussion of IDA, but it should be pretty familiar to people who have engaged with IDA before.) Reactions fall into three camps: a) IDA is hopelessly difficult, b) IDA is focusing on far-away problems that will be easy by the time they are relevant, and c) optimistic about IDA.
- Very few people think about how to solve the full problem, that is, solve alignment in the limit of arbitrarily intelligent AI. MIRI doesn't think about the question because it seems obviously doomed to them, while the broader ML community wants to wait until we know how to build the system. The other approaches are [debate](https://blog.openai.com/debate/) ([AN #5](https://mailchi.mp/0ae5d69de63b/alignment-newsletter-5)), which is very related to IDA, and inverse reinforcement learning (IRL). However, there are key problems with IRL, and research hasn't shed much light on the core of the problem.
- AI safety via debate also shares the insight of IDA that we can use AI to help us define a better training signal for AI. (There's discussion of how debate works, that again should be familiar to anyone who has engaged with it before.) The biggest difficulty is whether human judges are actually capable of judging debates on truthfulness and usefulness, as opposed to eg. persuasiveness.
- There are three main categories of work to be done on IDA and debate -- engineering work to actually build systems, philosophical work to determine whether we would be happy with the output of IDA or debate, and ML research that allows us to try out IDA or debate with current ML techniques.
- We should focus on [prosaic AI](https://ai-alignment.com/prosaic-ai-control-b959644d79c2), that is, powerful AI built out of current techniques (so no unknown unknowns). This is easier to work on since it is very concrete, and even if AGI requires new techniques, it will probably still use current ones, and so work aligning current techniques should transfer. In addition, if current techniques go further than expected, it would catch people by surprise, which makes this case more important.
- With sufficient computation, current ML techniques can produce general intelligence, because evolution did so, and current ML looks a lot like evolution.
- The biggest crux between Paul and MIRI is whether prosaic AI can be aligned.
- One problem that MIRI thinks is irresolvable is the problem of inner optimizers, where even if you optimize a perfectly constructed objective function that captures what we want, you may create a consequentialist that has good behavior in training environments but arbitrarily bad behavior in test environments. However, we could try to solve this through techniques like adversarial training.
- The other problem is that constructing a good objective is incredibly difficult, and existing schemes are hiding the magic somewhere (for example, in IDA, it would be hidden in the amplification step).
- Research of the kind that MIRI does will probably be useful for answering the philosophical questions around IDA and debate.
- Ought's [Factored Cognition](https://ought.org/presentations/factored-cognition-2018-05) ([AN #12](https://mailch...newsletter-12)) project is very related to IDA.
- Besides learning ML, and careers in strategy and policy, Paul is excited for people to start careers studying problems around IDA from a CS lens, a philosophical lens, or a psychological lens (in the sense of studying how humans decompose problems for IDA, or how they judge debates).
- Computer security problems that are about attacking AI (rather than defending against attacks in a world with AI) are often very related to long term AI alignment.
- It is important for safety researchers to be respectful of ML researchers, since they are justifiably defensive given the high levels of external interest in safety that's off-base.
- EAs often incorrectly think in terms of a system that has been given a goal to optimize.
- Probably the most important question in moral philosophy is what kinds of unaligned AI would be morally valuable, and how they compare to the scenario where we build an aligned AI.
- Among super weird backup plans, we could build unaligned AI that is in expectation as valuable as aligned AI, which allows us to sidestep AI risk. For example, we could simulate other civilizations that evolution would produce, and hand over control of the world to a civilization that would have done the same thing if our places were swapped. From behind a veil of ignorance of "civilizations evolution could have produced", or from a multiverse perspective, this has the same expected value as building an _aligned_ AI (modulo the resource cost of simulations), allowing us to sidestep AI risk.
- We might face an issue where society worries about being bigoted towards AI and so gives them rights and autonomy, instead of focusing on the more important question of whether their values or goals align with ours.


I like this framing - I think it's an improvement on the categorisation into specification and robustness problems from the AI safety gridworlds paper. In particular, it's useful to separate properties that we want an AI to have from mechanisms by which we might control or limit the damage from an AI that doesn't have those properties.

I do worry that this approach doesn't contain scalability as a core concern in the same way that IDA and agent foundations do. Solutions to AI safety problems which work for sub-human-level AI and even human-level AI may not translate to superintelligent AI. Relatedly, I think that maintaining the same goals under distributional shift should be considered a specification problem, because as AIs get smarter they'll be able to handle tasks increasingly different from the ones they trained on, and also because scaling up a system is itself a form of distributional shift.

**Rohin's opinion:** I also like this framing. Unlike Richard, I am not worried about the absence of scalability as a core concern -- scalability seems to be a desideratum about _solutions_, while the blog post aims to categorize _problems_. I'm also more agnostic about the categorization of distributional shift -- I can see it as both a specification problem and a robustness problem, and want to figure out more precisely what I think the difference between specification and robustness is.

Building safe artificial intelligence: specification, robustness, and assurance

Pedro A. Ortega, Vishal Maini DeepMind safety team

In this blog post, the DeepMind safety team divides AI safety into the problems of specification, robustness and assurance. Specification ensures that an AI system's behaviour aligns with the operator's true intentions, i.e that our "ideal specification" of a fully alligned AI system is consistent with the AI's actual behavior. Problems in this category include avoiding side-effects and the creation of dangerous sub-agents. Robustness ensures that an AI system continues to operate within safe limits upon facing peturbations like adversarial inputs or distributional shift, both by preventing these problems arising and by being able to recover from them. It also includes criteria like safe exploration and cautious generalisation. Assurance ensures that we can understand and control AI systems during operation, by monitoring them and enforcing restrictions. Interpretability and interruptability are examples of monitoring and enforcement respectively. I'd encourage you to look at the table in the original post, since it also categorises many more AI safety problems.


This does feel like an important generator of many disagreements I've had. I'd split rationality realism into two subcases -- whether you expect that there is a simple "correct" algorithm for computation-bounded rationality, and whether you expect there is only a simple "correct" algorithm for rationality given infinite compute, but the bounded computation case may be a lot messier. (I'm guessing almost all rationality realists fall in the latter category, but I'm not sure.)

I'd expect most of the people working on reducing existential risk from AI to be much more realist about rationality, since we often start working on this based on astronomical waste arguments and utilitarianism, which seems very realist about preferences. (At least, this was the case for me.) This is worrying -- it seems plausible to me that there isn't a "correct" rationality or intelligence algorithm (even in the infinite compute case), but that we wouldn't realize this because people who believe that also wouldn't want to work on AI alignment.

Realism about rationality

In the same way that moral realism claims that there is one true morality (even though we may not know it yet), rationality realism is the claim that there is one "correct" algorithm for rationality or intelligence. This post argues that many disagreements can be traced back to differences on how much one identifies with the rationality realism mindset. For example, people who agree with rationality realism are more likely to think that there is a simple theoretical framework that captures intelligence, that there is an "ideal" decision theory, that certain types of moral reasoning are "correct", that having contradictory preferences or beliefs is really bad, etc. The author's skepticism about this mindset also makes them skeptical about agent foundations research.


Measurements generally seem great. One story for impact is that we have a measurement that we think is strongly correlated with x-risk, and we use that measurement to select an AI system that scores low on such a metric. This seems distinctly good and I think would in fact reduce x-risk! But I want to clarify that I don’t think it would convince me that the system was safe with high confidence. The conceptual arguments against high confidence in safety seem quite strong and not easily overcome by such measurements. (I’m thinking of <@objective robustness failures@>(@2-D Robustness@) of the form “the model is trying to pursue a simple proxy, but behaves well on the training distribution until it can execute a treacherous turn”.)

You can also tell stories where the measurements reveal empirical facts that then help us have high confidence in safety, by allowing us to build better theories and arguments, which can rule out the conceptual arguments above.

Separately, these measurements are also useful as a form of legible evidence about risk to others who are more skeptical of conceptual arguments.

RFP: Measuring and forecasting risks

Measurement and forecasting is useful for two reasons. First, it gives us empirical data that can improve our understanding and spur progress. Second, it can allow us to quantitatively compare the safety performance of different systems, which could enable the creation of safety standards. So what makes for a good measurement?

1. **Relevance to AI alignment:** The measurement exhibits a failure mode that becomes worse as models become larger, or tracks a potential capability that may emerge with further scale (which in turn could enable deception, hacking, resource acquisition, etc).
2. **Forward-looking:** The measurement helps us understand _future_ issues, not just those that exist today. Isolated examples of a phenomenon are good if we have nothing else, but we’d much prefer to have a systematic understanding of when a phenomenon occurs and how it tends to quantitatively increase or decrease with various factors. See for example <@scaling laws@>(@Scaling Laws for Neural Language Models@).
3. **Rich data source:** Not all trends in MNIST generalize to CIFAR-10, and not all trends in CIFAR-10 generalize to ImageNet. Measurements on data sources with rich factors of variation are more likely to give general insights.
4. **Soundness and quality:** This is a general category for things like “do we know that the signal isn’t overwhelmed by the noise” and “are there any reasons that the measurement might produce false positives or false negatives”.

What sorts of things might you measure?

1. As you scale up task complexity, how much do you need to scale up human-labeled data to continue to maintain good performance and avoid reward hacking? If you fail at this and there are imperfections in the reward, how bad does this become?
2. What changes do we observe based on changes in the _quality_ of the human feedback (e.g. getting feedback from amateurs vs experts)? This could give us information about the acceptable “difference in intelligence” between a model and its supervisor.
3. What happens when models are pushed out of distribution along a factor of variation that was not varied in the pretraining data?
4. To what extent do models provide wrong or undesired outputs in contexts where they are capable of providing the right answer?


RFP: Techniques for enhancing human feedback

Consider a topic previously analyzed in <@aligning narrowly superhuman models@>(@The case for aligning narrowly superhuman models@): how can we use human feedback to train models to do what we want in cases where the models are _more_ knowledgeable than the humans providing the feedback? A variety of techniques have been proposed to solve this problem, including <@iterated amplification@>(@Supervising strong learners by amplifying weak experts@), <@debate@>(@AI safety via debate@), <@recursive reward modeling@>(@Scalable agent alignment via reward modeling@), <@market making@>(@AI safety via market making@), and [generalizing from short deliberations to long deliberations](https://ai-alignment.com/turning-reflection-up-to-11-1bd6171afd21). This RFP solicits proposals that aim to test these or other mechanisms on existing systems. There are a variety of ways to set up the experiments so that the models are more knowledgeable than the humans providing the feedback, for example:

1. Train a language model to accurately explain things about a field that the feedback providers are not familiar with.
2. Train an RL agent to act well in an environment where the RL agent can observe more information than the feedback providers can.
3. Train a multilingual model to translate between English and a foreign language that the feedback providers do not know.