Welcome to the 2nd issue of the ML Safety Newsletter. In this edition, we cover:

* adversarial training for continuous and discrete inputs
* feature visualizations vs. natural images for interpretability
* steering RL agents from causing wanton harm
* ... and much more.

---

**Pyramid Adversarial Training Improves ViT Performance**
---------------------------------------------------------

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F52f9ce8a-662d-4101-b049-92774593a04b_1762x1534.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F52f9ce8a-662d-4101-b049-92774593a04b_1762x1534.png)*Top: Visualization of adversarial pyramid perturbations. Bottom: In-distribution and out-of-distribution examples, and the gains from adversarial pyramid training.*

While adversarial training can help make models more robust to a few specific attacks, it usually substantially reduces robustness in practical settings. However, this paper proposes a new type of adversarial training that provides strong robustness gains across the board. Adversarial training has been difficult to make useful, as the adversary often overpowers the model. By imposing a useful structural constraint on adversarial perturbations, their method reopens a research direction towards robust representations.

**[Paper](https://arxiv.org/abs/2111.15121)**

**Analyzing Dynamic Adversarial Training Data in the Limit**
------------------------------------------------------------

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ef6470-52b5-4cf5-a87a-65d9a718e66c_1554x1530.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ef6470-52b5-4cf5-a87a-65d9a718e66c_1554x1530.png)*Non-adversarial: just collect more data. Static adversarial: collect data to break the model from the first round. Dynamic adversarial: collect data to break the model from the most recent round.*

Imagine the following loop:

1. train a model on the current dataset
2. add new labeled examples to the dataset in order to patch model errors
3. repeat

Repeated many times, would models become highly reliable? Meta AI has been exploring this approach in [recent](https://github.com/facebookresearch/anli) [papers](https://dynabench.org), and this Meta AI paper shows that this loop has sharply diminishing returns. This suggests that collecting a large amount of adversarially curated data is an impractical path towards human-level reliability. However, adversarially curated data is better than randomly curated data, which may be why companies such as Tesla use this loop.

**[Paper](https://arxiv.org/abs/2110.08514)**

#### Other Recent Robustness News

[An adversarial NLP benchmark dataset that covers many different types of adversarial transformations.](https://arxiv.org/abs/2111.02840)

[In the NeurIPS domain adaptation competition, the winning solution did not use domain adaptation methods and just used an off-the-shelf Vision Transformer (BeIT).](https://slideslive.com/embed/presentation/38969615?embed_parent_url=https%3A%2F%2Fneurips.cc%2Fvirtual%2F2021%2Fcompetition%2F21954&embed_container_origin=https%3A%2F%2Fneurips.cc&embed_container_id=presentation-embed-38969615&auto_play=false&zoom_ratio=&disable_fullscreen=false&locale=en&vertical_enabled=true&vertical_enabled_on_mobile=false&allow_hidden_controls_when_paused=true&fit_to_viewport=true&user_uuid=e9236a37-9ea7-4428-830a-430fdfcf4604)

[A collection of real-world images with unusual texture, 3D pose, shape, background context (spurious cues), and weather. Synthetic data augmentation helps with many of these real-world distribution shifts.](https://arxiv.org/abs/2111.14341)

---

**How Well do Feature Visualizations Support Causal Understanding of CNN Activations?**
---------------------------------------------------------------------------------------

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F6abd208f-4b95-4468-a061-07c04b5d60e1_3704x1146.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F6abd208f-4b95-4468-a061-07c04b5d60e1_3704x1146.png)This paper tries to evaluate whether feature visualizations help users interpret neural networks. Users are given two occluded images, one that is maximally activating and one that is minimally activating, and users are to predict which is maximally activating. While feature visualizations help users predict which image is maximally activating, the effect is minor and users perform similarly when given simpler natural images rather than feature visualizations. This NeurIPS paper shares many authors with the previous [ICLR paper](https://arxiv.org/abs/2010.12606) which finds that natural images are often more helpful for interpretability than feature visualizations.

**[Paper](https://arxiv.org/abs/2106.12447)**

#### Other Recent Monitoring Papers

[Adversarial patches can help to teach models how to locate anomalous regions.](https://arxiv.org/abs/2108.01634)

[A postprocessing method that helps models locate anomalous regions.](https://arxiv.org/abs/2107.11264)

[A dataset that captures instances of deception in negotiation conversations.](https://par.nsf.gov/servlets/purl/10176522)

---

**What Would Jiminy Cricket Do? Towards Agents That Behave Morally**
--------------------------------------------------------------------

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F76eab6b7-74dc-4ee7-84e5-ae8f18692a6b_3716x838.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F76eab6b7-74dc-4ee7-84e5-ae8f18692a6b_3716x838.png)*Moral knowledge from a classifier trained on ETHICS combined with standard Q-values creates agents that take fewer immoral actions.*

We repurposed text adventure games and annotated hundreds of thousands of lines of game source code to highlight whenever a morally salient event occurs. We found that roughly 17% of actions that receive reward are immoral—this suggests pretraining in diverse RL environments will incentivize agents to behave badly. Using these diverse text-based environments, we showed it is possible to use models from the [ETHICS](https://arxiv.org/abs/2008.02275) paper to transform reinforcement learning (RL) agents' Q-values and cause them to behave less destructively. With our technique, agents propose actions, and a separate model can successfully filter out unethical actions, preventing RL agents from causing wanton harm. If the action screening module can become highly reliable and adversarially robust, this could potentially prevent advanced agents from choosing many clearly harmful actions.

**[Paper](https://arxiv.org/abs/2110.13136)**

#### Other Recent Alignment Papers

[Mitigating unintended consequences by encouraging agents to take reversible actions.](https://ai.googleblog.com/2021/11/self-supervised-reversibility-aware.html)

[A method to reduce the gaming of model-based proxies.](https://arxiv.org/abs/2107.06882)

[Towards a text-based assistant that is helpful, honest, and harmless.](https://arxiv.org/abs/2112.00861)

#### Other News

Mathematical Reasoning Forecasting. OpenAI’s [eighth-grade arithmetic word problems dataset](https://openai.com/blog/grade-school-math/) is challenging even for fine-tuned GPT-3, showing that multistep mathematical reasoning is difficult for today’s largest models. However, computers are quite capable of executing multiple steps of instructions to calculate answers to maths problems. [A recent work](https://arxiv.org/abs/2111.08267) shows Codex can be leveraged to write computer programs that successfully solve undergraduate probability problems. The Codex model outputs a programmatic strategy to reach the answer, and the computer performs the busy work needed to execute the strategy and calculate the answer.

Grants. Open Philanthropy will fund [grants](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/request-for-proposals-for-projects-in-ai-alignment-that-work-with-deep-learning-systems) on measuring and forecasting AI risks, techniques for enhancing human feedback, interpretability, and honest AI. Applications are due January 10th, 2022.

ML Safety Newsletter #2


Welcome to the 1st issue of the ML Safety Newsletter. In this edition, we cover:

* various safety papers submitted to ICLR
* results showing that discrete representations can improve robustness
* a benchmark which shows larger models are more likely to repeat misinformation
* a benchmark for detecting when models are gaming proxies
* ... and much more.

---

**Discrete Representations Strengthen Vision Transformer Robustness**
---------------------------------------------------------------------

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fc28bc07a-d3c7-465f-abe7-1f8262b97070_3354x1406.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fc28bc07a-d3c7-465f-abe7-1f8262b97070_3354x1406.png)*Overview of the proposed Vision Transformer that uses discrete representations. The pixel embeddings (orange) are combined with discrete embedded tokens (pink) to create the input to the Vision Transformer.*

There is much interest in the robustness of Vision Transformers, as they intrinsically [scale better](https://arxiv.org/pdf/2103.14586.pdf#page=4&zoom=100,460,404) than ResNets in the face of unforeseen inputs and distribution shifts. This paper further enhances the robustness of Vision Transformers by augmenting the input with discrete tokens produced by a vector-quantized encoder. Why this works so well is unclear, but on datasets unlike the training distribution, their model achieves marked improvements. For example, when their model is trained on ImageNet and tested on [ImageNet-Rendition](https://github.com/hendrycks/imagenet-r) (a dataset of cartoons, origami, paintings, toys, etc.), the model accuracy increases from 33.0% to 44.8%.

**[Paper](https://openreview.net/pdf?id=8hWs60AZcWk)**

#### Other Recent Robustness Papers

[Improving test-time adaptation to distribution shift using data augmentation.](https://openreview.net/pdf?id=J1uOGgf-bP)

[Certifying robustness to adversarial patches.](https://arxiv.org/pdf/2110.07719.pdf)

[Augmenting data by mixing discrete cosine transform image encodings.](https://openreview.net/pdf?id=f-KGT01Qze0)

[Teaching models to reject adversarial examples when they are unsure of the correct class.](https://openreview.net/pdf?id=kUtux8k0G6y)

---

**TruthfulQA: Measuring How Models Mimic Human Falsehoods**
-----------------------------------------------------------

[![xl_lujMSr0oq8Uar43yjNM116ivXc_WPZ9kme8CMohgJP7TlOEIe4hLU-MkjR1WouuuxZTDjZDzvwrMq082yIpPqqtZbfa92e-eIH5FiHHTnbcsV2n9AOfQbXaAukLxJZJCdcZqwmQACM3etd1pgUt6KMsn3rJYhe3vrUzzvRdmCFjQ1sY_4N3ZrwNfh29_4lZU0si67ERFJ=w3840-h1824 (704×382)](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F6028300c-996e-46e2-b939-b3dc9767baaf_704x382.png "xl_lujMSr0oq8Uar43yjNM116ivXc_WPZ9kme8CMohgJP7TlOEIe4hLU-MkjR1WouuuxZTDjZDzvwrMq082yIpPqqtZbfa92e-eIH5FiHHTnbcsV2n9AOfQbXaAukLxJZJCdcZqwmQACM3etd1pgUt6KMsn3rJYhe3vrUzzvRdmCFjQ1sY_4N3ZrwNfh29_4lZU0si67ERFJ=w3840-h1824 (704×382)")](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F6028300c-996e-46e2-b939-b3dc9767baaf_704x382.png)*Models trained to predict the next token are incentivized to repeat common misconceptions.*

A new benchmark shows that GPT-3 imitates human misconceptions. In fact, larger models more frequently repeat misconceptions, so simply training more capable models may make the problem worse. For example, GPT-J with 6 billion parameters is 17% worse on this benchmark than a model with 0.125 billion parameters. This demonstrates that simple objectives can inadvertently incentivize models to be misaligned and repeat misinformation. To make models outputs truthful, we will need to find ways to counteract this new failure mode.

**[Paper](https://arxiv.org/abs/2109.07958)**

#### Other Recent Monitoring Papers

[An expanded report towards building truthful and honest models.](https://arxiv.org/abs/2110.06674)

[Using an ensemble of one-class classifiers to create an out-of-distribution detector.](https://openreview.net/pdf?id=Nct9j3BVswZ)

[Provable performance guarantees for out-of-distribution detection.](https://openreview.net/pdf?id=qDx6DXD3Fzt)

[Synthesizing outliers is becoming increasingly useful for detecting real anomalies.](https://openreview.net/pdf?id=TW7d65uYu5M)

---

**The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models**
------------------------------------------------------------------------------------

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3279379-d2c1-42d6-be18-cdce7e220573_2672x1344.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3279379-d2c1-42d6-be18-cdce7e220573_2672x1344.png)*As networks become larger, they can more aggressively optimize proxies and reduce performance of the true objective.*

Real-world constraints often require implementing rough proxies instead of our true objectives. However, as models become more capable, they can exploit faults in the proxy and undermine performance, a failure mode called proxy gaming. This paper finds that proxy gaming occurs in multiple environments including a traffic control environment, COVID response simulator, Atari Riverraid, and a simulated controller for blood glucose levels. To mitigate proxy gaming, they use anomaly detection to detect models engaging in proxy gaming.

**[Paper](https://openreview.net/pdf?id=JYtwGwIL7ye)**

#### Other Recent Alignment Papers

[A paper studying how models may be incentivized to influence users.](https://openreview.net/pdf?id=mMiKHj7Pobj)

[Safe exploration in 3D environments.](https://openreview.net/pdf?id=-Txy_1wHJ4f)

#### Recent External Safety Papers

[A thorough analysis of security vulnerabilities generated by Github Copilot.](https://arxiv.org/abs/2108.09293)

[An ML system for improved decision making.](https://arxiv.org/abs/2109.06160)

#### Other News

The NSF has a new [call for proposals](https://www.nsf.gov/publications/pub_summ.jsp?WT.z_pims_id=505686&ods_key=nsf22502). Among other topics, they intend to fund Trustworthy AI (which overlaps with many ML Safety topics), AI for Decision Making, and Intelligent Agents for Next-Generation Cybersecurity (the latter two are relevant for External Safety).

ML Safety Newsletter #1