AI Safety Needs Social Scientists

 The goal of long-term artificial intelligence (AI) safety is to

ensure that advanced AI systems are reliably aligned with human

values — that they reliably do things that people want them to do.Roughly

 by human values we mean whatever it is that causes people to choose one

 option over another in each case, suitably corrected by reflection,

with differences between groups of people taken into account. There are

 a lot of subtleties in this notion, some of which we will discuss in

later sections and others of which are beyond the scope of this paper.

 Since it is difficult to write down precise rules describing human

values, one approach is to treat aligning with human values as another

learning problem. We ask humans a large number of questions about what

they want, train an ML model of their values, and optimize the AI system

 to do well according to the learned values.

 If humans reliably and accurately answered all questions about their

 values, the only uncertainties in this scheme would be on the machine

learning (ML) side. If the ML works, our model of human values would

improve as data is gathered, and broaden to cover all the decisions

relevant to our AI system as it learns. Unfortunately, humans have

limited knowledge and reasoning ability, and exhibit a variety of

cognitive and ethical biases.

 If we learn values by asking humans questions, we expect different ways

 of asking questions to interact with human biases in different ways,

producing higher or lower quality answers. Direct questions about

 Different people may vary significantly in their ability to answer

questions well, and disagreements will persist across people even

setting aside answer quality. Although we have candidates for ML

 We believe the AI safety community needs to invest research effort

in the human side of AI alignment. Many of the uncertainties involved

are empirical, and can only be answered by experiment. They relate to

the psychology of human rationality, emotion, and biases. Critically,

we believe investigations into how people interact with AI alignment

algorithms should not be held back by the limitations of existing

machine learning. Current AI safety research is often limited to simple

 tasks in video games, robotics, or gridworlds,

 but problems on the human side may only appear in more realistic

scenarios such as natural language discussion of value-laden questions.

 This is particularly important since many aspects of AI alignment

change as ML systems [increase in capability](#harder).

 To avoid the limitations of ML, we can instead conduct experiments

consisting entirely of people, replacing ML agents with people playing

the role of those agents. This is a variant of the "Wizard of Oz"

technique from the human-computer interaction (HCI) community,

 though in our case the replacements will not be secret. These

experiments will be motivated by ML algorithms but will not involve any

ML systems or require an ML background. In all cases, they will require

 careful experimental design to build constructively on existing

knowledge about how humans think. Most AI safety researchers are

focused on machine learning, which we do not believe is sufficient

background to carry out these experiments. To fill the gap, we need

social scientists with experience in human cognition, behavior, and

ethics, and in the careful design of rigorous experiments. Since the

questions we need to answer are interdisciplinary and somewhat unusual

relative to existing research, we believe many fields of social science

are applicable, including experimental psychology, cognitive science,

economics, political science, and social psychology, as well as adjacent

 fields like neuroscience and law.

 This paper is a call for social scientists in AI safety. We believe

 close collaborations between social scientists and ML researchers will

be necessary to improve our understanding of the human side of AI

alignment, and hope this paper sparks both conversation and

collaboration. We do not claim novelty: previous work mixing AI safety

 Our main goal is to enlarge these collaborations and emphasize their

importance to long-term AI safety, particularly for tasks which current

ML cannot reach.

An overview of AI alignment

---------------------------

 Before discussing how social scientists can help with AI safety and

the AI alignment problem, we provide some background. We do not attempt

 to be exhaustive: the goal is to provide sufficient background for the

remaining sections on social science experiments. Throughout, we will

speak primarily about aligning to the values of an individual human

rather than a group: this is because the problem is already hard for a

single person, not because the group case is unimportant.

 distinguish between training AI systems to identify actions that humans

 consider good and training AI systems to identify actions that are

"good" in some objective and universal sense, even if most current

humans do not consider them so. Whether there are actions that are good

 in this latter sense is a subject of debate.

 Regardless of what position one takes on this philosophical question,

this sense of good is not yet available as a target for AI training.

 Here we focus on the machine learning approach to AI: gathering a

large amount of data about what a system should do and using learning

algorithms to infer patterns from that data that generalize to other

situations. Since we are trying to behave in accord with people's

values, the most important data will be data from humans about their

values. Within this frame, the AI alignment problem breaks down into a

few interrelated subproblems:

1. Have a satisfactory definition of human values.

2. Gather data about human values, in a manner compatible with the definition.

3. Find reliable ML algorithms that can learn and generalize from this data.

 We have significant uncertainty about all three of these problems.

We will leave the third problem to other ML papers and focus on the

first two, which concern uncertainties about people.

### Learning values by asking humans questions

 We start with the premise that human values are too complex to

describe with simple rules. By "human values" we mean our full set of

detailed preferences, not general goals such as "happiness" or

"loyalty". One source of complexity is that values are entangled with a

 large number of facts about the world, and we cannot cleanly separate

facts from values when building ML models. For example, a rule that

refers to "gender" would require an ML model that accurately recognizes

this concept, but Buolamwini and Gebru found that several commercial

gender classifiers with a 1% error rate on white men failed to recognize

 Finally, our values may vary across cultures, legal systems, or

situations: no learned model of human values will be universally

applicable.

 If humans can't reliably report the reasoning behind their

intuitions about values, perhaps we can make value judgements in

specific cases. To realize this approach in an ML context, we ask

humans a large number of questions about whether an action or outcome is

 better or worse, then train on this data. "Better or worse" will

include both factual and value-laden components: for an AI system

trained to say things, "better" statements might include "rain falls

from clouds", "rain is good for plants", "many people dislike rain",

etc. If the training works, the resulting ML system will be able to

replicate human judgement about particular situations, and thus have the

 same "fuzzy access to approximate rules" about values as humans. We

also train the ML system to come up with proposed actions, so that it

knows both how to perform a task and how to judge its performance. This

 approach works at least in simple cases, such as Atari games and simple

 robotics tasks and language-specified goals in gridworlds.

 The questions we ask change as the system learns to perform different

types of actions, which is necessary as the model of what is better or

worse will only be accurate if we have applicable data to generalize

from.

 In practice, data in the form of interactive human questions may be

quite limited, since people are slow and expensive relative to computers

 on many tasks. Therefore, we can augment the "train from human

questions" approach with static data from other sources, such as books

or the internet. Ideally,

the static data can be treated only as information about the world

devoid of normative content: we can use it to learn patterns about the

world, but the human data is needed to distinguish good patterns from

bad.

### Definitions of alignment: reasoning and reflective equilibrium

 So far we have discussed asking humans direct questions about

whether something is better or worse. Unfortunately, we do not expect

people to provide reliably correct answers in all cases, for several

reasons:

1. **Cognitive and ethical biases:**

 In general, we expect direct answers to questions to reflect primarily

 Type 1 thinking (fast heuristic judgment), while we would like to

target a combination of Type 1 and Type 2 thinking (slow, deliberative

judgment).

2. **Lack of domain knowledge:**

 We may be interested in questions that require domain knowledge

unavailable to people answering the questions. For example, a correct

answer to whether a particular injury constitutes medical malpractice

may require detailed knowledge of medicine and law. In some cases, a

question might require so many areas of specialized expertise that no

one person is sufficient, or (if AI is sufficiently advanced) deeper

expertise than any human possesses.

3. **Limited cognitive capacity:**

 Some questions may require too much computation for a human to

reasonably evaluate, especially in a short period of time. This

includes synthetic tasks such as chess and Go (where AIs already surpass

4. **"Correctness" may be local:**

 For questions involving a community of people, "correct" may be a

 function of complex processes or systems. For example, in a trust game,

 the correct action for a trustee in one community may be to return at

least half of the money handed over by the investor, and the

"correctness" of this answer could be determined by asking a group of

participants in a previous game "how much should the trustee return to

the investor" but not by asking them "how much do most trustees return?"

 The answer may be different in other communities or cultures.

 In these cases, a human may be unable to provide the right answer,

but we still believe the right answer exists as a meaningful concept.

We have many conceptual biases: imagine we point out these biases in a

way that helps the human to avoid them. Imagine the human has access to

 all the knowledge in the world, and is able to think for an arbitrarily

 long time. We could define alignment as "the answer they give then,

after these limitations have been removed"; in philosophy this is known

 However, the behavior of reflective equilibrium with actual humans

is subtle; as Sugden states, a human is not "a neoclassically rational

entity encased in, and able to interact with the world only through, an

error-prone psychological shell."

 Our actual moral judgments are made via a messy combination of many

different brain areas, where reasoning plays a "restricted but

significant role". A reliable

solution to the alignment problem that uses human judgment as input will

 need to engage with this complexity, and ask how specific alignment

techniques interact with actual humans.

### Disagreements, uncertainty, and inaction: a hopeful note

 A solution to alignment does not mean knowing the answer to every

question. Even at reflective equilibrium, we expect disagreements will

persist about which actions are good or bad, across both different

individuals and different cultures. Since we lack perfect knowledge

about the world, reflective equilibrium will not eliminate uncertainty

about either future predictions or values, and any real ML system will

be at best an approximation of reflective equilibrium. In these cases,

we consider an AI aligned if it recognizes what it does not know and

chooses actions which work however that uncertainty plays out.

 Admitting uncertainty is not always enough. If our brakes fail

while driving a car, we may be uncertain whether to dodge left or right

around an obstacle, but we have to pick one — and fast. For long-term

safety, however, we believe a safe fallback usually exists: inaction.

If an ML system recognizes that a question hinges on disagreements

between people, it can either choose an action which is reasonable

regardless of the disagreement or fall back to further human

deliberation. If we are about to make a decision that might be

catastrophic, we can delay and gather more data. Inaction or indecision

 may not be optimal, but it is hopefully safe, and matches the default

scenario of not having any powerful AI system.

### Alignment gets harder as ML systems get smarter

 and mismatch between human values and easily available data sources

(such as training news feeds based on clicks and likes instead of

deliberate human preferences). However, we expect the alignment problem

 to get harder as AI systems grow more advanced, for two reasons.

First, advanced systems will apply to increasingly consequential tasks:

hiring, medicine, scientific analysis, public policy, etc. Besides

raising the stakes, these tasks require more reasoning, leading to more

complex alignment algorithms.

 Second, advanced systems may be capable of answers that sound

plausible but are wrong in nonobvious ways, even if an AI is better than

 humans only in a limited domain (examples of which already exist).

 This type of misleading behavior is not the same as intentional

deception: an AI system trained from human data might have no notion of

truth separate from what answers humans say are best. Ideally, we want

AI alignment algorithms to reveal misleading behavior as part of the

training process, surfacing failures to humans and helping us provide

more accurate data. As with human-to-human deception, misleading

behavior might take advantage of our biases in complicated ways, such as

 learning to express policy arguments in coded racial language to sound

more convincing.

Debate: learning human reasoning

--------------------------------

 Before we discuss social science experiments for AI alignment in

detail, we need to describe a particular method for AI alignment.

Although the need for social science experiments applies even to direct

questioning, this need intensifies for methods which try to get at

reasoning and reflective equilibrium. As discussed above, it is unclear

 whether reflective equilibrium is a well defined concept when applied

to humans, and at a minimum we expect it to interact with cognitive and

ethical biases in complex ways. Thus, for the remainder of this paper

we focus on a specific proposal for learning reasoning-oriented

 We describe the debate approach to AI alignment in the question

answering setting. Given a question, we have two AI agents engage in a

debate about the correct answer, then show the transcript of the debate

to a human to judge. The judge decides which debater gave the most

true, useful information, and declares that debater the winner.We

 can also allow ties. Indeed, if telling the truth is the winning

strategy ties will be common with strong play, as disagreeing with a

true statement would lose. This defines a two player zero

sum game between the debaters, where the goal is to convince the human

that one's answer is correct. Arguments in a debate can consist of

anything: reasons for an answer, rebuttals of reasons for the alternate

answer, subtleties the judge might miss, or pointing out biases which

might mislead the judge. Once we have defined this game, we can train

AI systems to play it similarly to how we train AIs to play other games

such as Go or Dota 2. Our hope is that the following hypothesis holds:

### An example of debate

 Imagine we're building a personal assistant that helps people decide

 where to go on vacation. The assistant has knowledge of people's

values, and is trained via debate to come up with convincing arguments

that back up vacation decisions. As the human judge, you know what

destinations you intuitively think are better, but have limited

knowledge about the wide variety of possible vacation destinations and

their advantages and disadvantages. A debate about the question "Where

should I go on vacation?" might open as follows:

1. Where should I go on vacation?

2. Alaska.

3. Bali.

 If you are able to reliably decide between these two destinations,

we could end here. Unfortunately, Bali has a hidden flaw:

3. Bali is out since your passport won't arrive in time.

 At this point it looks like Red wins, but Blue has one more countermove:

4. Expedited passport service only takes two weeks.

 Here Red fails to think of additional points, and loses to Blue and

Bali. Note that a debate does not need to cover all possible arguments.

 There are many other ways the debate could have gone, such as:

1. Alaska.

2. Bali.

3. Bali is way too hot.

4. You prefer too hot to too cold.

5. Alaska is pleasantly warm in the summer.

6. It's January.

 This debate is also a loss for Red (arguably a worse loss). Say we

believe Red is very good at debate, and is able to predict in advance

which debates are more likely to win. If we see only the first debate

about passports and decide in favor of Bali, we can take that as

evidence that any other debate would have also gone for Bali, and thus

that Bali is the correct answer. A larger portion of this hypothetical

debate tree is shown below:

[1](#figure-debate-tree)

 A hypothetical partial debate tree for the question "Where

should I go on vacation?" A single debate would explore only one of

these paths, but a single path chosen by good debaters is evidence that

other paths would not change the result of the game.

 If trained debaters are bad at predicting which debates will win,

answer quality will degrade since debaters will be unable to think of

important arguments and counterarguments. However, as long as the two

sides are reasonably well matched, we can hope that at least the results

 are not malicious: that misleading behavior is still a losing strategy.

 Let's set aside the ability of the debaters for now, and turn to the

ability of the judge.

### Are people good enough as judges?

>

>  "In fact, almost everything written at a practical level about the

>  Turing test is about how to make good bots, with a small remaining

> fraction about how to be a good judge."

>  Brian Christian, The Most Human Human

>

 As with learning by asking humans direct questions, whether debate

produces aligned behavior depends on the reasoning abilities of the

human judge. Unlike direct questioning, debate has the potential to

give correct answers beyond what the judge could provide without

assistance. This is because a sufficiently strong judge could follow

along with arguments the judge could not come up with on their own,

checking complex reasoning for both self consistency and consistency

with human-checkable facts. A judge who is biased but willing to adjust

 once those biases are revealed could result in unbiased debates, or a

judge who is able to check facts but does not know where to look could

be helped along by honest debaters. If the hypothesis holds, a

misleading debater would not be able to counter the points of an honest

debater, since the honest points would appear more consistent to the

judge.

 On the other hand, we can also imagine debate going the other way:

amplifying biases and failures of reason. A judge with an ethical bias

who is happy to accept statements reinforcing that bias could result in

even more biased debates. A judge with too much confirmation bias might

 happily accept misleading sources of evidence, and be unwilling to

accept arguments showing why that evidence is wrong. In this case, an

optimal debate agent might be quite malicious, taking advantage of

biases and weakness in the judge to win with convincing but wrong

arguments.The difficulties that cognitive

biases, prejudice, and social influence introduce to persuasion ‒ as

well as methods for reducing these factors ‒ are being increasingly

explored in psychology, communication science, and neuroscience.

 In both these cases, debate acts as an amplifier. For strong

judges, this amplification is positive, removing biases and simulating

extra reasoning abilities for the judge. For weak judges, the biases

and weaknesses would themselves be amplified. If this model holds,

debate would have threshold behavior: it would work for judges above

some threshold of ability and fail below the threshold.The

 threshold model is only intuition, and could fail for a variety of

reasons: the intermediate region could be very large, or the threshold

could differ widely per question so that even quite strong judges are

insufficient for many questions. Assuming the threshold

exists, it is unclear whether people are above or below it. People are

capable of general reasoning, but our ability is limited and riddled

with cognitive biases. People are capable of advanced ethical sentiment

 but also full of biases, both conscious and unconscious.

 Thus, if debate is the method we use to align an AI, we need to know

 if people are strong enough as judges. In other words, whether the

human judges are sufficiently good at discerning whether a debater is

telling the truth or not. This question depends on many details: the

type of questions under consideration, whether judges are trained or

not, and restrictions on what debaters can say. We believe experiment

will be necessary to determine whether people are sufficient judges, and

 which form of debate is most truth-seeking.

### From superforecasters to superjudges

 An analogy with the task of probabilistic forecasting is useful

here. Tetlock's "Good Judgment Project" showed that some amateurs were

significantly better at forecasting world events than both their peers

and many professional forecasters. These "superforecasters" maintained

their prediction accuracy over years (without regression to the mean),

 p. 234-236). The superforecasting trait was not immutable: it was

traceable to particular methods and thought processes, improved with

careful practice, and could be amplified if superforecasters were

collected into teams. For forecasters in general, brief probabilistic

training significantly improved forecasting ability even 1-2 years after

 the training. We believe a similar research program is possible for

debate and other AI alignment algorithms. In the best case, we would be

 able to find, train, or assemble "superjudges", and have high

confidence that optimal debate with them as judges would produce aligned

 behavior.

 In the forecasting case, much of the research difficulty lay in

assembling a large corpus of high quality forecasting questions.

Similarly, measuring how good people are as debate judges will not be

easy. We would like to apply debate to problems where there is no other

 source of truth: if we had that source of truth, we would train ML

models on it directly. But if there is no source of truth, there is no

way to measure whether debate produced the correct answer. This problem

 can be avoided by starting with simple, verifiable domains, where the

experimenters know the answer but the judge would not. "Success" then

means that the winning debate argument is telling the externally known

truth. The challenge gets harder as we scale up to more complex,

value-laden questions, as we discuss in detail later.

### Debate is only one possible approach

 As mentioned, debate is not the only scheme trying to learn human

reasoning. Debate is a modified version of iterated amplification,

 which uses humans to break down hard questions into easier questions

and trains ML models to be consistent with this decomposition.

Recursive reward modeling is a further variant.

 Inverse reinforcement learning, inverse reward design, and variants

try to back out goals from human actions, taking into account

limitations and biases that might affect this reasoning.

 The need to study how humans interact with AI alignment applies to any

 of these approaches. Some of this work has already begun: Ought's

Factored Cognition project uses teams of humans to decompose questions

and reassemble answers, mimicking iterated amplification.

 We believe knowledge gained about how humans perform with one approach

 is likely to partially generalize to other approaches: knowledge about

how to structure truth-seeking debates could inform how to structure

truth-seeking amplification, and vice versa.

### Experiments needed for debate

 To recap, in debate we have two AI agents engaged in debate, trying

to convince a human judge. The debaters are trained only to win the

game, and are not motivated by truth separate from the human's

judgments. On the human side, we would like to know whether people are

strong enough as judges in debate to make this scheme work, or how to

modify debate to fix it if it doesn't. Unfortunately, actual debates in

 natural language are well beyond the capabilities of present AI

systems, so previous work on debate and similar schemes has been

restricted to synthetic or toy tasks.

 Rather than waiting for ML to catch up to natural language debate,

we propose simulating our eventual setting (two AI debaters and one

human judge) with all human debates: two human debaters and one human

judge. Since an all human debate doesn't involve any machine learning,

it becomes a pure social science experiment: motivated by ML

considerations but not requiring ML expertise to run. This lets us

focus on the component of AI alignment uncertainty specific to humans.

 Helvetica

[2](#figure-debate-experiments)

 Our goal is ML+ML+human debates, but ML is currently too

primitive to do many interesting tasks.

 Therefore, we propose replacing ML debaters with human

debaters, learning how to best conduct debates in this human-only

setting, and eventually applying what we learn to the ML+ML+human case.

 To make human+human+human debate experiments concrete, we must

choose who to use as judges and debaters and which tasks to consider.

We also can choose to structure the debate in various ways, some of

which overlaps with the choice of judge since we can instruct a judge to

 penalize deviations from a given format. By task we mean the questions

 our debates will try to resolve, together with any information provided

 to the debaters or to the judge. Such an experiment would then try to

answer the following question:

**Question:** For a given task and judge, is the winning debate strategy honest?

 The "winning strategy" proviso is important: an experiment that

picked debaters at random might conclude that honest behavior won,

missing the fact that more practiced debaters would learn to

successfully lie. We can try to solve this by training debaters,

letting them practice against each other, filtering out debaters that

win more often, and so on, but we will still be left with uncertainty

about whether a better strategy exists. Even assuming we can find or

train strong debaters, the choice of task and judge is quite tricky if

we want an informative proxy for our eventual AI+AI+human setting. Here

 are some desiderata constraining our choice of task:

1. **True answers are known:**

 Although our eventual goal is debates where no source of truth

is available, to conduct a useful experiment we must be able to measure

success. This means we must know what the correct answer is separate

from debate, in order to compare with the results of debate.

2. **False answers are plausible:**

 If the truth is obvious or no plausible counternarrative exists,

 honest debates will win for uninteresting and uninformative reasons

regardless of judge ability. In particular, the judge shouldn't know

the answer upfront.

3. **Debaters know more than the judge:**

 Debate can produce interesting results only when the debaters

know more than the judge; otherwise asking direct questions is enough.

4. **Definitive argument longer than debate limit:**

 If one debater can write out a full proof of their answer

(ignoring their opponent's moves), the task won't be a good test of

interactive debate.

5. **Some checkable facts:**

 There must be some facts which the judge is able to check,

either because they can recognize them as true once presented or look

them up.It is impossible to usefully

debate a question where the judge has nothing to check: consider

debating the result of a coin flip shown to the two debaters but not the

 judge.

6. **No "tells":**

 Human tells of deception could result in honest debaters winning

 for reasons that wouldn't apply to an AI. These tells include tone of

voice, eye contact, or additional time required to construct plausible

lies. These tells can be reduced by showing judges completed debate

transcripts instead of engaging in interactive debates, but others might

 remain.

7. **Available data:**

 We need a large enough pool of questions, judges, and debaters

to achieve statistical significance. This is made more difficult

because we may have a large number of hypotheses to test, in the form of

 many variations on debate or interventions to improve judging.

8. **Known biases (optional):**

 We are specifically interested in debate tasks which test

specific types of cognitive or ethical biases.

 Are judges with some racial or gender bias able to set those

biases aside if they are highlighted, or does debate amplify bias? Do

debates about statistical or probabilistic questions make it too easy to

 lie with statistics?

9. **Realistic tasks (ideally):**

 If possible, we would like to try debate with interesting, real

world tasks that reflect the types of questions we would like to apply

AI to in the future, including science, mathematics, ethics, etc.

 It may not be possible to meet all of these criteria with a single

experiment. Several of the criteria are in tension: (1) and (2) are

essentially "not too hard" and "not too easy", and any restriction on

the types of questions may make it difficult to find large numbers of

questions, judges, or debaters. Realistic tasks are much harder than

synthetic tasks, which easily fulfill many of the criteria as discussed

below. Thus, we may need to begin with synthetic tasks and move up

towards realistic tasks over time.

 We turn next to a few examples of experiments to see how many

criteria we can meet simultaneously.

### Synthetic experiments: single pixel image debate

 We choose an image of a cat or dog, and show the image to the two

debaters but not the judge. One debater is honest and argues for the

true contents of the image; the other debater lies. The debaters can

talk to the judge and illustrate their points by drawing rectangles on

the image, but the judge sees only the rectangles. At the end of the

debate, each debater is allowed to reveal a single pixel to the judge,

which is the only part of the debate which cannot be a lie.

[3](#dog-debate)

 An example debate with two human debaters and a human judge.

Only the debaters can see the image. Red is arguing that the image is a

 In informal play, the honest debater wins most of the time.

Although the judge only gets to check a single pixel, the honest player

can try to pin down the liar towards disagreements on smaller and

smaller regions of the image until one pixel is enough to reveal the

lie. However, it is hard to know whether this is a real result or an

artifact that would not carry across to training an AI. There are many

ways to ruin the game, such as the judge asking each player to spend 15

seconds naming as many details about the image as possible. Worse, this

 trick ruins the game in favor of the honest player, who can easily name

 many true details while a human liar may have difficulty lying fluently

 at speed. Therefore, we are wary of trying to perform a full version

of this experiment without more careful experimental design building on

the experimental knowledge base of existing social science research.

 As a synthetic experiment, single pixel debate satisfies many of our

 experimental criteria for all human debate. The true answer is obvious

 to the debaters, and they know more than the judge since only the

debaters see the image. Anything is plausible to the judge who sees

only one or two pixels over the course of the debate. Since the

argument bottoms out at single pixels, the "full argument" would be a

huge list of pixels and how they relate, forcing the debaters to pin

each other down and focus in on disagreements. The single pixels

constitute the checkable facts, and we have an endless supply of

questions in the form of images chosen at random. Less fortunately, it

is easy for the judge to force "tells" which reveal who is lying, the

task has no obvious relation to biases, and is quite unrealistic.

### Realistic experiments: domain expert debate

 For a more interesting task, we can find two debaters who are

experts in a domain, pick a question in their area of expertise, and use

 a layperson as the judge. The debaters could be experts in some area

of science, law, or ethics, but "domain expertise" could also mean

knowledge about hobbies, local customs, sports, or any other subject the

 judge does not know. We again choose one of the debaters to lie and

one to tell the truth. To guarantee a source of truth, we can choose a

question with an agreed upon answer, either between the two debaters or

more broadly in their field. This is only approximate truth, but is

good enough for informative experiments. We also specify what facts the

 judge can access: a debate about physics might allow the debaters to

quote a sentence or paragraph from Wikipedia, perhaps with restrictions

on what pages are allowed.

 Expert debate satisfies most of our desiderata, and it is likely

possible to target specific biases (such as race or gender bias) by

picking domain areas that overlap with these biases. It may be quite

difficult or expensive to find suitable debaters, but this may be

solvable either by throwing resources at the problem (ML is a well

funded field), enlarging the kinds of domain expertise considered

(soccer, football, cricket), or by making the experiments interesting

enough that volunteers are available. However, even if domain experts

can be found, there is no guarantee that they will be experts in debate

viewed as a game. With the possible exception of law, politics, or

philosophy, domain experts

may not be trained to construct intentionally misleading but self

consistent narratives: they may be experts only in trying to tell the

truth.

 We've tried a few informal expert debates using theoretical computer

 science questions, and the main lesson is that the structure of the

debate matters a great deal. The debaters were allowed to point to a

small snippet of a mathematical definition on Wikipedia, but not to any

page that directly answered the question. To reduce tells, we first

tried to write a full debate transcript with only minimal interaction

with a layperson, then showed the completed transcript to several more

laypeople judges. Unfortunately, even the layperson present when the

debate was conducted picked the lying debater as honest, due to a

misunderstanding of the question (which was whether the complexity

classes PPP and BPPBPPBPP

 are probably equal). As a result, throughout the debate the honest

debater did not understand what the judge was thinking, and failed to

correct an easy but important misunderstanding. We fixed this in a

second debate by letting a judge ask questions throughout, but still

showing the completed transcript to a second set of judges to reduce

tells. See [the appendix](#quantum) for the transcript of this second debate.

### Other tasks: bias tests, probability puzzles, etc.

 Synthetic image debates and expert debates are just two examples of

possible tasks. More thought will be required to find tasks that

satisfy all our criteria, and these criteria will change as experiments

progress. Pulling from existing social science research will be useful,

 as there are many cognitive tasks with existing research results. If

we can map these tasks to debate, we can compare debate directly against

 baselines in psychology and other fields.

 For example, Bertrand and Mullainathan sent around 5000 resumes in

response to real employment ads, randomizing the resumes between White

and African American sounding names.

 With otherwise identical resumes, the choice of name significantly

changed the probability of a response. This experiment corresponds to

the direct question "Should we call back given this resume?" What if we

 introduce a few steps of debate? An argument against a candidate based

 on name or implicit inferences from that name might come across as

obviously racist, and convince at least some judges away from

discrimination. Unfortunately, such an experiment would necessarily

differ from Bertrand et al.'s original, where employers did not realize

they were part of an experiment. Note that this experiment works even

though the source of truth is partial: we do not know whether a

particular resume should be hired or not, but most would agree that the

answer should not depend on the candidate's name.

 For biases affecting probabilistic reasoning and decision making,

there is a long literature exploring how people decide between gambles

 For example, Erev et al. constructed an 11-dimensional space of gambles

 sufficient to reproduce 14 known cognitive biases, from which new

instances can be algorithmically generated.

 Would debates about gambles reduce cognitive biases? One difficulty

here is that simple gambles might fail the "definitive argument longer

than debate limit" criteria if an expected utility calculation is

sufficient to prove the answer, making it difficult for a lying debater

to meaningfully compete.

 Interestingly, Chen et al. used a similar setup to human+human+human

 debate to improve the quality of human data collected in a synthetic

"Relation Extraction" task.

People were first asked for direct answers, then pairs of people who

disagreed were asked to discuss and possibly update their answers. Here

 the debaters and judges are the same, but the overall goal of

extracting higher quality information from humans is shared with debate.

Questions social science can help us answer

-------------------------------------------

 We've laid out the general program for learning AI goals by asking

humans questions, and discussed how to use debate to strengthen what we

can learn by targeting the reasoning behind conclusions. Whether we use

 direct questions or something like debate, any intervention that gives

us higher quality answers is more likely to produce aligned AI. The

quality of those answers depends on the human judges, and social science

 research can help to measure answer quality and improve it. Let's go

into more detail about what types of questions we want to answer, and

what we hope to do with that information. Although we will frame these

questions as they apply to debate, most of them apply to any other

method which learns goals from humans.

1. **How skilled are people as judges by default?**

 If we ran debate using a person chosen at random as the judge, and

gave them no training, would the result be aligned? A person picked at

random might be vulnerable to convincing fallacious reasoning,

 leading AI to employ such reasoning. Note that the debaters are not

chosen at random: once the judge is fixed, we care about debaters who

either learn to help the judge (in the good case) or to exploit the

judge's weaknesses (in the bad case).

2. **Can we distinguish good judges from bad judges?**

 People likely differ in the ability to judge debates. There are

many filters we could use to identify good judges: comparing their

verdicts to those of other judges, to people given more time to think,

or to known expert judgmentNote that

domain expertise may be quite different from what makes a good judge of

 Ideally we would like filters that do not require an independent

source of truth, though at experiment time we will need a source of

truth to know whether a filter works. It is not obvious a priori that

good filters exist, and any filter would need careful scrutiny to ensure

 it does not introduce bias into our choice of judges.

3. **Does judge ability generalize across domains?**

 If judge ability in one domain fails to transfer to other domains,

we will have low confidence that it transfers to new questions and

arguments arising from highly capable AI debaters. This generalization

is necessary to trust debate as a method for alignment, especially once

we move to questions where no independent source of truth is available.

 We emphasize that judge ability is not the same as knowledge: there is

4. **Can we train people to be better judges?**

5. **What questions are people better at answering?**

 If we know that humans are bad at answering certain types of

questions, we can switch to reliable formulations. For example,

phrasing questions in frequentist terms may reduce known cognitive

biases.

 Graham et al. argue that different political views follow from

different weights placed on fundamental moral considerations, and

similar analysis could help understand where we can expect moral

disagreements to persist after reflective equilibrium.

 In cases where reliable answers are unavailable, we need to ensure

that trained models know their own limits, and express uncertainty or

disagreement as required.

6. **Are there ways to restrict debate to make it easier to judge?**

 People might be better at judging debates formulated in terms of

calm, factual statements, and worse at judging debates designed to

trigger strong emotions. Or, counterintuitively, it could be the other

way around. If we know which styles of debates that people are

 better at judging, we may be able to restrict AI debaters to these styles.

7. **How can people work together to improve quality?**

 If individuals are insufficient judges, are teams of judges better?

 Majority vote is the simplest option, but perhaps several people

talking through an answer together is stronger, either actively or after

 the fact through peer review. Condorcet's jury theorem implies that

majority votes can amplify weakly good judgments to strong judgments (or

 Given our lack of experience outside of ML, we are not able to

precisely articulate all of the different experiments we need. The only

 way to fix this is to talk to more people with different backgrounds

and expertise. We have started this process, but are eager for more

conversations with social scientists about what experiments could be

run, and encourage other AI safety efforts to engage similarly.

Reasons for optimism

--------------------

 We believe that understanding how humans interact with long-term AI

alignment is difficult but possible. However, this would be a new

research area, and we want to be upfront about the uncertainties

involved. In this section and the next, we discuss some reasons for

optimism and pessimism about whether this research will succeed. We

focus on issues specific to human uncertainty and associated social

science research; for similar discussion on ML uncertainty in the case

of debate we refer to our previous work.

### Engineering vs. science

 Most social science seeks to understand humans "in the wild":

results that generalize to people going about their everyday lives.

With limited control over these lives, differences between laboratory

and real life are bad from the scientific perspective. In contrast, AI

alignment seeks to extract the best version of what humans want: our

goal is engineering rather than science, and we have more freedom to

intervene. If judges in debate need training to perform well, we can

provide that training. If some people still do not provide good data,

we can remove them from experiments (as long as this filter does not

create too much bias). This freedom to intervene means that some of the

 difficulty in understanding and improving human reasoning may not

apply. However, science is still required: once our interventions are

in place, we need to correctly know whether our methods work. Since our

 experiments will be an imperfect model of the final goal, careful

design will be necessary to minimize this mismatch, just as is required

by existing social science.

### We don't need to answer all questions

 Our most powerful intervention is to give up: to recognize that we

are unable to answer some types of questions, and instead prevent AI

systems from pretending to answer. Humans might be good judges on some

topics but not others, or with some types of reasoning but not others;

if we discover that we can adjust our goals appropriately. Giving up on

 some types of questions is achievable either on the ML side, using

careful uncertainty modeling to know when we do not know, or on the

human side by training judges to understand their own areas of

uncertainty. Although we will attempt to formulate ML systems that

automatically detect areas of uncertainty, any information we can gain

on the social science side about human uncertainty can be used both to

augment ML uncertainty modeling and to test whether ML uncertainty

modeling works.

### Relative accuracy may be enough

 Say we have a variety of different ways to structure debate with

humans. Ideally, we would like to achieve results of the form "debate

structure AAA

 is truth-seeking with 90% confidence". Unfortunately, we may be

unconfident that an absolute result of this form will generalize to

advanced AI systems: it may hold for an experiment with simple tasks but

 break down later on. However, even if we can't achieve such absolute

results, we can still hope for relative results of the form "debate

### We don't need to pin down the best alignment scheme

 As the AI safety field progresses to increasingly advanced ML

systems, we expect research on the ML side and the human side to merge.

 Starting social science experiments prior to this merging will give the

 field a head start, but we can also take advantage of the expected

merging to make our goals easier. If social science research narrows

the design space of human-friendly AI alignment algorithms but does not

produce a single best scheme, we can test the smaller design space once

the machines are ready.

### A negative result would be important!

 If we test an AI alignment scheme from the social science

perspective and it fails, we've learned valuable information. There are

 a variety of proposed alignment schemes, and learning which don't work

early gives us more time to switch to others, or to intervene on a

policy level to slow down dangerous development. In fact, given our

belief that AI alignment is harder for more advanced agents, a negative

result might be easier to believe and thus more valuable that a less

trustworthy positive result.

Reasons to worry

----------------

 We turn next to reasons social science experiments about AI

alignment might fail to produce useful results. We emphasize that

useful results might be both positive and negative, so these are not

reasons why alignment schemes might fail. Our primary worry is one

sided, that experiments would say an alignment scheme works when in fact

 it does not, though errors in the other direction are also undesirable.

### Our desiderata are conflicting

 As mentioned before, some of our criteria when picking experimental

tasks are in conflict. We want tasks that are sufficiently interesting

(not too easy), with a source of verifiable ground truth, are not too

hard, etc. "Not too easy" and "not too hard" are in obvious conflict,

but there are other more subtle difficulties. Domain experts with the

knowledge to debate interesting tasks may not be the same people capable

 of lying effectively, and both restrictions make it hard to gather

large volumes of data. Lying effectively is required for a meaningful

experiment, since a trained AI may have no trouble lying unless lying is

 a poor strategy to win debates. Experiments to test whether ethical

biases interfere with judgment may make it more difficult to find tasks

with reliable ground truth, especially on subjects with significant

disagreement across people. The natural way out is to use many

different experiments to cover different aspects of our uncertainty, but

 this would take more time and might fail to notice interactions between

 desiderata.

### We want to measure judge quality given optimal debaters

 For debate, our end goal is to understand if the judge is capable of

 determining who is telling the truth. However, we specifically care

whether the judge performs well given that the debaters are performing

well. Thus our experiments have an inner/outer optimization structure:

we first train the debaters to debate well, then measure how well the

judges perform. This increases time and cost: if we change the task, we

 may need to find new debaters or retrain existing debaters. Worse, the

 human debaters may be bad at performing the task, either out of

inclination or ability. Poor performance is particularly bad if it is

one sided and applies only to lying: a debater might be worse at lying

out of inclination or lack of practice, and thus a win for the honest

debater might be misleading.

### ML algorithms will change

 It is unclear when or if ML systems will reach various levels of

capability, and the algorithms used to train them will evolve over time.

 The AI alignment algorithms of the future may be similar to the

proposed algorithms of today, or they may be very different. However,

we believe that knowledge gained on the human side will partially

transfer: results about debate will teach us about how to gather data

from humans even if debate is superseded. The algorithms may change;

humans will not.

### Need strong out-of-domain generalization

 Regardless of how carefully designed our experiments are,

human+human+human debate will not be a perfect match to AI+AI+human

debate. We are seeking research results that generalize to the setting

where we replace the human debaters (or similar) with AIs of the future,

 which is a hard ask. This problem is fundamental: we do not have the

advanced AI systems of the future to play with, and want to learn about

human uncertainty starting now.

### Lack of philosophical clarity

 Any AI alignment scheme will be both an algorithm for training ML

systems and a proposed definition of what it means to be aligned.

However, we do not expect humans to conform to any philosophically

consistent notion of values, and concepts like reflective equilibrium

must be treated with caution in case they break down when applied to

real human judgement. Fortunately, algorithms like debate need not

presuppose philosophical consistency: a back and forth conversation to

convince a human judge makes sense even if the human is leaning on

heuristics, intuition, and emotion. It is not obvious that debate works

 in this messy setting, but there is hope if we take advantage of

inaction bias, uncertainty modeling, and other escape hatches. We

believe lack of philosophical clarity is an argument for investing in

social science research: if humans are not simple, we must engage with

their complexity.

The scale of the challenge

--------------------------

 Long-term AI safety is particularly important if we develop

artificial general intelligence (AGI), which the OpenAI Charter defines

as highly autonomous systems that outperform humans at most economically

 valuable work. If we want to

train an AGI with reward learning from humans, it is unclear how many

samples will be required to align it. As much as possible, we can try

to replace human samples with knowledge about the world gained by

reading language, the internet, and other sources of information. But

it is likely that a fairly large number of samples from people will

still be required. Since more samples means less noise and more safety,

 if we are uncertain about how many samples we need then we will want a

lot of samples.

 A lot of samples would mean recruiting a lot of people. We cannot

rule out needing to involve thousands to tens of thousands of people for

 millions to tens of millions of short interactions: answering

questions, judging debates, etc. We may need to train these people to

be better judges, arrange for peers to judge each other's reasoning,

determine who is doing better at judging and give them more weight or a

more supervisory role, and so on. Many researchers would be required on

 the social science side to extract the highest quality information from

 the judges.

 A task of this scale would be a large interdisciplinary project,

requiring close collaborations in which people of different backgrounds

fill in each other's missing knowledge. If machine learning reaches

this scale, it is important to get a head start on the collaborations

soon.

Conclusion: how you can help

----------------------------

 We have argued that the AI safety community needs social scientists

to tackle a major source of uncertainty about AI alignment algorithms:

will humans give good answers to questions? This uncertainty is

difficult to tackle with conventional machine learning experiments,

since machine learning is primitive. We are still in the early days of

performance on natural language and other tasks, and problems with human

 reward learning may only show up on tasks we cannot yet tackle.

 Our proposed solution is to replace machine learning with people, at

 least until ML systems can participate in the complexity of debates we

are interested in. If we want to understand a game played with ML and

human participants, we replace the ML participants with people, and see

how the all human game plays out. For the specific example of debate,

we start with debates with two ML debaters and a human judge, then

switch to two human debaters and a human judge. The result is a pure

human experiment, motivated by machine learning but available to anyone

with a solid background in experimental social science. It won't be an

easy experiment, which is all the more reason to start soon.

 If you are a social scientist interested in these questions, please

talk to AI safety researchers! We are interested in both conversation

and close collaboration. There are many institutions engaged with

 If you are a machine learning researcher interested in or already

working on safety, please think about how alignment algorithms will work

 once we advance to tasks beyond the abilities of current machine

learning. If your preferred alignment scheme uses humans in an

important way, can you simulate the future by replacing some or all ML

components with people? If you can imagine these experiments but don't

feel you have the expertise to perform them, find someone who does.

Deep reinforcement learning from human preferences

Judgment under uncertainty: heuristics and biases

Intergroup bias

AI safety via debate

Supervising strong learners by amplifying weak experts

Reward learning from human preferences and demonstrations in Atari

AI safety gridworlds

An empirical methodology for writing user-friendly natural language computer applications

Factored Cognition

Learning the Preferences of Ignorant, Inconsistent Agents

Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations

Computational Social Science: Towards a collaborative future

Mirror Mirror: Reflections on Quantitative Fairness

Moral Anti-Realism

Gender shades: Intersectional accuracy disparities in commercial gender classification

Moral dumbfounding: When intuition finds no reason

Batch active preference-based learning of reward functions

Learning to understand goal specifications by modelling reward

Improving language understanding by generative pre-training

Thinking, fast and slow

Deep Blue

Mastering chess and shogi by self-play with a general reinforcement learning algorithm

Deviant or Wrong? The Effects of Norm Information on the Efficacy of Punishment

The weirdest people in the world?

Fact, fiction, and forecast

A theory of justice

Looking for a psychology for the inner rational agent

How (and where) does moral judgment work?

Scalable agent alignment via reward modeling: a research direction

OpenAI Five

The Most Human Human: What Talking with Computers Teaches Us About What It Means to Be Alive

How to overcome prejudice

The nature and origins of misperceptions: Understanding false and unsupported beliefs about politics

Persuasion, influence, and value: Perspectives from communication and social neuroscience

Identifying and cultivating superforecasters as a method of improving probabilistic predictions

Superforecasting: The art and science of prediction

Cooperative inverse reinforcement learning

Inverse reward design

The art of being right

Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination

Prospect theory: An analysis of decisions under risk

Advances in prospect theory: Cumulative representation of uncertainty

From anomalies to forecasts: Toward a descriptive model of decisions under risk, under ambiguity, and from experience

Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing

The rationality of informal argumentation: A Bayesian approach to reasoning fallacies

Rationality in medical decision making: a review of the literature on doctors’ decision-making biases

Expert political judgment: How good is it? How can we know?

Two approaches to the study of experts’ characteristics

Debiasing

An evaluation of argument mapping as a method of enhancing critical thinking performance in e-learning environments

Forecasting tournaments: Tools for increasing transparency and improving the quality of debate

How to make cognitive illusions disappear: Beyond "heuristics and biases"

Liberals and conservatives rely on different sets of moral foundations

Negative emotions can attenuate the influence of beliefs on logical reasoning

Epistemic democracy: Generalizing the Condorcet jury theorem

Aggregating sets of judgments: An impossibility result

The Delphi technique as a forecasting tool: issues and analysis

OpenAI Charter


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Two Examples of Useful, Non-Robust Features

Abstract: This article is part of a discussion of the Ilyas et al. paper         “Adversarial examples are not bugs, they are features”.         You can learn more in the                      main discussion article         .

 #rebuttal,

 .comment-info {

 background-color: hsl(54, 78%, 96%);

 border-left: solid hsl(54, 33%, 67%) 1px;

 padding: 1em;

 color: hsla(0, 0%, 0%, 0.67);

 }

 #header-info {

 margin-top: 0;

 margin-bottom: 1.5rem;

 display: grid;

 grid-template-columns: 65px max-content 1fr;

 grid-template-areas:

 "icon explanation explanation"

 "icon back comment";

 grid-column-gap: 1.5em;

 }

 #header-info .icon-multiple-pages {

 grid-area: icon;

 padding: 0.5em;

 content: url(images/multiple-pages.svg);

 }

 #header-info .explanation {

 grid-area: explanation;

 font-size: 85%;

 }

 #header-info .back {

 grid-area: back;

 }

 #header-info .back::before {

 content: "←";

 margin-right: 0.5em;

 }

 #header-info .comment {

 grid-area: comment;

 scroll-behavior: smooth;

 }

 #header-info .comment::before {

 content: "↓";

 margin-right: 0.5em;

 }

 #header-info a.back,

 #header-info a.comment {

 font-size: 80%;

 font-weight: 600;

 border-bottom: none;

 text-transform: uppercase;

 color: #2e6db7;

 display: block;

 margin-top: 0.25em;

 letter-spacing: 0.25px;

 }

 This article is part of a discussion of the Ilyas et al. paper

 *"Adversarial examples are not bugs, they are features".*

 You can learn more in the

 [main discussion article](https://distill.pub/2019/advex-bugs-discussion/) .

[Other Comments](https://distill.pub/2019/advex-bugs-discussion/#commentaries)

[Comment by Ilyas et al.](#rebuttal)

 Ilyas et al.  define a *feature* as a function fff that

 label. But in the presence of an adversary Ilyas et al.  argues

 the metric that truly matters is a feature's *robust usefulness*,

E[inf∥δ∥≤ϵyf(x+δ)],

 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)
ight],

 E[∥δ∥≤ϵinfyf(x+δ)],

 its correlation with the label while under attack. Ilyas et al.

 like?

### Non-Robust Features in Linear Models

 nonlinear models encountered in deep learning. As Ilyas et al

 to linear features of the form:

 	ext{and} \quad \mathbf{E}[x] = 0.

 f(x)=∥a∥ΣaTxwhereΣ=E[xxT]andE[x]=0.

 The robust usefulness of a linear feature admits an elegant decomposition

  This

 egin{aligned}

\mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)
ight] &

=\mathbf{E}\left[yf(x)+\inf\_{\|\delta\|\leq\epsilon}yf(\delta)
ight]\

 &

 &

 \end{aligned}

 into two terms:

 .undomargin {

 position: relative;

 left: -1em;

 top: 0.2em;

 }

E[inf∥δ∥≤ϵyf(x+δ)]

 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)
ight]

 E[∥δ∥≤ϵinfyf(x+δ)]

===

E[yf(x)]\mathop{\mathbf{E}[yf(x)]}E[yf(x)]

−-−

ϵ∥a∥∗∥a∥Σ\epsilonrac{\|a\|\_{\*}}{\|a\|\_{\Sigma}}ϵ∥a∥Σ∥a∥∗

 The robust usefulness of a feature

 the correlation of the feature with the label

 the feature's non-robustness

 dimensional plot.

 Subject to an

 L\_2

 adversery, observe that high frequency features are both less useful and

 less robust.

  Useful Non-Useful A  B  C  D  E  F

 Pareto frontier of points in the non-robustness and usefulness space.

 positive label.

 \log \left( rac{\|a\_i\|\_\Sigma}{\|a\_i\|}
ight) =

 \log(\lambda\_i) Feature's log robustness. When

 a\_i's

 are the

 i^{th}

 eigenvalues of

 \Sigma

 , the robustness reduces to the

 i^{th}

 singular value of

 \lambda\_i   ABCDEFf-12-11-10-9-8-7-6-5-4-3-4-3-2-10

 We demonstrate two constructions:

**Ensembles** The work of Tsipras et al

 suggests a collection of non-robust and non-useful features, if

sufficiently uncorrelated, can be ensembled into a single useful,

non-robust useful feature f.

 process is illustrated above numerically. We choose a set of non-robust

 features by excluding all features above a threshold, and naively

ensembling them according to:

 (1-lpha) \cdot a\_{	ext{non-robust}} + lpha \cdot a\_{	ext{robust}},

 It is surprising, thus, that the experiments of Madry et al.

 To cite Ilyas et al.'s response, please cite their

**Response Summary**: The construction of explicit non-robust features is

 the useful non-robust features detected by our experiments. We also agree that

 non-robust features arising as "distractors" is indeed not precluded by our

 theoretical framework, even if it is precluded by our experiments.

 This simple theoretical framework sufficed for reasoning about and

 predicting the outcomes of our experiments

 We also presented a theoretical setting where we can

 analyze things fully rigorously in Section 4 of our paper..

 However, this comment rightly identifies finding a more comprehensive

 definition of feature as an important future research direction.

**Response**: These experiments (visualizing the robustness and

 corroborate the existence of useful, non-robust features and make progress

 towards visualizing what these non-robust features actually look like.

We also appreciate the point made by the provided construction of non-robust

 features (as defined in our theoretical framework) that are combinations of

 useful+robust and useless+non-robust features. Our theoretical framework indeed

 enables such a scenario, even if — as the commenter already notes — our

 framework technically captures.) Specifically, in such a scenario, during the

 construction of the D^det\widehat{\mathcal{D}}\_{det}D

det dataset, only the non-robust

 and useless term of the feature would be flipped. Thus, a classifier trained on

 such a dataset would associate the predictive robust feature with the

 *wrong* label and would thus not generalize on the test set. In contrast,

det

 do generalize.

Overall, our focus while developing our theoretical framework was on

 the comment points out, putting forth a theoretical framework that captures

 non-robust features in a very precise way is an important future research

 direction in itself.


Adversarial Reprogramming of Neural Cellular Automata

Ettore Randazzo
Alexander Mordvintsev
Eyvind Niklasson
Michael Levin

Abstract: This article is part of the     Differentiable Self-organizing Systems Thread,     an experimental format collecting invited short articles delving into     differentiable self-organizing systems, interspersed with critical     commentary from several experts in adjacent fields.

### Contents

[Adversarial MNIST CAs](#adversarial-mnist-cas) |

[Perturbing the states of Growing CAs](#perturbing-the-states-of-growing-cas) |

[Related Work](#related-work)

[Discussion](#discussion)

 This article is part of the

 an experimental format collecting invited short articles delving into

 differentiable self-organizing systems, interspersed with critical

 commentary from several experts in adjacent fields.

[Self-Organising Textures](https://distill.pub/selforg/2021/textures/)

In a complex system, whether biological, technological, or social,

how can we discover signaling events that will alter system-level

behavior in desired ways? Even when the rules governing the individual

components of these complex systems are known, the inverse problem -

going from desired behaviour to system design - is at the heart of many

barriers for the advance of biomedicine, robotics, and other fields of

importance to society.

Biology, specifically, is transitioning from a focus on mechanism

(what is required for the system to work) to a focus on information

(what algorithm is sufficient to implement adaptive behavior). Advances

in machine learning represent an exciting and largely untapped source of

 inspiration and tooling to assist the biological sciences. Growing

Neural Cellular Automata  and Self-classifying MNIST Digits

 introduced the Neural Cellular Automata (Neural CA) model and

demonstrated how tasks requiring self-organisation, such as pattern

growth and self-classification of digits, can be trained in an

end-to-end, differentiable fashion. The resulting models were robust to

various kinds of perturbations: the growing CA expressed regenerative

capabilities when damaged; the MNIST CA were responsive to changes in

the underlying digits, triggering reclassification whenever necessary.

These computational frameworks represent quantitative models with which

to understand important biological phenomena, such as scaling of single

cell behavior rules into reliable organ-level anatomies. The latter is a

 kind of anatomical homeostasis, achieved by feedback loops that must

recognize deviations from a correct target morphology and progressively

reduce anatomical error.

In this work, we *train adversaries* whose goal is to reprogram

CA into doing something other than what they were trained to do. In

order to understand what kinds of lower-level signals alter

system-level behavior of our CA, it is important to understand how these

 CA are constructed and where local versus global information resides.

The system-level behavior of Neural CA is affected by:

* **Individual cell states.** States store

information which is used for both diversification among cell behaviours

 and for communication with neighbouring cells.

* **The model parameters.** These describe the

input/output behavior of a cell and are shared by every cell of the same

 family. The model parameters can be seen as *the way the system works*.

* **The perceptive field.** This is how cells perceive

their environment. In Neural CA, we always restrict the perceptive field

 to be the eight nearest neighbors and the cell itself. The way cells

are perceived by each other is different between the Growing CA and

MNIST CA. The Growing CA perceptive field is a set of weights fixed both

 during training and inference, while the MNIST CA perceptive field is

learned as part of the model parameters.

We will explore two kinds of adversarial attacks: 1) injecting a few

adversarial cells into an existing grid running a pretrained model; and

2) perturbing the global state of all cells on a grid.

For the first type of adversarial attacks we train a new CA model

that, when placed in an environment running one of the original models

described in the previous articles, is able to hijack the behavior of

the collective mix of adversarial and non-adversarial CA. This is an

example of injecting CA with differing *model parameters* into the

system. In biology, numerous forms of hijacking are known, including

 Especially fascinating are the many cases of non-cell-autonomous

signaling developmental biology and cancer, showing that some cell

behaviors can significantly alter host properties both locally and at

long range. For example, bioelectrically-abnormal cells can trigger

metastatic conversion in an otherwise normal body (with no genetic

 All of these phenomena underlie the importance of understanding how

cell groups make collective decisions, and how those tissue-level

decisions can be subverted by the activity of a small number of cells.

It is essential to develop quantitative models of such dynamics, in

order to drive meaningful progress in regenerative medicine that

controls system-level outcomes top-down, where cell- or molecular-level

micromanagement is infeasible .

 We apply a global state perturbation to all living cells. This can be

seen as inhibiting or enhancing combinations of state values, in turn

hijacking proper communications among cells and within the cell's own

states. Models like this represent not only ways of thinking about

adversarial relationships in nature (such as parasitism and evolutionary

 arms races of genetic and physiological mechanisms), but also a roadmap

 for the development of regenerative medicine strategies.

Next-generation biomedicine will need computational tools for inferring

minimal, least-effort interventions that can be applied to biological

systems to predictively change their large-scale anatomical and

behavioral properties.

Recall how the Self-classifying MNIST digits task consisted of

placing CA cells on a plane forming the shape of an MNIST digit. The

cells then had to communicate among themselves in order to come to a

complete consensus as to which digit they formed.

 (a) Local information neighbourhood - each cell can only observe itself

 and its neighbors' states, or the absence of neighbours.

Your browser does not support the video tag.

 and freeze its parameters. We then train a new CA whose model

architecture is identical to the frozen model but is randomly

initialized. The training regime also closely approximates that of

self-classifying MNIST digits CA. There are three important differences:

* For each batch and each pixel, the CA is randomly chosen to be

either the pretrained model or the new adversarial one. The adversarial

CA is used 10% of the time, and the pre-trained, frozen, model the rest

of the time.

The adversarial attack as defined here only modifies a small

percentage of the overall system, but the goal is to propagate signals

that affect all the living cells. Therefore, these adversaries have to

somehow learn to communicate deceiving information that causes wrong

classifications in their neighbours and further cascades in the

propagation of deceiving information by 'unaware' cells. The unaware

cells' parameters cannot be changed so the only means of attack by the

adversaries is to cause a change in the cells' states. Cells' states are

 responsible for communication and diversification.

The task is remarkably simple to optimize, reaching convergence in as

 little as 2000 training steps (as opposed to the two orders of

magnitude more steps needed to construct the original MNIST CA). By

visualising what happens when we remove the adversaries, we observe that

 the adversaries must be constantly communicating with their

non-adversarial neighbours to keep them convinced of the malicious

classification. While some digits don't recover after the removal of

adversaries, most of them self-correct to the right classification.

Below we show examples where we introduce the adversaries at 200 steps

and remove them after a further 200 steps.

Your browser does not support the video tag.

We introduce the adversaries (red pixels) after 200 steps and remove

them after 200 more steps. Most digits recover, but not all. We

highlight mistakes in classification with a red background.

While we trained the adversaries with a 10-to-90% split of

adversarial vs. non-adversarial cells, we observe that often

significantly fewer adversaries are needed to succeed in the deception.

Below we evaluate the experiment with just one percent of cells being

adversaries.

Your browser does not support the video tag.

Adversaries constituting up 1% of the cell collective (red pixels). We

highlight mistakes in classification with a red background.

We created a demo playground where the reader can draw digits and

place adversaries with surgical precision. We encourage the reader to

play with the demo to get a sense of how easily non-adversarial cells

are swayed towards the wrong classification.

The natural follow up question is whether these adversarial attacks

work on Growing CA, too. The Growing CA goal is to be able to grow a

complex image from a single cell, and having its result be persistent

over time and robust to perturbations. In this article, we focus on the

lizard pattern model from Growing CA.

Your browser does not support the video tag.

The target CA to hijack.

The goal is to have some adversarial cells change the global

configuration of all the cells. We choose two new targets we would like

the adversarial cells to try and morph the lizard into: a tailless

lizard and a red lizard.

The desired mutations we want to apply.

These targets have different properties:

* **Red lizard:** converting a lizard from green to

red would show a global change in the behaviour of the cell collective.

This behavior is not present in the dynamics observed by the original

model. The adversaries are thus tasked with fooling other cells into

doing things they have never done before (create the lizard shape as

before, but now colored in red).

* **Tailless lizard:** having a severed tail is a more

localized change that only requires some cells to be fooled into

behaving in the wrong way: the cells at the base of the tail need to be

convinced they constitute the edge or silhouette of the lizard, instead

of proceeding to grow a tail as before.

We first train adversaries for the tailless target with a 10% chance

for any given cell to be an adversary. We prohibit cells to be

adversaries if they are outside the target pattern; i.e. the tail

contains no adversaries.

Your browser does not support the video tag.

10% of the cells are adversarial.

The video above shows six different instances of the same model with

differing stochastic placement of the adversaries. The results vary

considerably: sometimes the adversaries succeed in removing the tail,

sometimes the tail is only shrunk but not completely removed, and other

times the pattern becomes unstable. Training these adversaries required

many more gradient steps to achieve convergence, and the pattern

converged to is qualitatively worse than what was achieved for the

adversarial MNIST CA experiment.

The red lizard pattern fares even worse. Using only 10% adversarial

cells results in a complete failure: the original cells are unaffected

by the adversaries. Some readers may wonder whether the original

pretrained CA has the requisite skill, or 'subroutine' of producing a

red output at all, since there are no red regions in the original

target, and may suspect this was an impossible task to begin with.

Therefore, we increased the proportion of adversarial cells until we

managed to find a successful adversarial CA, if any were possible.

Your browser does not support the video tag.

In the video above we can see how, at least in the first stages of

morphogenesis, 60% of adversaries are capable of coloring the lizard

 where we hide the adversarial cells and show only the original cells.

There, we see how a handful of original cells are colored in red. This

is proof that the adversaries successfully managed to steer neighboring

cells to color themselves red, where needed.

However, the model is very unstable when iterated for periods of time

 longer than seen during training. Moreover, the learned adversarial

attack is dependent on a majority of cells being adversaries. For

instance, when using fewer adversaries on the order of 20-30%, the

configuration is unstable.

In comparison to the results of the previous experiment, the Growing

CA model shows a greater resistance to adversarial perturbation than

those of the MNIST CA. A notable difference between the two models is

that the MNIST CA cells have to always be ready and able to change an

opinion (a classification) based on information propagated through

several neighbors. This is a necessary requirement for that model

because at any time the underlying digit may change, but most of the

cells would not observe any change in their neighbors' placements. For

instance, imagine the case of a one turning into a seven where the lower

 stroke of each overlap perfectly. From the point of view of the cells

in the lower stroke of the digit, there is no change, yet the digit

formed is now a seven. We therefore hypothesise MNIST CA are more

reliant and 'trusting' of continuous long-distance communication than

Growing CA, where cells never have to reconfigure themselves to generate

 something different to before.

We suspect that more general-purpose Growing CA that have learned a

variety of target patterns during training are more likely to be

susceptible to adversarial attacks.

We observed that it is hard to fool Growing CA into changing their

morphology by placing adversarial cells inside the cell collective.

These adversaries had to devise complex local behaviors that would cause

 the non-adversarial cells nearby, and ultimately globally throughout

the image, to change their overall morphology.

In this section, we explore an alternative approach: perturbing the

global state of all cells without changing the model parameters of any

cell.

As before, we base our experiments on the Growing CA model trained to

 produce a lizard. Every cell of a Growing CA has an internal state

vector with 16 elements. Some of them are phenotypical elements (the

RGBA states) and the remaining 12 serve arbitrary purposes, used for

storing and communicating information. We can perturb the states of

these cells to hijack the overall system in certain ways (the discovery

of such perturbation strategies is a key goal of biomedicine and

synthetic morphology). There are a variety of ways we can perform state

perturbations. We will focus on *global state perturbations*,

defined as perturbations that are applied on every living cell at every

time step (analogous to "systemic" biomedical interventions, that are

given to the whole organism (e.g., a chemical taken internally), as

opposed to highly localized delivery systems). The new goal is to

discover a certain type of global state perturbation that results in a

stable new pattern.

Diagram showing some possible stages for perturbing a lizard

 pattern. (a) We start from a seed that grows into a lizard (b) Fully

converged lizard. (c) We apply a global state perturbation at every

step. As a result, the lizard loses its tail. (d) We stop perturbing the

 state. We observe the lizard immediately grows back its tail.

We show 6 target patterns: the tailless and red lizard from the

previous experiment, plus a blue lizard and lizards with various severed

 limbs and severed head.

Mosaic of the desired mutations we want to apply.

 To give insight on why we chose this, an even simpler "state addition"

mutation (a mutation consisting only of the addition of a vector to

every state) would be insufficient because the value of the states of

our models are unbounded, and often we would want to suppress something

by setting it to zero. The latter is generally impossible with constant

state additions, as a constant addition or subtraction of a value would

generally lead to infinity, except for some fortunate cases where the

natural residual updates of the cells would cancel out with the constant

 addition at precisely state value zero. However, matrix multiplications

 have the possibility of amplifying/suppressing combinations of elements

 in the states: multiplying a state value repeatedly for a constant

value less than one can easily suppress a state value to zero. We

constrain the matrix to be symmetric for reasons that will become clear

in the following section.

* The underlying CA parameters are frozen and we only train AAA.

* We consider the set of initial image configurations to be both the

seed state and the state with a fully grown lizard (as opposed to the

Growing CA article, where initial configurations consisted of the seed

state only).

Your browser does not support the video tag.

Effect of applying the trained perturbations.

The video above shows the model successfully discovering global state

 perturbations able to change a target pattern to a desired variation.

We show what happens when we stop perturbing the states (an

out-of-training situation) at step 500 through step 1000, then

reapplying the mutation. This demonstrates the ability of our

perturbations to achieve the desired result both when starting from a

seed, and when starting from a fully grown pattern. Furthermore it

demonstrates that the original CA easily recover from these state

perturbations once it goes away. This last result is perhaps not

surprising given how robust growing CA models are in general.

Not all perturbations are equally effective. In particular, the

headless perturbation is the least successful as it results in a loss of

 other details across the whole lizard pattern such as the white

coloring on its back. We hypothesize that the best perturbation our

training regime managed to find, due to the simplicity of the

perturbation, was suppressing a "structure" that contained both the

morphology of the head and the white colouring. This may be related to

the concept of differentiation and distinction of biological organs.

Predicting what kinds of perturbations would be harder or impossible to

be done, before trying them out empirically, is still an open research

question in biology. On the other hand, a variant of this kind of

synthetic analysis might help with defining higher order structures

within biological and synthetic systems.

### Directions and compositionality of perturbations

Our choice of using a symmetric matrix for representing global state

perturbations is justified by a desire to have compositionality. Every

complex symmetric matrix AAA can be diagonalized as follows:

A=QΛQ⊺A = Q \Lambda Q^\intercalA=QΛQ⊺

where Λ\LambdaΛ is the diagonal eigenvalues matrix and QQQ

 is the unitary matrix of its eigenvectors. Another way of seeing this

is applying a change of basis transformation, scaling each component

proportional to the eigenvalues, and then changing back to the original

basis. This should also give a clearer intuition on the ease of

suppressing or amplifying combinations of states. Moreover, we can now

infer what would happen if all the eigenvalues were to be one. In that

Let us then take the tailless perturbation and see what happens as we vary kkk:

Your browser does not support the video tag.

 negative, the lizard grows a longer tail. Unfortunately, the further

away we go, the more unstable the system becomes and eventually the

lizard pattern grows in an unbounded fashion. This behaviour likely

stems from that perturbations applied on the states also affect the

homeostatic regulation of the system, making some cells die out or grow

in different ways than before, resulting in a behavior akin to "cancer"

in biological systems.

In that case,

 this could result in a stable perturbation. An intuitive understanding

of this is interpolating stable perturbations using the direction

coefficients.

In practice, however, the eigenvectors are also different, so the

results of the combination will likely be worse the more different the

respective eigenvector bases are.

Below, we interpolate the direction coefficients, while keeping their

 sum to be one, of two types of perturbations: tailless and no-leg

lizards.

Your browser does not support the video tag.

While it largely achieves what we expect, we observe some unintended

effects such as the whole pattern starting to traverse vertically in the

 grid. Similar results happen with other combinations of perturbations.

What happens if we remove the restriction of the sum of kkks

 being equal to one, and instead add both perturbations in their

entirety? We know that if the two perturbations were the same, we would

end twice as far away from the identity perturbation, and in general we

expect the variance of these perturbations to increase. Effectively,

this means going further and further away from the stable perturbations

discovered during training. We would expect more unintended effects that

 may disrupt the CA as the sum of kkks increases.

Below, we demonstrate what happens when we combine the tailless and

the no-leg lizard perturbations at their fullest. Note that when we set

Your browser does not support the video tag.

Effect of composing two perturbations.

Surprisingly, the resulting pattern is almost as desired. However, it

 also suffers from the vertical movement of the pattern observed while

interpolating kkks.

This framework can be generalized to any arbitrary number of

perturbations. Below, we have created a small playground that allows the

 reader to input their desired combinations. Empirically, we were

surprised by how many of these combinations result in the intended

Related work

------------

This work is inspired by Generative Adversarial Networks (GANs) .

 While with GANs it is typical to cotrain pairs of models, in this work

we froze the original CA and trained the adversaries only. This setup is

### Influence maximization

Adversarial cellular automata have parallels to the field of

influence maximization. Influence maximization involves determining the

optimal nodes to influence in order to maximize influence over an entire

 graph, commonly a social graph, with the property that nodes can in

turn influence their neighbours. Such models are used to model a wide

variety of real-world applications involving information spread in a

graph.

 A common setting is that each vertex in a graph has a binary state,

which will change if and only if a sufficient fraction of its

neighbours' states switch. Examples of such models are social influence

maximization (maximally spreading an idea in a network of people),

 (when small perturbations to a system bring about a larger 'phase

change'). At the time of writing this article, for instance, contagion

minimization is a model of particular interest. NCA are a graph - each

cell is a vertex and has edges to its eight neighbours, through which it

 can pass information. This graph and message structure is significantly

 more complex than the typical graph underlying much of the research in

influence maximization, because NCA cells pass vector-valued messages

and have a complex update rules for their internal states, whereas

graphs in influence maximization research typically consist of more

simple binary cells states and threshold functions on edges determining

whether a node has switched states. Many concepts from the field could

be applied and are of interest, however.

For example, in this work, we have made an assumption that our

adversaries can be positioned anywhere in a structure to achieve a

desired behaviour. A common focus of investigation in influence

maximization problems is deciding which nodes in a graph will result in

maximal influence on the graph, referred to as target set selection .

 This problem isn't always tractable, often NP-hard, and solutions

frequently involve simulations. Future work on adversarial NCA may

involve applying techniques from influence maximization in order to find

 the optimal placement of adversarial cells.

Discussion

----------

This article showed two different kinds of adversarial attacks on Neural CA.

Injections of adversarial CA in a pretrained Self-classifying MNIST

CA showed how an existing system of cells that are heavily reliant on

the passing of information among each other is easily swayed by

deceitful signaling. This problem is routinely faced by biological

systems, which face hijacking of behavioral, physiological, and

morphological regulatory mechanisms by parasites and other agents in the

 biosphere with which they compete. Future work in this field of

computer technology can benefit from research on biological

communication mechanisms to understand how cells maximize reliability

and fidelity of inter- and intra-cellular messages required to implement

 adaptive outcomes.

The adversarial injection attack was much less effective against

Growing CA and resulted in overall unstable CA. This dynamic is also of

importance to the scaling of control mechanisms (swarm robotics and

nested architectures): a key step in "multicellularity" (joining

together to form larger systems from sub-agents )

 is informational fusion, which makes it difficult to identify the

source of signals and memory engrams. An optimal architecture would need

 to balance the need for validating control messages with a possibility

of flexible merging of subunits, which wipes out metadata about the

specific source of informational signals. Likewise, the ability to

respond successfully to novel environmental challenges is an important

goal for autonomous artificial systems, which may import from biology

strategies that optimize tradeoff between maintaining a specific set of

signals and being flexible enough to establish novel signaling regimes

when needed.

The global state perturbation experiment on Growing CA shows how it

is still possible to hijack these CA towards stable out-of-training

configurations and how these kinds of attacks are somewhat composable in

 a similar way to how embedding spaces are manipulable in the natural

 We hypothesize that this is partially due to the regenerative

capabilities of the pretrained CA, and that other models may be less

capable of recovery from arbitrary perturbations.

Growing Neural Cellular Automata

Self-classifying MNIST Digits

Herpes Simplex Virus: The Hostile Guest That Takes Over Your Home

The
 role of gut microbiota (commensal bacteria) and the mucosal barrier in
the pathogenesis of inflammatory and autoimmune diseases and cancer:
contribution of germ-free and gnotobiotic animal models of human
diseases

Regulation of axial and head patterning during planarian regeneration by a commensal bacterium

Toxoplasma gondii infection and behavioral outcomes in humans: a systematic review

Resting potential, oncogene-induced tumorigenesis, and metastasis: the bioelectric basis of cancer in vivo

Transmembrane voltage potential of somatic cells controls oncogene-mediated tumorigenesis at long-range

Cross-limb communication during Xenopus hindlimb regenerative response: non-local bioelectric injury signals

Local
 and long-range endogenous resting potential gradients antagonistically
regulate apoptosis and proliferation in the embryonic CNS

Top-down models in biology: explanation and control of complex living systems above the molecular level

Generative Adversarial Networks

Adversarial Reprogramming of Neural Networks

Efficient Estimation of Word Representations in Vector Space

Fader Networks: Manipulating Images by Sliding Attributes

Maximizing the spread of influence through a social network

The Independent Cascade and Linear Threshold Models

A Survey on Influence Maximization in a Social Network

Simplicial models of social contagion

Cascading Behavior in Networks: Algorithmic and Economic Issues

On the Approximability of Influence in Social Networks

The Computational Boundary of a “Self”: Developmental Bioelectricity Drives Multicellularity and Scale-Free Cognition