Owain Evans
Andreas Stuhlm
Noah D Goodman

Learning the Preferences of Ignorant, Inconsistent Agents

Introduction The problem of learning a person's preferences from observations of their choices features prominently in economics  (Hausman 2011) , in cognitive science  (Baker, Saxe, and Tenenbaum 2011; Ullman et al. 2009) , and in applied machine learning  (Jannach et al. 2010; Ermon et al. 2014) . To name just one example, social networking sites use a person's past behavior to select what stories, advertisements, and potential contacts to display to them. A promising approach to learning preferences from observed choices is to Copyright c 2016, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. invert a model of rational choice based on sequential decision making given a real-valued utility function  (Russell and Norvig 1995) . This approach is known as Inverse Reinforcement Learning  (Ng and Russell 2000)  in an RL setting and as Bayesian Inverse Planning  (Baker, Saxe, and Tenenbaum 2009)  in the setting of probabilistic generative models. This kind of approach usually assumes that the agent makes optimal decisions up to "random noise" in action selection  (Kim et al. 2014; Zheng, Liu, and Ni 2014) . However, human deviations from optimality are more systematic. They result from persistent false beliefs, sub-optimal planning, and from biases such as time inconsistency and framing effects  (Kahneman and Tversky 1979) . If such deviations are modeled as unstructured errors, we risk mistaken preference inferences. For instance, if an agent repeatedly fails to choose a preferred option due to a systematic bias, we might conclude that the option is not preferred after all. Consider someone who smokes every day while wishing to quit and viewing their actions as regrettable. In this situation, a model that has good predictive performance might nonetheless fail to identify what this person values. In this paper, we explicitly take into account structured deviations from optimality when inferring preferences. We construct a model of sequential planning for agents with inaccurate beliefs and time-inconsistent biases (in the form of hyperbolic discounting). We then do Bayesian inference over this model to jointly infer an agent's preferences, beliefs and biases from sequences of actions in a simple Gridworld-style domain. To demonstrate that this algorithm supports accurate preference inferences, we first exhibit a few simple cases where our model licenses conclusions that differ from standard approaches, and argue that they are intuitively plausible. We then test this intuition by asking impartial human subjects to make preference inferences given the same data as our algorithm. This is based on the assumption that people have expertise in inferring the preferences of others when the domain is simple and familiar from everyday experience. We find that our algorithm is able to make the same kinds of inferences as our human judges: variations in choice are explained as being due to systematic factors such as false beliefs and strong temptations, not unexplainable error. The possibility of false beliefs and cognitive biases means that observing only a few actions often fails to identify a single set of preferences. We show that humans recognize this ambiguity and provide a range of distinct explanations for the observed actions. When preferences can't be identified uniquely, our model is still able to capture the range of explanations that humans offer. Moreover, by computing a Bayesian posterior over possible explanations, we can predict the plausibility of explanations for human subjects.
 Computational Framework Our goal is to infer an agent's preferences from observations of their choices in sequential decision problems. The key question for this project is: how are our observations of behavior related to the agent's preferences? In more technical terms, what generative model  (Tenenbaum et al. 2011)  best describes the agent's approximate sequential planning given some utility function? Given such a model and a prior on utility functions, we could "invert" it (by performing full Bayesian inference) to compute a posterior on what the agent values. The following section describes the class of models we explore in this paper. We first take an informal look at the specific deviations from optimality that our agent model includes. We then define the model formally and show our implementation as a probabilistic program, an approach that clarifies our assumptions and enables easy exploration of deviations from optimal planning.
 Deviations from optimality We consider two kinds of deviations from optimality: False beliefs and uncertainty Agents can have false or inaccurate beliefs. We represent beliefs as probability distributions over states and model belief updates as Bayesian inference. Planning for such agents has been studied in work on POMDPs  (Kaelbling, Littman, and Cassandra 1998) . Inferring the preferences of such agents was studied in recent work  (Baker and Tenenbaum 2014; Panella and Gmytrasiewicz 2014) . Here, we are primarily interested in the interaction of false beliefs with other kinds of sub-optimality. Temporal inconsistency Agents can be time-inconsistent (also called "dynamically inconsistent"). Time-inconsistent agents make plans that they later abandon. This concept has been used to explain human behaviors such as procrastination, temptation and pre-commitment  (Ainslie 2001) , and has been studied extensively in psychology  (Ainslie 2001)  and in economics  (Laibson 1997; O'Donoghue and Rabin 2000) . A prominent formal model of human time inconsistency is the model of hyperbolic discounting  (Ainslie 2001) . This model holds that the utility or reward of future outcomes is discounted relative to present outcomes according to a hyperbolic curve. For example, the discount for an outcome occurring at delay d from the present might be modeled as a multiplicative factor 1 1+d . The shape of the hyperbola means that the agent takes $100 now over $110 tomorrow, but would prefer to take $110 after 31 days to $100 after 30 days. The inconsistency shows when the 30th day comes around: now, the agent switches to preferring to take the $100 immediately. This discounting model does not (on its own) determine how an agent plans sequentially. We consider two kinds of time-inconsistent agents. These agents differ in terms of whether they accurately model their future choices when they construct plans. First, a Sophisticated agent has a fully accurate model of its own future decisions. Second, a Naive agent models its future self as assigning the same (discounted) values to options as its present self. The Naive agent fails to accurately model its own time inconsistency.  1  We illustrate Naive and Sophisticated agents with a decision problem that we later re-use in our experiments. The problem is a variant of Gridworld where an agent moves around the grid to find a place to eat (Figure  1 ). In the left pane (Figure  1a ), we see the path of an agent, Alice, who moves along the shortest path to the Vegetarian Cafe before going left and ending up eating at Donut Store D2. This behavior is sub-optimal independent of whether her preference is for the Vegetarian Cafe or the Donut Store, but can be explained in terms of Naive time-inconsistent planning. From her starting point, Alice prefers to head for the Vegetarian Cafe (as it has a higher undiscounted utility than the Donut Store). She does not predict that when close to the Donut Store (D2), she will prefer to stop there due to hyperbolic discounting. The right pane (Figure  1b ) shows what Beth, a Sophisticated agent with similar preferences to Alice, would do in the same situation. Beth predicts that, if she took Alice's route, she would end up at the Donut Store D2. So she instead takes a longer route in order to avoid temptation. If the longer route wasn't available, Beth could not get to the Vegetarian Cafe without passing the Donut Store D2. In this case, Beth would either go directly to Donut Store D1, which is slightly closer than D2 to her starting point, or (if utility for the Vegetarian Cafe is sufficiently high) she would correctly predict that she will be able to resist the temptation.
 Formal model definition We first define an agent with full knowledge and no time inconsistency, 2 and then generalize to agents that deviate from optimality. We will refer to states s ∈ S, actions a ∈ A, a deterministic utility function U : S ×A → R, a stochastic action choice function C : S → A, and a stochastic state transition function T : S × A → S. To refer to the probability that C(s) returns a, we use C(a; s). Optimal agent: full knowledge, no discounting Like all agents we consider, this agent chooses actions in proportion to exponentiated expected utility (softmax):  [a]  The noise parameter α modulates between random choice (α = 0) and perfect maximization (α = ∞). Expected utility depends on both current and future utility: C(a; s) ∝ e αEUs EU s [a] = U (s, a) + E s ,a [EU s [a ]] with s ∼ T (s, a) and a ∼ C(s ). Note that expected future utility recursively depends on C-that is, on what the agent assumes about how it will make future choices. Figure  2 : We specify agents' decision-making processes as probabilistic programs. This makes it easy to encode arbitrary biases and decision-making constraints. When automated inference procedures invert such programs to infer utilities from choices, these constraints are automatically taken into account. Note the mutual recursion between agent and expUtility: the agent's reasoning about future expected utility includes a (potentially biased) model of its own decision-making.
 Time-inconsistent agent To compute expected utility, we additionally take the expectation over states. Now EU p(s),o,d [a] is defined as: Inferring preferences We define a space of possible agents based on the dimensions described above (utility function U , prior p(s), discount parameter k, noise parameter α). We additionally let Y be a variable for the agent's type, which fixes whether the agent discounts at all, and if so, whether the agent is Naive or Sophisticated. So, an agent is defined by a tuple θ := (p(s), U, Y, k, α), and we perform inference over this space given observed actions. The posterior joint distribution on agents conditioned on action sequence a 0:T is: E s∼p(s|o) 1 1 + kd U (s, a) + E s , P (θ|a 0:T ) ∝ P (a 0:T |θ)P (θ) (1) The likelihood function P (a 0:T |θ) is given by the multistep generalization of the choice function C corresponding to θ. For the prior P (θ), we use independent uniform priors on bounded intervals for each of the components. In the following, "the model" refers to the generative process that
 Sophisticated Figure  3 : Given data corresponding to Figure  1 , the model infers a joint posterior distribution on preferences, beliefs and other agent properties (such as discount strength) that reveals relations between different possible inferences from the data. The darker a cell, the higher its posterior probability. involves a prior on agents and a likelihood for choices given an agent.
 Agents as probabilistic programs We implemented the model described above in the probabilistic programming language WebPPL (Goodman and Stuhlmüller 2014). WebPPL provides automated inference over functional programs that involve recursion. This means that we can directly translate the recursions above into programs that represent an agent and the world simulation used for expected utility calculations. All of the agents above can be captured in a succinct functional program that can easily be extended to capture other kinds of sub-optimal planning. Figure  2  shows a simplified example (including hyperbolic discounting but not uncertainty over state). For the Bayesian inference corresponding to Equation 1 we use a discrete grid approximation for the continuous variables (i.e. for U , p(s), k and α) and perform exact inference using enumeration with dynamic programming.
 Model inferences We now demonstrate that the model described above can infer preferences, false beliefs and time inconsistency jointly from simple action sequences similar to those that occur frequently in daily life. We later validate this intuition in our experiments, where we show that human subjects make inferences about the agent that are similar to those of our model. Example 1: Inference with full knowledge We have previously seen how modeling agents as Naive and Sophisticated might predict the action sequences shown in Figures  1a and 1b  respectively. We now consider the inference problem. Given that these sequences are observed, what can be inferred about the agent? We assume for now that the agent has accurate beliefs about the restaurants and that the two Donut Stores D1 and D2 are identical (with D1 closer to the starting point).  4  We model each restaurant as having an immediate utility (received on arriving at the restaurant) and a delayed utility (received one time-step after). This interacts with hyperbolic discounting, allowing the model to represent options that are especially "tempting" when they can be obtained with a short delay. For the Naive episode (Figure  1a ) our model infers that either softmax noise is very high or that the agent is Naive (as explained for Alice above). If the agent is Naive, the utility of the Vegetarian Cafe must be higher than the Donut Store (otherwise, the agent wouldn't have attempted to go to the Cafe), but not too much higher (or the agent wouldn't give in to temptation, which it in fact does). This relationship is exhibited in Figure  3   Example 2: Inference with uncertainty In realistic settings, people do not have full knowledge of all facts relevant to their choices. Moreover, an algorithm inferring preferences will itself be uncertain about the agent's uncertainty. What can the model infer if it doesn't assume that the agent has full knowledge? Consider the Sophisticated episode (Figure  1b ). Suppose that the Noodle Shop is closed, and that the agent may or may not know about this. This creates another possible inference, in addition to Sophisticated avoidance of temptation and high noise: The agent might prefer the Noodle Shop and might not know that it is closed. This class of inferences is shown in Figure  3  (bottom): When the agent has a strong prior belief that the shop is open, the observations are most plausible if the agent also assigns high utility to the Noodle Shop (since only then will the agent attempt to go there). If the agent does not believe that the shop is open, the Noodle Shop's utility does not matter-the observations have the same plausibility either way. In addition, the model can make inferences about the agent's discounting behavior (Figure  3  right): When utility for the Vegetarian Cafe is low, the model can't explain the data well regardless of discount rate k (since, in this case, the agent would just go to the Donut Store directly). The data is best explained when utility for the Vegetarian Cafe and discount rate are in balance-since, if the utility is very high relative to k, the agent could have gone directly to the Vegetarian Cafe, without danger of giving in to the Donut Store's temptation. Example 3: Inference from multiple episodes Hyperbolic discounting leads to choices that differ systematically from those of a rational agent with identical preferences. A time-inconsistent agent might choose one restaurant more often than another, even if the latter restaurant provides more
 Experiments with Human Subjects We have shown that, given short action sequences, our model can infer whether (and how) an agent is timeinconsistent while jointly inferring appropriate utilities. We claim that this kind of inference is familiar from everyday life and hence intuitively plausible. This section provides support for this claim by collecting data on the inferences of human subjects. In our first two experiments, we ask subjects to explain the behavior in Figures  1a and 1b . This probes not just their inferences about preferences, but also their inferences about biases and false beliefs that might have influenced the agent's choice. Experiment 1: Inference with full knowledge Experiment 1 corresponds to Example 1 in the previous section (where the agent is assumed to have full knowledge). Two groups of subjects were shown Figures  1a and 1b , having already seen two prior episodes showing evidence of a preference for the Vegetarian Cafe over the other restaurants. People were then asked to judge the plausibility of different explanations of the agent's behavior in each episode.  5  Results are shown in Figure  5 . In both Naive (Figure  1a ) and Sophisticated (1b) conditions, subjects gave the highest ratings to explanations involving giving in to temptation (Naive) or avoiding temptation (Sophisticated). Alternative explanations suggested that the agent wanted variety (having taking efficient routes to the Vegetarian Cafe in previous episodes) or that they acted purely based on a preference (for a long walk or for the Donut Store). Experiment 2: Inference with uncertainty Experiment 2 corresponds to Example 2 above. Subjects see one of the two episodes in Figure  1  (with Figure  1a  modified so D1 and D2 can differ in utility and Figure  1b  modified so the Noodle Shop is closed). There is no prior information about the agent's preferences, and the agent is not known to have accurate beliefs. We asked subjects to write explanations for the agent's behavior in the two episodes and coded these explanations into four categories. Figure  6  specifies which formal agent properties correspond to which category. While not all explanations correspond to something the model can infer about the agent, the most common explanations map cleanly onto the agent properties θ-few explanations provided by people fall into the "Other" category (Figure  7 ). The model inferences in this figure show the marginal likelihood of the observed actions given the corresponding property of θ, normalized across the four property types. In both the Naive and the Sophisticated case, the model and people agree on what the three highest-scoring properties are. Explanations involving false beliefs and preferences rate more highly than those involving time inconsistency. This is because, even if we specify whether the agent is Naive/So-  Experiment 3: Inference from multiple episodes Following Example 3 above, subjects (n=50) saw the episodes in Figure  4  and inferred whether the agent prefers the Vegetarian Cafe or the Donut Store. Like the model, the majority of subjects inferred that the agent prefers the Vegetarian Cafe. Overall, 54% (+/-7 for 95% CI) inferred a preference of Vegetarian Cafe over the Donut Store, compared to the 59% posterior probability assigned by the model. Episode 2 (in which the agent does not choose the Donut Store) is identical to the Sophisticated episode from Figure  1 . Experiments 1 and 2 showed that subjects explain this episode in terms of Sophisticated time-inconsistent planning. Together with Experiment 3, this suggests that subjects use this inference about the agent's planning to infer the agent's undiscounted preferences, despite having seen the agent choose the Donut Store more frequently.
 Conclusion AI systems have the potential to improve our lives by helping us make choices that involve integrating vast amounts of information or that require us to make long and elaborate plans. For instance, such systems can recommend and filter the information we see on social networks or music services and can construct intricate plans for travel or logistics. For these systems to live up to their promise, we must be willing to delegate some of our choices to them-that is, we need such systems to reliably act in accordance with our preferences and values. It can be difficult to formally specify our preferences in complex domains; instead, it is desirable to have systems learn our preferences, just as learning in other domains is frequently preferable to manual specification. This learning requires us to build in assumptions about how our preferences relate to the observations the AI system receives. As a starting point, we can assume that our choices result from optimal rational planning given a latent utility function. However, as our experiments with human subjects show, this assumption doesn't match people's intuitions on the relation between preferences and behavior, and we find little support for the simplistic model where what is chosen most is inferred to be the most valued. We exhibited more realistic models of human decision-making, which in turn supported more accurate preference inferences. By approaching preference inference as probabilistic induction over a space of such models, we can maintain uncertainty about preferences and other agent properties when the observed actions are ambiguous. This paper has only taken a first step in the direction we advocate. Two priorities for further work are applications to more realistic domains and the development of alternatives to using human preference inferences as a standard by which to evaluate algorithms. The goal for this emerging subfield of AI is to make systems better able to support humans even in domains where human values are complex and nuanced, and where human choices may be far from optimal. Figure 1 : 1 Figure 1: Agents with hyperbolic discounting exhibit different behaviors depending on whether they model their future discounting behavior in a manner that is (a) Naive (left) or (b) Sophisticated (right).
 o ,a EU p(s|o),o ,d+1 [a ] with s ∼ T (s, a), o ∼ p(o|s ) and a ∼ C(p(s|o), o , d + 1) (for the Naive agent) or a ∼ C(p(s|o), o , 0) (for the Sophisticated agent).
 (top left), which shows the model posterior for the utilities of the Donut Store and Vegetarian Cafe (holding fixed the other agent components Y , k, and α).
 Figure 4 : 4 Figure 4: The observations in Experiment 3 show the Donut Chain Store being chosen twice and the Vegetarian Cafe once.
 Figure 5 : 5 Figure5: Explanations in Experiment 1 for the agent's behavior in Figure1a(Naive) and 1b (Sophisticated). Subjects (n=120) knew that the agent has accurate knowledge, and saw prior episodes providing evidence of a preference for the Vegetarian Cafe. Subjects selected scores in {1, 2, 3}.
 Figure 6 :Figure 7 : 67 Figure6: Map from properties invoked in human explanations to formalizations in the model. The left column describes the property. The center column shows how we formalized it in terms of the variables used to define an agent θ. The right column gives an explanation (from our dataset of human subjects) that invokes this property.
 Now the agent's choice and expected utility function are parameterized by a delay d, which together with a constant k controls how much to discount future utility: var agent = function(state, delay){ return Marginal( function(){ var action = uniformDraw(actions) var eu = expUtility(state, action, delay) factor(alpha * eu) return action }) } var expUtility = function(state, action, delay){ if (isFinal(state)){ return 0 } else { var u = 1/(1 + k * delay) * utility(state, action) return u + Expectation(function(){ var nextState = transition(state, action) var nextAction = sample(agent(nextState, delay+1)) return expUtility(nextState, nextAction, delay+1) }) } } EU s,d [a] = 1 1 + kd U (s, a) + E s ,a The agent's choice and expected utility functions are now parameterized by the distribution p(s) and the current ob- servation o: C(a; p(s), o, d) ∝ e αEU p(s),o,d [a] C(a; s, d) ∝ e αEU s,d[a]    [EU s ,d+1[a ]] with s ∼ T (s, a). For the Naive agent, a ∼ C(s , d + 1), whereas for the Sophisticated agent, a ∼ C(s , 0). When we compute what the agent actually does in state s, we set d to 0. As a consequence, only the Sophisticated agent correctly predicts its future actions.3  An implementation of the Naive agent as a probabilistic program is shown in Figure2.Time-inconsistent agent with uncertaintyWe now relax the assumption that the agent knows the true world state. Instead, we use a distribution p(s) to represent the agent's belief about which state holds. Using a likelihood function p(o|s), the agent can update this belief: p(s|o) ∝ p(s)p(o|s)
 = open) > 0.85 "He was heading towards the noodle shop first, but when he got there, it was closed, so he continued on the path and ended up settling for ... vegetarian cafe." Property Formalization Example explanation from our human subjects Agent doesn't know p(D1 = open) < 0.15 "He decided he wanted to go to the Donut Store for lunch. He Donut Store D1 is open. did not know there was a closer location" Agent falsely believes Noodle Shop is open. p(N Agent prefers D2 to D1. U (D2) > U(D1) "He might also enjoy the second donut shop more than the first" Agent is Naive / Sophisti- Y = Naive/Soph. "He ... headed for the Vegetarian Cafe, but he had to pass by cated. the Donut shop on his way. The temptation was too much to fight, so he ended up going into the Donut Shop."
			 Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence
			 The distinction and formal definition of Naive and Sophisticated agents is discussed in O'Donoghue and Rabin (1999).
			 This is the kind of agent assumed in the standard setup of an MDP (Russell and Norvig 1995)  3 This foresight allows the Sophisticated agent to avoid tempting states when possible. If such states are unavoidable, the Sophisticated agent will choose inconsistently.
			 In Experiment 2, we allow the utilities for D1 and D2 to be different. See row 3 of Figure6and the "Preference" entry for Sophisticated in Figure7.
			 In a pilot study, we showed subjects the same stimuli and had them write free-form explanations. In Experiment 1, subjects had to judge four of the explanations that occurred most frequently in this pilot.

Abstract: An important use of machine learning is to learn what people value. What posts or photos should a user be shown? Which jobs or activities would a person find rewarding? In each case, observations of people's past choices can inform our inferences about their likes and preferences. If we assume that choices are approximately optimal according to some utility function, we can treat preference inference as Bayesian inverse planning. That is, given a prior on utility functions and some observed choices, we invert an optimal decision-making process to infer a posterior distribution on utility functions. However, people often deviate from approximate optimality. They have false beliefs, their planning is sub-optimal, and their choices may be temporally inconsistent due to hyperbolic discounting and other biases. We demonstrate how to incorporate these deviations into algorithms for preference inference by constructing generative models of planning for agents who are subject to false beliefs and time inconsistency. We explore the inferences these models make about preferences, beliefs, and biases. We present a behavioral experiment in which human subjects perform preference inference given the same observations of choices as our model. Results show that human subjects (like our model) explain choices in terms of systematic deviations from optimal behavior and suggest that they take such deviations into account when inferring preferences.


Unifying Principles and Metrics for Safe and Assistive AI

Introduction Recent years have witnessed immense progress in research on safe and assistive AI systems as well as on the potential impact of AI on the future of work. These directions of research address two sides of a common, fundamental concern: how would humans work with AI systems? While research on AI safety focuses on designing AI systems that allow humans to safely instruct and control them (e.g.,  (Russell, Dewey, and Tegmark 2015; Zilberstein 2015; Hadfield-Menell et al. 2016; Russell 2017; Hadfield-Menell et al. 2017 )), research on AI and the future of work focuses on the impact of AI on members of the workforce who may be unable to do so  (Arntz, Gregory, and Zierahn 2016; Manyika et al. 2017; Nedelkoska and Quintini 2018) . This paper presents the view that in addition to the productive streams of research outlined above, we need unifying metrics and declarative objectives that would allow a more uniform evaluation of AI systems on the extent to which an AI system is suitable for working with specific classes of human operators. It also presents a common principle for human-centered AI systems that allows the development of such metrics. Consequently, rather than proposing a specific new design for AI systems, the focus of this paper Copyright c 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. is on elucidating the declarative principles and types of metrics that would lead to concerted progress on the problem. The advantage of this declarative approach to framing the problem is that it enables an assessment of progress independent of the internal design being used in an AI system, and it will help draw out the strengths and weaknesses of different design approaches. Without such a specification, design differences can make solution paradigms difficult to compare. E.g., one might develop a complex system architecture that builds user profiles and provides appropriate assistance. This system would have very different input requirements and design and performance parameters than a formulation that addresses the same problem by computing assistance policies using planning under partial observability while incorporating the value of information to learn more about a user's poorly articulated objectives and constraints. Better declarative objectives and metrics for assistive AI systems would also help ensure that, regardless of the methods being used, progress amounts to advancement towards safe and assistive AI systems. More pragmatically, such metrics will not only help end-users assess the utility of a given AI system but they will also help AI researchers and developers identify more readily the dimensions along which further research will be beneficial for applications of their interest. The next section presents a succession of intuitive principles for safe and assistive AI systems, and shows that evaluating the compatibility of a system with such principles (in particular P2) helps clarify the required types of metrics. The paper concludes by drawing the attention of our community towards research on the operationalization of such metrics along with promising research directions on developing systems that do well on them.
 Unifying Principles for Safe and Assistive AI Systems We focus on taskable AI systems that carry out user-assigned high-level tasks using arbitrary mechanisms for reasoning and planning over multiple time steps. E.g., household robots that can be given objectives such as setting the table or doing laundry, co-manufacturing robots that can assist workers in creating complex assemblies with heavy components, digital assistants that can plan a vacation given the The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21) user's preferences, etc. Such systems serve as sound integrative platforms and model end-to-end applications where the AI system is responsible for assistance in the execution of long-horizon tasks. AI systems are frequently evaluated in terms of performance measures such as the computational complexity of computing the required behavior, training data requirements, and the quality of the computed behavior in terms of execution time, resources used, risk of unsafe outcomes etc. We can consider systems that optimize such performance metrics as Level 0 of assistive AI systems. Level I of assistive AI systems Recent AI research has also focused on assistive properties of AI systems. We begin with a rather common-sensical principle defining Level I of such safe and assistive AI systems: P1: An AI system must make it easy for its operators to use it safely. The italicized terms denote dimensions along which compatibility of AI systems with principle P1 can be evaluated; while a lot of current AI research utilizes one or more of these dimensions for evaluation, a closer analysis reveals some new insights. In the context of this paper we consider using an AI system to be synonymous with instructing it to change its behavior as desired. Different interfaces may be used for this, including programming, text, speech, gestures etc. We consider the operators of an AI system as those persons who use it in the sense described above. For instance, if a self-driving car gives all its passengers the right to give it instructions, then all of them are its operators; if it gives instruction-rights to only a qualified set of users, perhaps adults who pass an assisted-driving exam, then the class of operators is defined by that qualification exam. Safety refers to the overall safety of the AI system's interactions with its environment, which may be physical or online. These dimensions of compatibility with P1 serve as natural dimensions for evaluating Level I assistive AI systems: E1. How inclusive is the set of operators? Systems whose operators require PhD-level expertise in AI may be less desirable for broader deployments. E2. How easy is it for the system's operators to change its behavior? E3. Which set of tasks can the AI system be used for? E4. What form of safety guarantees does the system provide? Systems that are unable to provide an upper bound on expected risks are clearly less desirable than those that stipulate conditions under which upper bounds can be provided. P1 serves to highlight the interplay between these dimensions of compliance and evaluation. Safety guarantees are often inversely related with the size of the operator set. A system may provide a high level of safety, but only under the requirement that its operators take extensive training programs. At one end of the spectrum, automated robot vacuum cleaners require almost no prior skills and perform limited, well-defined tasks. Safety issues are still present-a robot vacuum cleaner may pull on electrical cables that have been left on the floor; auto-completion software may send emails to unintended recipients. However, the lower expected damage from using such applications has made them broadly accepted in society. AI-powered industrial robots are at the other end of the spectrum: these devices require specialized training as well as operating environments in order to ensure safety (see for instance, ISO/TS 15066:2016 on collaborative robots). Typically, these systems operate in regions that humans cannot access when the robot is active unless the human is hand-guiding the robot within a safe operating envelope that limits the speed and range of operation. Their functionality is closely controlled and monitored. Only skilled engineers change their functionality while day-to-day operators monitor execution of predictable behavior and control run-stops from designated safe areas. Similarly, the safety of current airborne drone operations is ensured by requiring specially trained drone operators  (Marshall 2020) . Thus, principle P1 holds for AI systems today with varying degrees of compliance along E1-E4. The examples above illustrate how practical implementations often rely upon implicitly defined operator classes to provide acceptable levels of safety. Such approaches rely upon limiting the users and the scope of a system to achieve an acceptable level of compatibility with P1: it is easy for such systems' users to operate it safely because the set of users is required to be sufficiently skilled, and its functionality is sufficiently limited for that group to be able to safely use the device. However, a broader emphasis on the need to specify safety guarantees with respect to different classes of operators would help mitigate some of the risks associated with broadly deployed AI systems. In contrast to classical automated systems, AI systems feature a more nuanced interplay between the class of tasks that a system can be used for (E3) and the other dimensions above. Traditionally deployed systems (even automated systems) have a very well-defined boundary of use cases. This allows for an easier classification of safety. Taskable AI systems on the other hand, are expected to change their behavior and functionality as they learn and adapt to new environments or new user-given objectives. For such systems, we need better methods for deriving the range of instructions that different operator-groups are allowed to provide. Scenarios like the self-driving car that needs to decide whether to obey a child's instruction allude to this requirement. Methods for assessing user and AI competency can also allow the AI system to expand its range of functionality by drawing upon the expertise of its operator  (Basich et al. 2020)  while ensuring an acceptable level of safety. A major limitation of P1 and its associated metrics is that it does not evaluate the amount of training required for an individual to qualify as an operator for the system. This creates a blind-spot in evaluation of the ease-of-use or safety of an AI system: since user-training occurs outside the requirements outlined by P1, an unsafe AI system (or one that is deployed in an unsafe manner) would simply claim that its so-called operator was insufficiently trained! Furthermore, if P1 were a sufficient categorization of safe and assistive AI systems, we would have no need for explainable AI as compliance with P1 does not require the system to be easy to understand. An implicit emphasis on assessing AI systems only along some aspects of P1 may also explain the increasing prevalence of concerns about the workers who may be left behind in the future workplace. From this perspective it is unsurprising that these concerns have gained renewed interest at a time when AI applications have reached a level of maturity where they are being used by non-AI-experts in situations that have some inherent safety risks. However, the "assistive" nature of such AI systems is undermined by the need for highly skilled individuals who could safely debug, understand and modify the behavior of such systems. Level II of assistive AI systems In order to address the limitations of P1, we consider the following as a guiding principle for safe and assistive AI: P2: An AI system must make it easy for its operators to learn how to use it safely. P2 changes the notion of operators from those who are qualified to use a given AI system to those who are qualified to start learning how to use it. In addition to the metrics associated with P1, P2 introduces a new dimension: E5. How easy is it to learn how to use the AI system? What are the expected prerequisites and costs of training for its operators? Can training be provided on-the-job? This dimension could also be viewed as evaluating the resources required to train operators for P1 systems. Most AI systems in use today would perform poorly on this new dimension, and consequently, on compatibility with P2 as a whole. Explainable AI (e.g.,  (Ribeiro, Singh, and Guestrin 2016; Hayes and Shah 2017; Chakraborti et al. 2017; Hoffman et al. 2018; Gunning and Aha 2019; Weld and Bansal 2019; Anderson et al. 2019; Eifler et al. 2020 )) plays a key role along this dimension because systems that are easy to understand or that can explain themselves naturally make it easier for people to learn how to use them. P2 leverages the unique strengths of AI as a field of research. AI research already addresses the problem of estimating users' skills; research on intelligent tutoring systems and AI for education addresses the problem of identifying skill gaps. This can be used to determine the minimal differential training to be provided to an operator. P2 places the onus of training on the deployed AI system and opens up a new direction of interdisciplinary research connecting existing research directions in AI with research in humansystems engineering and in industrial engineering for the development of productive training modalities and the operationalization of metrics for E5. It also allows AI systems to formally characterize different scopes of functionality for different classes of operators, e.g., operators that use manufacturing robots for pre-determined tasks, those that give the robots new instructions, or those that are ready to learn about giving the robot new instructions. P2 is not required for every AI system-P1 would be sufficient for systems that place minimal requirements on operator qualifications (e.g., robot vacuum cleaners) and for non-adaptive AI systems that require a small set of operators. On the other hand, P2 serves as a better declarative foundation for evaluating taskable AI systems that are meant to assist large numbers of non-AI-experts on a wide range of tasks. Increasing concerns about job roles that would feature a high-degree of interaction with AI systems (and the workers that are likely to be left behind) allude to the pressing need for including E5, a dimension for evaluation under P2 (and not P1) as a part of an AI system's evaluation. AI systems that are not beneficial (either in terms of AI safety or in terms of the future of work) fare poorly on P2. E.g., systems that can thwart their users' objectives by wireheading and those that may derive incorrect objective functions from user instructions make it difficult for an operator to learn how to provide instructions that are specific enough to be safe, and fare poorly along E5. Similarly, systems that require extensive training investment to be used effectively and safely fail along E5. In this way P2 serves as a unifying principle encompassing research on AI safety as well as on AI for a beneficial future of work.
 Promising Directions of Research P2 serves as a declarative principle for guiding research on assistive AI systems as well as for developing metrics for evaluating AI systems and their deployments. Converting this principle into tangible metrics calls for interdisciplinary research including AI and other fields associated with human factors. The increasing prevalence of research thrusts on safe and assistive AI systems  (Fern et al. 2014; Russell, Dewey, and Tegmark 2015; Amodei et al. 2016; Gunning and Aha 2019)  makes this a particularly opportune phase for formalizing the metrics and the interfaces required for evaluating AI systems for compatibility with P2 along dimensions E1-E5. Recent research on AI safety and explainable AI develops methods improving the ease of use and safety of AI systems along P2 (see, for instance, the ICML 2020 Workshop on Explainable AI). Placing AI systems that compute user-skill aligned explanations  (Sreedharan, Srivastava, and Kambhampati 2018; Sreedharan et al. 2019 ) in a loop with AI systems for identifying user-skills and skill-gaps can help develop AI systems that gradually present users with new functionality and explain it, thereby training their users onthe-fly and as needed. Such systems would be better tuned towards P2, and towards addressing the underlying problems of AI safety and the future of work. Critically, ensuring progress towards safe and assistive AI systems requires that AI systems with arbitrary internal designs support assessment along the metrics developed for E1-E5. This raises a new set of research questions: Can we develop non-intrusive AI-interface requirements for supporting such evaluations in the face of changing operating environments and objectives? The need for such interfaces is even more pressing for systems that learn and those that undergo system updates after deployment. What is the minimal external interface that an AI system must support so as to allow its independent evaluation? How would changing the nature of such interfaces change the complexity of conducting such an evaluation? One would expect that AI sys-tems that offer more transparency would be easier to evaluate. Could we use the inherent reasoning capabilities of AI systems to develop interface requirements that would allow more adept systems to make such evaluations easier? E.g., rather than testing a manufacturing robot to discover its response to every possible situation, could we ask higher-level queries such as "under which situations would you be able to create the proposed assembly?" Clearly, the ease of assessment of an AI system would depend on the class of queries that it can answer. Recent work suggests that a minimal query-response interface for AI systems that connects the system with a simulator and observes its responses to high-level instructions can be quite powerful. Such an interface has a few distinct advantages. Current AI systems are already tested with simulators and they are inherently required to be able to take user instructions, so these interface requirements can be considered to be natural. They also allow the autonomous synthesis of query-policies: running the query-policy on a black-box taskable AI system can help construct an interpretable model of the limits and capabilities of that system  (Verma, Marpally, and Srivastava 2021) . Such models can be used to support the evaluations discussed above. Extensions of such interface requirements to arbitrary AI systems would help ensure that our AI systems are amenable to independent evaluation. Such a paradigm would allow users to assess their AI systems while freeing AI researchers and developers to utilize arbitrary internal implementations. Systems with interfaces that support more efficient and accurate independent assessment would be rewarded with greater public adoption of their products. Progress on these threads would help prevent undesirable situations such as insufficient support for independent evaluation of powerful AI systems, and the negative consequences of deployment of an insufficiently evaluated system.

Abstract: The prevalence and success of AI applications have been tempered by concerns about the controllability of AI systems about AI's impact on the future of work. These concerns reflect two aspects of a central question: how would humans work with AI systems? While research on AI safety focuses on designing AI systems that allow humans to safely instruct and control AI systems, research on AI and the future of work focuses on the impact of AI on humans who may be unable to do so. This Blue Sky Ideas paper proposes a unifying set of declarative principles that enable a more uniform evaluation of arbitrary AI systems along multiple dimensions of the extent to which they are suitable for use by specific classes of human operators. It leverages recent AI research and the unique strengths of the field to develop human-centric principles for AI systems that address the concerns noted above.


Martin Dresler
Anders Sandberg
Christoph Bublitz
Kathrin Ohla
Carlos Trenado
Aleksandra Mroczko-Wąsowicz
○,◆ Simone Kuḧn
Dimitris Repantis

Hacking the Brain: Dimensions of Cognitive Enhancement

INTRODUCTION An increasingly complex world exerts increasing demands on cognitive functionsfunctions that evolved for a fundamentally different environment. Daily life in an information society and a postindustrial economy require cognitive skills that have to be acquired through slow, effortful, and expensive processes of education and training. Likewise, these skills can become obsolete as the world changes ever faster or be lost by the processes of aging. People also vary in their mental abilities, allowing them to acquire certain skills more quickly or slower, which may have significant effects on life outcomes. Strategies to improve the acquisition and maintenance of cognitive skills are thus increasingly important on both an individual and societal level. These challenges of our times have fostered the See https://pubs.acs.org/sharingguidelines for options on how to legitimately share published articles. exploration of strategies to enhance human brain function. While people have since time immemorial sought to improve their performance, the present era is unique in that not only the challenges are growing rapidly but so are technologies that promise to meet them. Just like the hacking culture in the realm of computer software and hardware, an increasing number of individuals experiment with strategies to creatively overcome the natural limitations of human cognitive capacityin other words, to hack brain function. This development has led to both enthusiasm and dread, as observers have sharply differing intuitions about the feasibility, utility, risks, and eventual impact of enhancement technologies on the world. One reason for the often polarized debates has been the lack of hard evidence. Without empirical findings, it is easy to maintain any position, as well as regard opponents as having unfounded views. A further essential source of disagreement and theoretical confusion is a tendency to view enhancement as a unitary phenomenon to be judged as a whole, rather than as a broad set of techniques with important differences and diverging implications. Only on the basis of a clear picture on how a particular enhancement strategy might affect specific cognitive processes in specific populations, along with side effects and costs to be expected, an informed theoretical debate can evolve and a promising empirical research designs to test the strategy can be proposed. In the following, we aim to elucidate seven essential dimension of cognitive enhancement, namely, (a) its mode of action, (b) the cognitive domain targeted, (c) personal factors, (d) its time scale, (e) side effects, (f) availability, and (g) social acceptance (see Figure  1 ). Further, we will review empirical data of prominent examples of cognitive enhancers that differ across these dimensions and thereby illustrate some of their nuanced implications. The aim of our Review is to sketch a general framework that will foster both theoretical discussions and empirical research.
 MODE OF ACTION A widely cited definition characterizes enhancement as interventions in humans that aim to improve mental functioning beyond what is necessary to sustain or restore good health.  1  While the current bioethical debate on cognitive enhancement is strongly focused on pharmacological ways of enhancement, improving mental capabilities also by nonpharmacological means has to be considered as cognitive enhancement proper according to the given characterization. We have reviewed elsewhere the efficacy of a number of nonpharmacological enhancers.  2, 3  To systematize the vast variety of different approaches of cognitive enhancement, we suggest clustering enhancement strategies into three major areas according to their main mode of action. Even though boundaries are not strict, most cognitive enhancement strategies can be considered to work as either biochemical, physical, or behavioral interventions (Figure  2 ). In the following, we will give an overview on the different cognitive enhancement strategies within these clusters.
 Biochemical Strategies. The prototypical cognitive enhancers addressed in the public debate are biochemical agents. However, biochemical interventions are not restricted to pharmaceutical "smart drugs". Also application of ordinary substances such as oxygen has been shown to increase, e.g., memory processes  4, 5  and neural activation in memory-related brain regions.  6  Biochemical enhancers with the longest tradition in human history are strategies to make use of certain nutritional components. Most widely used are probably glucose  7  and caffeine,  8, 9  which both have demonstrated cognition-enhancing effects in numerous studies. In addition to coffee, other beverages from caffeine-bearing plants such as guarana have shown to enhance cognition.  10  While the noncaffeine components in caffeine-bearing plants might exert independent effects on cognition,  11  it has been doubted that industrially designed drinks contain cognitive enhancing components that go beyond caffeine, glucose, or guarana extract.  12  Further nutritional components with some evidence for cognitive enhancing effects are flavonoids, e.g., in cocoa,  13, 14  curry powder (most likely due to the curcumin that it contains,  15, 16  folic acid,  17  or omega-3 fatty acids.  18  Besides specific dietary supplements, also the absence of food might enhance cognition: some evidence has been reported that fasting and general caloric restriction might improve memory in elderly individuals.  19, 20  Also certain traditional natural remedies have been discussed as cognitive enhancers: besides herbs that also grow in Western regions such as salvia,  21  particularly traditional Chinese and
 ACS Chemical Neuroscience
 Review Indian herbal medicines such as Bacopa monnieri have been ascribed with cognitive enhancing effects.  22, 23  However, with ginseng and ginkgo biloba, the most prominent examples of such traditional Asian herbal remedies so far have failed to consistently show positive effects on cognitive functions in healthy individuals.  24, 25  A further biochemical intervention with a long history concerns drugs that are being used recreationally and that have demonstrated the potential to enhance certain cognitive functions. For example, nicotine improves attention and memory,  26−28  and even alcohol, despite impairing many cognitive functions, might enhance others such as creative processes  29, 30  or, retroactively, memory.  31  Pharmaceuticals are in particular by the public regarded as prototypical cognitive enhancers: synthetic stimulants such as amphetamine, methylphenidate, or modafinil, or antidementia drugs such as acetylcholinesterase inhibitors and memantine are at the core of public debate on cognitive enhancement. However, evidence for their efficacy for augmenting brain function and cognition in healthy subjects is often markedly lower than assumed in theoretical discussions.  32−37  Importantly, the lack of an objective effect on cognition can be accompanied by a considerable placebo effect: for example, users who believed to have received mixed-amphetamine salts subjectively rated themselves as performing better and even show minor objective performance increases, independent of actual medication state.  38  While pharmacological enhancers are typically designed to affect or mimic certain neurotransmitters, also neural signaling molecules themselves such as adrenaline,  39  GABA,  40  glucocorticoids,  41  ovarian hormones,  42  and different neuropeptides  43−45  have been suggested as cognitive enhancers. A further biochemical strategy for cognitive enhancement consists of genetic modifications, which have been demonstrated to augment several learning and memory processes in animal models.  46−51  Although progress has also been made in elucidating the genetic basis of cognitive traits in humans,  52  genetic modifications in humans still have to be considered as future strategies rather than currently available enhancement options. 2.2. Physical Strategies. The current most widely discussed physical strategies for cognitive enhancement include a number of brain stimulation technologies. Whereas the cognition enhancing effects of invasive methods such as deep brain stimulation  53, 54  are restricted to subjects with pathological conditions, several forms of allegedly noninvasive stimulation strategies are increasingly used on healthy subjects, among them electrical stimulation methods such transcranial direct current stimulation (tDCS  55  ), transcranial alternating current stimulation (tACS  56  ), transcranial random noise stimulation (tRNS  57  ), transcranial pulsed current stimulation (tPCS  58, 59  ), transcutaneous vagus nerve stimulation (tVNS  60  ), or median nerve stimulation (MNS  61  ). Details of the stimulation procedures appear to be crucial: commercial do-it-yourself electrical brain stimulators might impair rather than enhance cognition,  62  and systematic reviews have shed doubt on a clear and simple enhancing effect of electrical brain stimulation on different cognitive domains also under controlled laboratory conditions.  63, 64  Recent studies have even questioned if some of the most commonly used setups for electrical brain stimulation have neurophysiologically meaningful effects at all.  65−68  On this background, the development of noninvasive deep brain stimulation via temporally interfering electric fields might provide a more systematic and targeted mechanism compared to the currently used approaches.  68  Besides electrical stimulation methods, also for transcranial magnetic stimulation (TMS  69  ), optical stimulation with lasers,  70  and several forms of acoustic stimulation, such as transcranial focused ultrasound stimulation,  71  binaural beats,  72, 73  or auditory stimulation of the EEG theta rhythm  74  or sleep EEG slow oscillations,  75  a potential for cognitive enhancement has been reported. Physical enhancement methods that target brain processes more indirectly include whole body vibrations,  76  stochastic resonance motor control improvement,  77, 78  and several forms of neurofeedback, 79 with, e.g., EEG neurofeedback in the upper alpha band enhancing memory,  80  working memory,  81  and visuospatial skills.  82  Besides classical neurofeedback training that involves unspecific but active effort of the subject, also neurofeedback interventions that automatically feedback low energy currents in response to EEG activity have been developed, thereby allowing the subject to receive the training procedure passively.  83  Recently, the use of fMRI neurofeedback, utilizing multivariate pattern analysis, has shown the potential to increase sustained attention  84  or visuospatial memory.  85  Finally, humans have always deployed physical tools to assist cognitive functioning. In current developments that converge minds and machines, these tools become more closely integrated with the person.  86  Crowdfunding or biohacking communities have developed numerous novel technical devices to increase cognitive functions transiently with, e.g., wearable electronic memory aids or augmented reality gadgets,  87, 88  or more permanently as in the case of cognition enhancing or extending bodily implants.  88, 89  Neural implants or prosthetics have progressed; in controlled laboratory settings, implants could facilitate human memory.  90  In addition, Brain−Computer Interfaces connect the central nervous system with computers through wearable or implanted electrodes and may afford a range of applications that enhance cognitive functions or joint outputs of minds coupled with machines.  91, 92  2.3. Behavioral Strategies. Although not commonly recognized as such by the public, cognitive enhancers with the most wide use and longest history are probably behavioral strategies: a rapidly growing body of evidence shows that everyday activities such as sleep  93  or physical exercise  94−96  improve cognitive functioning. Also well-established cultural activities such as musical training,  97, 98  dancing, 99 or learning a second language  100  have been demonstrated to enhance cognition beyond the specifically trained skills. In addition to these natural and cultural standard activities, several behavioral strategies have been developed to enhance certain brain functions intentionally. Two strategies that reach back to ancient times are mnemonic techniques to enhance learning and memory  101, 102  and meditation training to enhance attention processes and mindfulness.  103, 104  In contrast, commercial video games  105, 106  and customized computer trainings  107  represent historically very recent developments that are targeted to enhance specific cognitive capacities and skills. In contrast to several years of enthusiasm and widespread commercial application, however, more recent controlled studies and meta-analyses have shed some doubt on the efficacy of computerized brain training programs,  108  particularly criticizing claims of "far transfer" of training gains to cognitive domains considerably different from the specifically trained skills.  109, 110  ACS Chemical Neuroscience Review
 COGNITIVE DOMAIN The human mind is not a monolithic entity, but consists of a broad variety of cognitive functions. Not surprisingly, no single cognitive enhancer augments every cognitive function. Instead, most cognitive enhancers have specific profiles regarding their efficacy for different cognitive domains. Memory is, e.g., strongly enhanced by mnemonic strategies, but not by meditation; attention, in turn, is strongly enhanced by meditation training, but not training in mnemonic strategies.  101, 103, 104  Sleep, in contrast, enhances both cognitive capacities.  111, 112  Some computerized cognitive trainings have been found to enhance memory, processing speed and visuospatial skills, but not executive functions or attention.  107  It is currently highly debated in how far specific training strategies exert transfer effects also to nontrained cognitive domains.  113  Different cognitive tasks require different optimal levels of receptor activation, thus requiring different doses of pharmacological enhancers targeting the respective neurotransmitter system depending on the cognitive domain targeted.  114  Of note, effects of pharmacological enhancement on different cognitive domains might even differ depending on the cognitive test battery used, illustrating the fragility of the respective effects.  115  Some interventions might even enhance one but impair another cognitive domain: Intranasal application of oxytocin has been shown to enhance social cognition and cognitive flexibility but impairs long-term memory.  116, 117  Methylphenidate improves the ability to resist distraction but impairs cognitive flexibility.  118  Computerized working memory training has been reported to enhance working memory, reasoning, and arithmetic abilities; however, it might deteriorate creativity.  119  Also for amphetamines and modafinil, potential impairments on creativity are discussed besides their enhancing effects on other domains.  120, 34  In contrast, alcohol might enhance creative processes while impairing most other cognitive functions.  29  The costs and benefits of a single cognitive enhancer might even change through slight changes in the application process: for example, electrical stimulation of posterior brain regions was found to facilitate numerical learning, whereas automaticity for the learned material was impaired. In contrast, stimulation on frontal brain regions impaired the learning process, whereas automaticity for the learned material was enhanced.  121  Brain stimulation has thus been suggested to be a zero-sum game, with costs in some cognitive functions always being paid for gains in others.  122  This implies that enhancement may have to be tuned to the task at hand, in order to focus on the currently most important cognitive demands.
 PERSONAL FACTORS The efficacy of cognitive enhancers does not only differ for different cognitive domains, but also for different users. An important factor in this regard are the cognitive skills of the individual prior to the enhancement intervention. Many pharmaceuticals, including amphetamine,  123  modafinil, and methylphenidate,  124  work mainly in individuals with low baseline performance. In some cases, even impairments in individuals with higher performance at baseline have been reported, e.g., in the case of amphetamine,  125  methylphenidate,  124  nicotine,  27  or acute choline supplementation.  126  The phenomenon of enhanced cognition in individuals with low baseline performance and impairments in those with high baseline performance can be explained by the classical inverted U-model,  127, 128  where performance is optimal at intermediate levels of the targeted neurochemicals and impaired at levels that are either too low or too high.  129−131  For some drugs such as methylphenidate, enhancement dependency on the baseline might even differ between cognitive functions, with performance in specific tasks being improved in low, while impaired in high, performers,  124  but showing the opposite pattern for other tasks.  132  The baseline-dependency of cognitive enhancement is not restricted to pharmaceuticals: also in the case of video games,  133  cognitive training,  134  or brain stimulation,  135, 136  individuals starting at a lower baseline performance benefit more than those with an already high performance at baseline. In contrast, sleep appears to improve memory particularly in subjects with a higher baseline performance of memory,  137  working memory  138  or intelligence.  139  Also mnemonic training appears to work particularly well in individuals with a higher cognitive baseline performance.  140  This has been interpreted in terms of an amplification model, in which high baseline performance and cognitive enhancement interventions show synergistic effects.  141  Cognitive enhancers can also affect individuals differently depending on basic biological, psychological, or social factors. For example, effects of training interventions on selective attention can depend on the genotype of the trainee;  142  effects of methylphenidate on creativity can depend on personality characteristics;  143  the cognition enhancing effects of sleep  144  or video games  145  are modulated by gender. In turn, such modulations of enhancement effects might reduce existing differences in cognitive profiles, as seen, e.g., in action video game training, that have the potential to eliminate gender differences in spatial attention and decrease the gender disparity in mental rotation ability.  146  Also the hormonal status of subjects affects how strongly they profit, e.g., from sleep  144−148  or brain stimulation.  149  Caffeine enhances working memory particularly in extraverted individuals,  150  and memory enhancement through sleep  151  or mnemonic training  140  has been reported to depend on the age of subjects. Health status affects how much users benefit from different kinds of cognitive enhancers, including pharmaceuticals, 3 mnemonics,  152  or sleep.  153−156  Finally, also socioenvironmental factors such as social resources, parental occupation, or family composition can modulate cognitive enhancement interventions, e.g., with cognitive training programs.  157
 TIME SCALE Interventions for cognitive enhancement differ in the specific time scale they require to achieve their aims. The prototypical "smart pill" discussed in popular accounts of cognitive enhancement needs practically no preparation time, exerts its effects within seconds or minutes, and lasts for several hours. While this is close to reality in the case of some pharmacological enhancers, the temporal pattern of most other enhancement strategies differs strongly from these time scales. In particular, the time needed for application and the duration of their effects markedly varies between enhancement interventions. Most pharmacological enhancers can be applied quickly and without further preparation; however, some drugs such as acetylcholinesterase inhibitors or memantine are thought to require longer periods of intake to be effective.  33  Also some nutritional enhancers such as glucose and caffeine exert their effects rather quickly, whereas other nutritional supplements have to be taken over extended periods to show an impact on cognition.  158, 18  Obviously, behavioral strategies like sleep,
 ACS Chemical Neuroscience Review exercise, video games or mnemonic training need hours or weeks to robustly enhance cognition. Some effects of meditation might even take years of training.  159  For brain stimulation methods, both immediate effects of acute stimulation, but also more delayed effects after repeated stimulation haven been observed.  55, 69  Technological gadgets or implants need some preparation to be installed and accommodated to, however then exert their cognition augmenting effects on demand. Enhancing effects of most quickly acting pharmacological or nutritional cognitive enhancers also wear off rather rapidly. In contrast to such transient effects, interventions such as brain stimulation,  160, 161, 57  sleep, 162 mnemonic strategies  163  or genetic modifications  46  have the potential for long-term up to chronic enhancement. However, in the latter case, the reversibility of the effects (and side effects) of an enhancement intervention might be a further aspect to be considered. Interventions can also differ regarding the time point of application relative to the situation when enhanced cognitive performance is needed. For example, application of stress hormones such as cortisol or adrenaline before or after memory encoding enhances memory, whereas application before retrieval impairs memory; 41 benzodiazepines impair memory when given before and enhance memory when given after encoding;  164  in contrast, caffeine before learning enhances memory under certain conditions but might impair memory when consumed afterward.  165, 9  Mnemonic strategies on the other hand work solely when taught/applied before/during encoding, but can hardly be applied afterward. Finally, some interventions can also influence the timing of cognitive performance itself: stimulants such as methylphenidate, modafinil, and caffeine might increase the time subjects take to perform a given task, with impairing effects under time pressure and potentially enhancing effects in the absence of temporal constraints.  166
 SIDE EFFECTS The pharmaceutical platitude that there is no effect without side effects holds true also for many nonpharmacological enhancement interventions. It appears obvious that cognitive enhancers differ in the severity and form of side effects: prima facie, deep brain stimulation or implants have higher risks for side effects than sleep or cognitive training. However, also more indirect enhancement strategies such as neurofeedback potentially bear the risk of side effects up to inducing epileptiform activity,  167  and even gentle intervention such as meditation training might exert negative effects on specific cognitive domains: a negative relationship between mindfulness and implicit learning  168, 169  and an increased susceptibility to false memory formation after mindfulness meditation  170  have been observed. Here, the intended training goal of nonjudgemental mindfulness opposes tasks where either a more critical or automatic mindset was needed. Further examples of side effects intrinsically associated with the enhancement goal are trade-offs between stability vs flexible updating of memory systems:  129  memories can also become "too stable" due to a memory enhancement intervention, as observed, e.g., for the anti-obesity drug rimonabant.  171  It has been suggested to differentiate enhancement strategies according to their level of invasiveness.  172, 173  However, while invasiveness has a more or less definite meaning in its original medical context, physically breaching the skin or entering the body deeply through an external orifice,  174  it is difficult to determine the level of invasiveness in the context of cognitive enhancement. Both nutritional supplements and pharmaceuticals enter the body, and thus could be considered invasive in a narrow medical sense, as might be certain forms of physical exercise due to the risk of bruises or scratches as common, e.g., in martial arts or a hike through the woods. Brain stimulation that does not break the skin would, by contrast, be classified as noninvasive. This taxonomy can be disputed for good reasons.  175  Besides known risks of these stimulation methods such as scalp burns from tDCS or seizures from TMS, the "known unknowns" have been suggested to pose potentially even greater risks: potential build-up effects across multiple sessions or in sensitive nontarget areas.  176  Of note, only few neuroscientists use brain stimulation on themselves for cognitive enhancement.  176  Given the still unclear risks and side effects of do-it-yourself brain stimulation use, it has been proposed to extend existing medical device legislation to cover also nonpharmacological and in particular physically acting cognitive enhancement devices.  177, 178  In contrast to strict medical definitions, the more intuitively assessed level of invasiveness of an intervention often seems to depend on familiarity and cultural traditions. This leads to the Western attitude according to which changing one's diet or performing exercise appears less invasive than taking pharmaceuticals or applying brain stimulation, independent of their actual effects on health. Related to the time scale dimension, side effects of short-vs long-term use of cognitive enhancers can be differentiated. For example, while side effects for the acute use of methylphenidate include increased heart rate, headache, anxiety, nervousness, dizziness, drowsiness, and insomnia, in the case of long-term use side effects such as abnormal prefrontal brain function and impaired plasticity have been reported.  179, 180  Also addiction is a well-known side effect for the long-term use of pharmacological enhancers, which is particularly detrimental to the aim of enhancement if combined with tolerance effects such that larger doses are needed to achieve the same effect (or prevent impairing withdrawal effects). Also behavioral addictions have been observed, e.g., physical exercise  181  or the use of technological gadgets.  182  A somewhat nonobvious negative effect of some cognitive enhancers is their illusional efficacy: users sometimes believe their performance to be enhanced by amphetamine in absence of any verified and objectively visible enhancing effects, even if administered in a double blind manner.  183, 184, 38  This is particularly counterproductive in cases of already highfunctioning individuals whose cognitive capabilities might be impaired rather than enhanced by amphetamine.  125, 184  Also for caffeine, under certain conditions higher subjectively perceived mental energy in the absence of objectively enhancing effects have been observed.  185  The often subtle effects of enhancers can be hidden or amplified by placebo effects.
 AVAILABILITY Cognitive enhancers differ in at least three aspects of availability: legal status, cost, and application time. In terms of legal regulation, different enhancement methods are regulated by sometimes drastically varying frameworks. Pharmaceuticals, for instance, are regulated by strict international control regimes that effectively prohibit nontherapeutic uses or by more lenient domestic drug laws. Brain-stimulation methods, by contrast, fall under medical device regulations, pertaining to basic safety standards in terms but not proscribing the uses to which they might be put.  177, 178  Behavioral strategies are usually not regulated at all. The regulatory landscape is thus vast and
 ACS Chemical Neuroscience Review possibly incoherent (for a review, see  ref 186) . Besides practical hurdles to the acquisition of illicit drugs for cognitive enhancement, the legal status appears to affect the motivation of users to decide which cognitive enhancers to take.  166  A common ethical argument in the enhancement debate concerns distributive justice: also legally available enhancers come with cost barriers, restricting individuals of low socioeconomic status from access.  187  A main factor in the costs of cognitive enhancers is their patentability, which is not restricted to pharmaceuticals.  188  However, in particular, behavioral enhancement strategies are typically not subject to patentability or other cost-driving factors: sleep, exercise, meditation, or training in mnemonic strategies are largely free of cost and, thus, in contrast to pharmaceuticals or technological strategies, are available independent from the financial background of the user. On the other hand, these behavioral strategies require some time and effort: the 24/7 working manager as the clichéuser of cognitive enhancement drugs might have the financial means to afford quickly taking his expensive smart pill between two meetings, but might be unable or unwilling to spend extended periods of time with sleep, meditation, or mnemonic training.
 SOCIAL ACCEPTANCE Largely independent from their specific enhancing effects on different cognitive capacities, social acceptance of cognitive enhancement interventions differs strongly depending on traditions, their perceived naturalness, and the perceived directness of their mode of action. Enhancement interventions with a tradition of thousands of years such as meditation and nutrition are typically much better accepted than many currently debated enhancement strategies such as brain stimulation and pharmaceuticals.  189  Also more "natural" interventions such as sleep or exercise are seen in a more positive light compared to technological innovations.  190  Moreover, in how far the mode of action is perceived as psychologically mediated or more biologically direct, affecting the brain indirectly through the senses or more directly through the cranium or metabolism, often plays a role for their social acceptance: if an enhancement intervention such as intense cognitive or physical training requires extended efforts or is seen as a quick and effortless shortcut to the same goal as in the case of smart pills or brain stimulation touches different intuitions about human virtues and is thus valuated differently. Even though views based on such purely intuitive aspects of tradition, naturalness, or directness often rely on cognitive biases rather than rational argument,  191  a negative social perception for whatever reason might generate indirect psychological costs for users, which in turn might influence also rational evaluations of the respective enhancement intervention.  192  Accordingly, one of the central points in the ethical controversy revolves around the question of whether enhancement strategies only relevantly differ with respect to their outcomes, i.e., their benefits and side effects,  193  or also with respect to their mode of operation.  194  Some argue that the relevant ethical distinction runs along the lines of enhancements that are active, in the sense of requiring participation, and those that work on persons more passively.  195  Not surprisingly, different views on cognitive enhancement prevail in different (sub)cultures, with, e.g., a more positive view on enhancement interventions in Asia  196  or in younger populations.  197  Empirical studies on attitudes toward cognitive enhancement interventions found medical safety, coercion, and fairness the most common concerns, with nonusers displaying more concerns regarding medical safety and fairness than users.  198  Sometimes readily available substances for cognitive enhancement such as caffeine, energy drinks, or herbal drugs are dubbed "soft enhancers"; 199 however, considering that prohibition of substances is not only based on their potential harm, but also on historical circumstances, this differentiation between soft and hard enhancers appears questionable. A further aspect that determines the social acceptance of cognitive enhancement is the aim of the given intervention. Taken by face value, the term cognitive enhancement denotes any action or intervention that improves cognitive capacities, independent from the specific aim of this improvement. The use of the term in the empirical, philosophical, and sociopolitical literature, however, varies with regard of the specific aim of enhancement interventions: people appear to be more tolerant toward enhancement of traits considered less fundamental to self-identity,  200  and also more tolerant toward enhancement in individuals with cognitive impairments or low performance baselines compared to enhancement of normal or high achievers.  201, 202  At least four different aims can be identified, each leading to different research strategies and different ethical evaluations of existing or potential enhancement strategies.  203  The least problematic concept of cognitive enhancement targets those everyday medical or psychological interventions that are meant to treat pathological deficiencies. Closely related are those cognitive enhancement interventions that aim to prevent or attenuate cognitive decline that is associated also with healthy aging.  204  Slightly less accepted appear to be those enhancement strategies that aim to improve cognition in completely healthy individuals, but still clearly stay within the normal limits of cognition. The probably most widely used and ethically most controversial concept of cognitive enhancement aims at the augmentation of cognitive capacities beyond normal function, as is represented in the clichéof high-functioning students or managers trying to further improve their performance by taking smart pills. Besides these differentiations between enhancement of impaired vs healthy cognition, another difference in the aims of cognitive enhancement touches the ultimate deed of the enhancement intervention: due to the central role of cognitive capacities in defining humans as a species, it is tempting to consider the improvement of these defining human capacities as a value in itself. However, most philosophical or religious approaches do not center on objective cognitive performance markers, but propose values only indirectly related to cognitive performance such as living a happier or more meaningful life in general. In this light, human enhancement in more general terms does not need to aim for individual cognitive or neural processes, but can also be achieved by sociopolitical reforms targeted at the population level.  205, 206  9. CONCLUSIONS Cognitive enhancement clearly is a multidimensional endeavor. However, not every dimension is important for every theoretical or empirical research question. For example, many empirical researchers of cognitive enhancement are primarily interested in the understanding of the neurobiological and psychological mechanisms underlying cognitive functions.  207  For this aim, the availability and social acceptance dimensions are largely irrelevant. In contrast, many theorists are interested in the social and ethical implications of cognitive enhancement,  208  where these dimensions might be of prime importance. Also side effects and temporal factors might be of secondary importance
 ACS Chemical Neuroscience Review to empirical researchers with an interest in the neural mechanisms of certain cognitive processes, whereas these would be highly relevant for users who ponder the question which cognitive enhancement strategy to choose for a certain aim. When comparing different cognitive enhancement strategies, different dimensions might thus be differently weighed or completely ignored, depending on the aim of the comparison. Up until now, direct comparisons between cognitive enhancement strategies with radically different modes of actions have rarely been made (but see, e.g., ref  165) , and more comprehensive comparisons across dimensions might be difficult: practical issues of information availability from the different dimensions aside, interventions typically differ on different dimensions and are thus difficult to compare globally. In addition, multiple interactions between different enhancers exist, which further complicates the situation. Interactions have been reported, e.g., for glucose and caffeine,  209  diet and exercise,  210  exercise and working memory training,  211  video games and sleep,  212  video games and brain stimulation,  213  exercise and brain stimulation,  214  and brain stimulation and sleep.  215, 216  Also different dimensions discussed here can interact in multiple ways, as, e.g., computerized cognitive training can differentially enhance different cognitive processes depending on personal factors such as age;  217  and social acceptance of different enhancement strategies depends on both the baseline performance of users and the cognitive domain targeted.  200, 201  Despiteor because ofthese complexities, in our opinion, both theoretical discussions and empirical research would strongly benefit from a more differentiated approach. Specific research questions might require the emphasis on some dimensions of cognitive enhancement over others, and for some research questions some dimensions might be entirely irrelevant. Nevertheless, keeping in mind that cognitive enhancement is not a monolithic phenomenon will help to solve and avoid a number of confusions and disagreements that are still present in the public debate on cognitive enhancement. This is an open access article published under a Creative Commons Non-Commercial No Derivative Works (CC-BY-NC-ND) Attribution License, which permits copying and redistribution of the article, and creation of adaptations, all for non-commercial purposes. Downloaded via 205.240.4.254 on March 24, 2022 at 02:39:01 (UTC).
 Figure 1 . 1 Figure 1. Cognitive enhancement interventions differ across several interdependent dimensions.
 Figure 2 . 2 Figure 2. Cognitive enhancement interventions different in their mode of actions.

Abstract: In an increasingly complex information society, demands for cognitive functioning are growing steadily. In recent years, numerous strategies to augment brain function have been proposed. Evidence for their efficacy (or lack thereof) and side effects has prompted discussions about ethical, societal, and medical implications. In the public debate, cognitive enhancement is often seen as a monolithic phenomenon. On a closer look, however, cognitive enhancement turns out to be a multifaceted concept: There is not one cognitive enhancer that augments brain function per se, but a great variety of interventions that can be clustered into biochemical, physical, and behavioral enhancement strategies. These cognitive enhancers differ in their mode of action, the cognitive domain they target, the time scale they work on, their availability and side effects, and how they differentially affect different groups of subjects. Here we disentangle the dimensions of cognitive enhancement, review prominent examples of cognitive enhancers that differ across these dimensions, and thereby provide a framework for both theoretical discussions and empirical research.