Dorsa Sadigh
S Shankar
Sastry Sanjit
A Seshia
Anca Dragan

Information Gathering Actions over Human Internal State

I. introduction Imagine driving on the highway. Another driver is in the lane next to you, and you need to switch lanes. Some drivers are aggressive and they will never brake to let you in. Others are more defensive and would gladly make space for you. You don't know what kind of driver this is, so you decide to gently nudge in towards the other lane to test their reaction. At an intersection, you might nudge in to test if the other driver is distracted and they might just let you go through (Fig.  1  bottom left). Our goal in this work is to give robots the capability to plan such actions as well. In general, human behavior is affected by internal states that a robot would not have direct access to: intentions, goals, preferences, objectives, driving style, etc. Work in robotics and perception has focused thus far on estimating these internal states by providing algorithms with observations of humans acting, be it intent prediction  [23] ,  [13] ,  [6] ,  [3] ,  [4] ,  [16] , Inverse Reinforcement Learning  [1] ,  [12] ,  [15] ,  [22] ,  [18] , driver style prediction  [11] , affective state prediction  [10] , or activity recognition  [20] . Human state estimation has also been studied in the context of human-robot interaction tasks. Here, the robot's reward function depends (directly or indirectly) on the human internal state, e.g., on whether the robot is able to adapt to the human's plan or preferences. Work in assistive teleoperation or in human assistance has cast this problem as a Partially Observable Markov Decision Process, in which the robot does observe the physical state of the world but not the human internal state -that it has to estimate from human actions. Because POMDP solvers are not computationally efficient, the solutions proposed thus far use the current estimate Fig.  1 : We enable robots to generate actions that actively probe humans in order to find out their internal state. We apply this to autonomous driving. In this example, the robot car (yellow) decides to inch forward in order to test whether the human driver (white) is attentive. The robot expects drastically different reactions to this action (bottom right shows attentive driver reaction in light orange, and distracted driver reaction in dark orange). We conduct a user study in which we let drivers pay attention or distract them with cellphones in order to put this state estimation algorithm to the test. of the internal state to plan (either using the most likely estimate, or the entire current belief), and adjust this estimate at every step  [8] ,  [7] ,  [5] . Although efficient, these approximations sacrifice an important aspect of POMDPs: the ability to actively gather information. Our key insight is that robots can leverage their own actions to help estimation of human internal state. Rather than relying on passive observations, robots can actually account for the fact that humans will react to their actions: they can use this knowledge to select actions that will trigger human reactions which in turn will clarify the internal state. We make two contributions: An Algorithm for Active Information Gathering over Human Internal State. We introduce an algorithm for planning robot actions that have high expected information gain. Our algorithm uses a rewardmaximization model of how humans plan their actions in response to those of the robot's  [19] , and leverages the fact that different human internal states will lead to different human reactions to speed up estimation. Fig.  1  shows an example of the anticipated difference in reaction between a distracted and an attentive driver. Application to Driver Style Estimation. We apply our algorithm to estimating a human driver's style during the interaction of an autonomous vehicle with a human-driven vehicle. Results in simulation as well as from a user study suggest that our algorithm's ability to leverage robot actions for estimation leads to significantly higher accuracy in identifying the correct human internal state. The autonomous car plans actions like inching forward at an intersection (Fig.  1 ), nudging into another car's lane, or braking slightly in front of a human-driven car, all to estimate whether the human driver is attentive. Overall, we are excited to have taken a step towards giving robots the ability to actively probe end-users through their actions in order to better estimate their goals, preferences, styles, and so on. Even though we chose driving as our application domain in this paper, our algorithm is general across different domains and types of human internal state. We anticipate that applying it in the context of human goal inference during shared autonomy, for instance, will lead to the robot purposefully committing to a particular goal in order to trigger a reaction from the user, either positive or negative, in order to clarify the desired goal. Of course, further work is needed in order to evaluate how acceptant end-users are of different kinds of such probing actions.
 II. Information Gathering Actions We start with a general formulation of the problem of a robot needing to maximize its reward by acting in an environment where a human is also acting. The human is choosing its actions in a manner that is responsive to the robot's actions, and also influenced by some internal variables. While most methods that have addressed such problems have proposed approximations based on passively estimating the internal variable and exploiting that estimate, here we propose a method for active information gathering that enables the robot to purposefully take actions that probe the human. Finally, we discuss our implementation in practice, which trades off between exploration and exploitation.
 A. General Formulation We define a human-robot system in which the human's actions depend on some human internal state ϕ that the robot does not directly observe. In a driving scenario, ϕ might correspond to driving style: aggressive or timid, attentive or distracted. In collaborative manipulation scenarios, ϕ might correspond to the human's current goal, or their preference about the task. We let x ∈ X be a continuous physical state of our system. For our running example of autonomous cars, this includes position, velocity and heading of the autonomous and human driven vehicles. Let ϕ ∈ Φ be the hidden variable, e.g., human driver's driving style. We assume the robot observes the current physical state x t , but not the human internal state ϕ. The robot and human can apply continuous controls u R and u H . The dynamics of the system evolves as robot's and human's control inputs arrive at each step: x t+1 = f H ( f R (x t , u t R ), u t H ). (1) Here, f R and f H represent how the actions of the robot and of the human respectively affect the dynamics, and can be applied synchronously or asynchronously. We assume that while x changes via (1) based on the human and robot actions, ϕ does not. For instance, we assume that the human maintains their preferences or driving style throughout the interaction. The robot's reward function in the task depends on the current state, the robot's action, as well as the action that the human takes at that step in response, r R (x t , u t R , u t H ). If the robot has access to the human's policy π H (x, u R , ϕ), maximizing robot reward can be modeled as a POMDP  [9]  with states (x, ϕ), actions u R , and reward r R . In this POMDP, the dynamics model can be computed directly from (1) with u t H = π H (x t , u t R , ϕ). The human's actions can serve as observations of ϕ via some P(u H |x, u R ). In Sec. II-C we introduce a model for both π H and P(u H |x, u R , ϕ) based on the assumption that the human is maximizing their own reward function. If we were able to solve the POMDP, the robot would estimate ϕ based on the human's actions, and optimally trade off between exploiting its current belief over ϕ, and actively taking information gathering actions meant to cause human reactions that give the robot a better estimate of the hidden variable ϕ. Because POMDPs cannot be solved tractably, several approximations have been proposed for similar problem formulations  [8] ,  [11] ,  [7] . These approximations are passively estimating the human internal state, and exploiting the belief to plan robot actions.  1  In this work, we take the opposite approach: we focus explicitly on active information gathering. We enable the robot to decide to actively probe the person to get a better estimate of ϕ. Our method can be leveraged in conjunction with exploitation methods, or be used alone when human state estimation is robot's primary objective.
 B. Reduction to Information Gathering At every step, the robot can update its belief over ϕ via: b t+1 (ϕ) ∝ b t (ϕ) • P(u H |x t , u R , ϕ). ( 2 ) To explicitly focus on taking actions to estimate ϕ, we redefine the robot's reward function to capture the information gain at every step: r R (x t , u R , u H ) = H(b t ) − H(b t+1 ) (3) with H(b) being the entropy over the belief: H(b) = − ∑ ϕ b(ϕ) log(b(ϕ)) ∑ ϕ b(ϕ) . ( 4 ) Optimizing expected reward now entails reasoning about the effects that the robot actions will have on what observations the robot will get, i.e., the actions that the human will take in response, and how useful these observations will be in shattering ambiguity about ϕ.
 C. Solution: Human Model & Model Predictive Control We solve the information gathering planning problem via Model Predictive Control (MPC)  [14] . At every time step, we find the optimal actions of the robot u * R by maximizing the expected reward function over a finite horizon. Notation. Let x 0 be the state at the current time step, i.e., at the beginning of the horizon. u R = (u 0 R , . . . , u N−1 R ) be a finite sequence of the robot's continuous actions, and u H = (u 0 H , . . . , u N−1 H ) be a finite sequence of human's continuous actions. Further, let R R (x 0 , u R , u H ) denote the reward over the finite horizon, if the agents started in x 0 and executed u R and u H , which can be computed via the dynamics in  (1) . MPC Maximization Objective. At every time step, the robot is computing the best actions for the horizon: u * R = arg max u R E ϕ R R (x 0 , u R , u * ϕ H (x 0 , u R )) (5) where u * ϕ H (x 0 , u R ) corresponds to the actions the human would take from state x 0 over the horizon of N steps if the robot executed actions u R . Here, the expectation is taken over the current belief over ϕ, b 0 . Simplifying (5) using the definition of reward from (3), we get: u * R = arg max u R E ϕ H(b 0 ) − H(b N ) , (6) u * R = arg max u R E ϕ − H(b N ) , (7) where the expectation remains with respect to b 0 . Human Model. We assume that the human maximizes their own reward function at every step. We let r ϕ H (x t , u t R , u t H ) represent human's reward function at time t, which is parametrized by the human internal state ϕ. Then, the sum of human rewards over horizon N is: R ϕ H (x 0 , u R , u H ) = N−1 ∑ t=0 r ϕ H (x t , u t R , u t H ) (8) Building on our previous work  [19] , which showed how the robot can plan using such a reward function when there are no hidden variables, we compute u * ϕ H (x 0 , u R ) through an approximation. We model the human as having access to u R a priori, and compute the finite horizon human actions that maximize the human's reward: u * ϕ H (x 0 , u R ) = arg max u H R ϕ H (x 0 , u R , u H ) (9) One can find r ϕ H through Inverse Reinforcement Learning (IRL)  [1] ,  [12] ,  [15] ,  [22] , by getting demonstrations of human behavior associated with a direct measurement of ϕ. The robot can use this reward in the dynamics model in order to compute human actions via  (9) . To update the belief b and compute expected reward in  (7) , we still need an observation model. We assume that actions with lower reward are exponentially less likely, building on the principle of maximum entropy  [22] : P(u H |x, u R , ϕ) ∝ exp(ϕ H (x, u R , u H )) (10) Optimization Procedure. To solve (5) (or equivalently (  7 )), we use a gradient descent optimization method, L-BFGS, designed for unconstrained nonlinear problems  [2] . Therefore, we would like to find the gradient of the objective in equation (  5 ) with respect to u R . Since the objective is the expectation of R R , we can reformulate this gradient as: ∂E ϕ R R (x 0 , u R , u * ϕ H (x 0 , u R )) ∂u R = ∑ ϕ ∂R R (x 0 , u R , u * ϕ H (x 0 , u R )) ∂u R • b 0 (ϕ) (11) Then, we only need to find ∂R R ∂u R , which is equivalent to: ∂R R (x 0 , u R , u * ϕ H (x 0 , u R )) ∂u R = ∂R R (x 0 , u R , u H ) ∂u H ∂u * ϕ H ∂u R + ∂R R (x 0 , u R , u H ) ∂u R | u H =u * ϕ H (x 0 ,u R ) (12 ) Because R R , as indicated by  (7) , simplifies to the negative entropy of the updated belief, we can compute both ∂R R (x 0 ,u R ,u H ) ∂u H and ∂R R (x 0 ,u R ,u H ) ∂u R | u H =u * ϕ H (x 0 ,u R ) symbolically. This leaves ∂u * ϕ H ∂u R . We use the fact that the gradient of R H will evaluate to zero at u * ϕ H : ∂R H ∂u H (x 0 , u R , u * ϕ H (x 0 , u R )) = 0 (13) Now, differentiating this expression with respect to u R will result in: ∂ 2 R H ∂u 2 H ∂u * ϕ H ∂u R + ∂ 2 R H ∂u H ∂u R ∂u R ∂u R = 0 (14) Then, solving for ∂u * ϕ H ∂u R enables us to find the following symbolic expression: ∂u * ϕ H ∂u R = [− ∂ 2 R H ∂u H ∂u R ][ ∂ 2 R H ∂u 2 H ] −1 . ( 15 ) This expression allows finding a symbolic expression for the gradient in equation  (11) .
 D. Explore-Exploit Trade-Off In practice, we use information gathering in conjunction with exploitation. We do not solely optimize the reward from Sec. II-B, but optimize it in conjunction with the robot's actual reward function assuming the current estimate of ϕ: r R (x t , u R , u H ) = H(b t ) − H(b t+1 ) ( 16 ) + λ • r goal (x t , u R , u H , b t ) At the very least, we do this as a measure of safety, e.g., we want an autonomous car to keep avoiding collisions even when it is actively probing a human driver to test their reactions. We choose λ experimentally, though existing techniques that can better adapt λ over time  [21] . Despite optimizing this trade-off, we do not claim that our method as-is can better solve the general POMDP formulation from Sec. II-A: only that it can be used to get better estimates of human internal state. The next sections test this in simulation and in practice, in a user study, and future work will look at how to leverage this ability to better solve human-robot interaction problems.
 III. Simulation Results In this section, we show simulation results that use the method from the previous section to estimate human driver type in the interaction between an autonomous vehicle and a human-driven vehicle. In this section, we consider three different autonomous driving scenarios. In these scenarios, the human is either distracted or attentive during different driving experiments. The scenarios are shown in Fig.  2 , where the yellow car is the autonomous vehicle, and the white car is the human driven vehicle. Our goal is to plan to actively estimate the human's driving style in each one of these scenarios, by using the robot's actions.
 A. Attentive vs. Distracted Human Driver Models Our technique requires reward functions r ϕ H that model the human behavior for a particular internal state ϕ. We obtain a generic driver model via Continuous Inverse Optimal Control with Locally Optimal Examples  [12]  from demonstrated trajectories in a driving simulator in an environment with multiple autonomous cars, which followed precomputed routes. We parametrize the human reward function as a linear combination of features, and learn weights on the features. We use various features including features for bounds on the control inputs, features that keep the vehicles within the road boundaries and close to the center of their lanes. Further, we use quadratic functions of speed to capture reaching the goal, and Gaussians around other vehicles on the road to enforce collision avoidance as part of the feature set. We then adjust the learned weights to model attentive vs. distractive drivers. Specifically, we modify the weights of the collision avoidance features, so the distracted human model has less weight for these features. Therefore, the distracted driver is more likely to collide with the other cars while the attentive driver has high weights for the collision avoidance feature.
 B. Manipulated Factors We manipulate the reward function that the robot is optimizing. In the passive condition, the robot optimizes a simple reward function for collision avoidance based on the current belief estimate. It then updates this belief passively, by observing the outcomes of its actions at every time step. In the active condition, the robot trades off between this reward function and the Information Gain from (3) in order to explore the human's driving style. We also manipulate the human internal state to be attentive or distracted. The human is simulated to follow the ideal model of reward maximization for our two rewards.
 C. Driving Simulator We use a simple point-mass model for the dynamics of the vehicle, where x = x y θ v is the state of the vehicle. Here, x and y are the coordinates of the vehicle, θ is the heading, and v is the speed. Each vehicle has two control inputs u = u 1 u 2 , where u 1 is the steering input, and u 2 is acceleration. Further, we let α be a friction coefficient. Then, the dynamics of each vehicle is formalized as: [ ẋ ẏ θ v] = [v • cos(θ) v • sin(θ) v • u 1 u 2 − α • v]. (17)
 D. Scenarios and Qualitative Results Scenario 1: Nudging In to Explore on a Highway. In this scenario, we show an autonomous vehicle actively exploring the human's driving style in a highway driving setting. We contrast the two conditions in Fig.  2 (a). In the passive condition, the autonomous car drives on its own lane without interfering with the human throughout the experiment, and updates its belief based on passive observations gathered from the human car. However, in the active condition, the autonomous car actively probes the human by nudging into her lane in order to infer her driving style. An attentive human significantly slows down (timid driver) or speeds up (aggressive driver) to avoid the vehicle, while a distracted In this scenario, we consider the two vehicles at an intersection, where the autonomous car actively tries to explore human's driving style by nudging into the intersection. The initial conditions of the vehicles are shown in Fig.  2(c ). In the passive condition, the autonomous car stays at its position without probing the human, and only optimizes for collision avoidance. This provides limited observations from the human car resulting in a low confidence belief distribution. Autonomous Vehicle Human Driven Vehicle Passive Estimation Active Info Gathering In the active condition, the autonomous car nudges into the intersection to probe the driving style of the human. An attentive human would slow down to stay safe at the intersection while a distracted human will not slow down.
 E. Quantitative Results Throughout the remainder of the paper, we use a common color scheme to plot results for our experimental conditions. We show this common scheme in Fig.  3 : darker colors (black and red) correspond to attentive humans, and lighter colors (gray and orange) correspond to distracted humans. Further, the shades of orange correspond to active information gathering, while the shades of gray indicate passive information gathering. We also use solid lines for real users, and dotted lines for scenarios with an ideal user model learned through inverse reinforcement learning. This table is representative for the legends of Fig.  4 , Fig.  ? ?, Fig.  5 , and Fig.  6 . Fig.  4  plots, using dotted lines, the beliefs over time for the attentive (left) and distracted (right) conditions, comparing in each the passive (dotted black and gray respectively) with the active method (dotted dark orange and light orange respectively). In every situation, the active method achieves a more accurate belief (higher values for attentive on the left, when the true ϕ is attentive, and lower values on the right, when the true ϕ is distracted). In fact, passive estimation sometimes incorrectly classifies drivers as attentive when they are distracted and vice-versa. The same figure also shows (in solid lines) results from our user study of what happens when the robot no longer interacts with an ideal model. We discuss these in the next section. Fig.  5  and Fig.  6  plot the corresponding robot and human trajectories for each scenario. The important takeaway from these figures is that there tends to be a larger gap between attentive and distracted human trajectories in the active condition (orange shades) than in the passive condition (gray shades), especially in scenarios 2 and 3. It is this difference that helps the robot better estimate ϕ: the robot in the active condition is purposefully choosing actions that will lead to large differences in human reactions, in order to more easily determine the human driving style.
 IV. User Study In the previous section, we explored planning for an autonomous vehicle that actively probes a human's driving style, by braking or nudging in and expecting to cause reactions from the human driver that would be different depending on their style. We showed that active exploration does significantly better at distinguishing between attentive and distracted drivers using simulated (ideal) models of drivers. Here, we show the results of a user study that evaluates this active exploration for attentive and distracted human drivers.
 A. Experimental Design We use the same three scenarios discussed in the previous section. Manipulated Factors. We manipulated the same two factors as in our simulation experiments: the reward function that the robot is optimizing (whether it is optimizing its reward through passive state estimation, or whether it is trading off with active information gathering), and the human internal state (whether the user is attentive or distracted). We asked our users to pay attention to the road and avoid collisions for the attentive case, and asked our users to play a game on a mobile phone during the distracted driving experiments. Dependent Measure. We measured the probability that the robot assigned along the way to the human internal state. Hypothesis. The active condition will lead to more accurate human internal state estimation, regardless of the true human internal state. Subject Allocation. We recruited 8 participants (2 female, 6 male) in the age range of 21-26 years old. All participants owned a valid driver license and had at least 2 years of driving experience. We ran the experiments using a 2D driving simulator with the steering input and acceleration input provided through a steering wheel and a pedals as shown in Fig.  1 . We used a within-subject experiment design with counterbalanced ordering of the four conditions.
 B. Analysis We ran a factorial repeated-measures ANOVA on the probability assigned to "attentive", using reward (active vs passive) and human internal state (attentive vs distracted) as factors, and time and scenario as covariates. As a manipulation check, attentive drivers had significantly higher estimated probability of "attentive" associated than distracted drivers (.66 vs .34, F = 3080.3, p < .0001). More importantly, there was a signifiant interaction effect between the factors (F = 1444.8, p < .000). We ran a post-hoc analysis with Tukey HSD corrections for multiple comparisons, which showed all four conditions to be significantly different from each other, all contrasts with p < .0001. In particular, the active information gathering did end up with higher probability mass on "attentive" than the passive estimation for the attentive users, and lower probability mass for the distracted user. This supports our hypothesis that our method works, and active information gathering is better at identifying the correct state. Fig.  4  compares passive (grays and blacks) and active (light and dark oranges) across scenarios and for attentive (left) and distracted (right) users. It plots the probability of attentive over time, and the shaded regions correspond to standard error. From the first column, we can see that our algorithm in all cases detects human's attentiveness with much higher probably than the passive information gathering technique shown in black. From the second column, we see that our algorithm places significantly lower probability on attentiveness, which is correct because those users were distracted users. These are in line with the statistical analysis, with active information gathering doing a better job estimating the true human internal state. Fig.  5  plots the robot trajectories for the active information gathering setting. Similar to Fig.  4 , the solid lines are the mean of robot trajectories and the shaded regions show the standard error. We plot a representative dimension of the robot trajectory (like position or speed) for attentive (dark orange) or distracted (light orange) cases. The active robot probed the user, but ended up taking different actions when the user was attentive vs. distracted in order to maintain safety. For example, in Scenario 1, the trajectories show the robot is nudging into the human's lane, but the robot decides to move back to its own lane when the human drivers are distracted (light orange) in order to stay safe. In Scenario 2, the robot brakes in front of the human, but it brakes less when the human is distracted. Finally, in Scenario 3, the robot inches forward, but again it stops when if the human is distracted, and even backs up to make space for her. Fig.  6  plots the user trajectories for both active information gathering (first row) and passive information gathering (second row) conditions. We compare the reactions of distracted (light shades) and attentive (dark shades) users. There are large differences directly observable, with user reactions tending to indeed cluster according to their internal state. These differences are much smaller in the passive case (second row, where distracted is light gray and attentive is black). For example, in Scenario 1 and 2, the attentive users (dark orange) keep a larger distance to the car that nudges in front of them or brakes in front of them, while the distracted drivers (light orange) tend to keep a smaller distance. In Scenario 3, the attentive drivers tend to slow down and do not cross the intersection, when the robot actively inches forward. None of these behaviors can be detected clearly in the passive information gathering case (second row). This is the core advantage of active information gathering: the actions are purposefully selected by the robot such that users would behave drastically differently depending on their internal state, clarifying to the robot what this state actually is. Overall, these results support our simulation findings, that our algorithm performs better at estimating the true human internal state by leveraging purposeful information gathering actions.
 V. discussion Summary. In this paper, we formalized the problem of active information gathering between robot and human agents, where the robot plans to actively explore and gather information about the human's internal state by leveraging the effects of its actions on the human actions. The generated strategy for the robot actively probes the human by taking actions that impact the human's action in such a way that they reveal her internal state. The robot generates strategies for interaction that we would normally need to hand-craft, like inching forward at a 4-way stop. We evaluated our method in simulation and through a user study for various autonomous driving scenarios. Our results suggest that robots are indeed able to construct a more accurate belief over the human's driving style with active exploration than with passive estimation. Limitations and Future Work. Our work is limited in many ways. First, state estimation is not the end goal, and finding how to trade off exploration and exploitation is still a challenge. Second, our optimization is close to real-time, but higher computational efficiency is still needed. Further, our work relies on a model (reward function) of the human for each ϕ, which might be difficult to acquire, and might not be accurate. Thus far, we have assumed a static ϕ, but in reality ϕ might change over time (e.g. the human adapts her preferences), or might even be influenced by the robot (e.g. a defensive driver becomes more aggressive when the robot probes her). We also have not tested the users' acceptance of information gathering actions. Although these actions are useful, people might not always react positively to being probed. Last but not least, exploring safely will be of crucial importance.
 Conclusion. We are encouraged by the fact that robots can generate useful behavior for interaction autonomously, and are excited to explore informationgathering actions on human state further, including beyond autonomous driving scenarios. Passive Robot stops to R through Fig.  6 : The user trajectories for each scenario. The gap between attentive and distracted drivers' actions is clear in the active information gathering case (first row). 1 :Fig. 2 : 12 Fig.2: Our three scenarios, along with a comparison of robot plans for passive estimation (gray) vs active information gathering (orange). In the active condition, the robot is purposefully nudging in or braking to test human driver attentiveness. The color of the autonomous car in the initial state is yellow, but changes to either gray or orange in cases of passive and active information gathering respectively. driver might not realize the autonomous actions and maintain their velocity, getting closer to the autonomous vehicle. It is this difference in reactions that enables the robot to better estimate ϕ. Scenario 2: Braking to Explore on a Highway. In the second scenario, we show the driving style can be explored by the autonomous car probing the human driver behind it. The two vehicles start in the same lane as shown in Fig.2(b), where the autonomous car is in the front. In the passive condition, the autonomous car drives straight without exploring or enforcing any interactions with the human driven vehicle. In the active condition, the robot slows down to actively probe the human and find out her driving style. An attentive human would slow down and avoid collisions while a distracted human will have a harder time to keep safe distance between the two cars. Scenario 3: Nudging In to Explore at an Intersection. In this scenario, we consider the two vehicles at an intersection, where the autonomous car actively tries to explore human's driving style by nudging into the intersection. The initial conditions of the vehicles are shown in Fig.2(c). In the passive condition, the autonomous car stays at its position without probing the human, and only optimizes for collision avoidance. This provides limited observations from the human car resulting in a low confidence belief distribution. In the active condition, the autonomous car nudges into the intersection to probe the driving style of the human. An attentive human would slow down to stay safe at the intersection while a distracted human will not slow down.
 Fig. 3 : 3 Fig. 3: Legends indicating active/passive robots, attentive/distracted humans, and real user/ideal model used for Fig.4, Fig.??, Fig.5, and Fig.6.
 Fig. 4 : 4 Fig. 4:The probability that the robot assigns to attentive as a function of time, for the attentive (left) and distracted (right). Each plot compares the active algorithm to passive estimation, showing that active information gathering leads to more accurate state estimation, in simulation and with real users.
 Fig. 5 : 5 Fig.5: Robot trajectories for each scenario in the active information gathering condition. The robot acts differently when the human is attentive (dark orange) vs. when the human is distracted (light orange) due to the trade-off with safety.
			 One exception is Nikolaidis et al. [17] , who propose to solve the full POMDP, albeit for discrete and not continuous state and action spaces.
			 Authorized licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 00:30:44 UTC from IEEE Xplore. Restrictions apply.

Abstract: Much of estimation of human internal state (goal, intentions, activities, preferences, etc.) is passive: an algorithm observes human actions and updates its estimate of human state. In this work, we embrace the fact that robot actions affect what humans do, and leverage it to improve state estimation. We enable robots to do active information gathering, by planning actions that probe the user in order to clarify their internal state. For instance, an autonomous car will plan to nudge into a human driver's lane to test their driving style. Results in simulation and in a user study suggest that active information gathering significantly outperforms passive state estimation.


The far future argument for confronting catastrophic threats to humanity: Practical significance and alternatives

Introduction Over several decades, scholars from a variety of fields have advanced an argument for confronting catastrophic threats to humanity, rooted in the far future benefits of doing so.  1  In this context, the far future can loosely be defined as anything beyond the next several millennia, but will often emphasize timescales of millions or billions of years, or even longer.  2  Likewise, the catastrophic threats in question-also known as global catastrophic risks (GCRs) and existential risks, among other things-are those that would affect the trajectory of human civilization over these timescales. The simplest case is catastrophes resulting in human extinction, which is a permanent result and thus affects the trajectory of human civilization into the far future. More subtle but comparably relevant cases include catastrophes resulting in the permanent collapse of human civilization, preventing humanity from ever achieving certain very great things, and catastrophes resulting in delays Sufficiently large catastrophes can affect human civilization into the far future: thousands, millions, or billions of years from now, or even longer. The far future argument says that people should confront catastrophic threats to humanity in order to improve the far future trajectory of human civilization. However, many people are not motivated to help the far future. They are concerned only with the near future, or only with themselves and their communities. This paper assesses the extent to which practical actions to confront catastrophic threats require support for the far future argument and proposes two alternative means of motivating actions. First, many catastrophes could occur in the near future; actions to confront them have near-future benefits. Second, many actions have cobenefits unrelated to catastrophes, and can be mainstreamed into established activities. Most actions, covering most of the total threat, can be motivated with one or both of these alternatives. However, some catastrophe-confronting actions can only be justified with reference to the far future. Attention to the far future can also sometimes inspire additional action. Confronting catastrophic threats best succeeds when it considers the specific practical actions to confront the threats and the various motivations people may have to take these actions. ß 2015 Elsevier Ltd. All rights reserved. in the subsequent rise of civilization toward these achievements. The scholarship argues that people should care about human civilization into the far future, and thus, to achieve far future benefits, should seek to confront these catastrophic threats. Call this the far future argument for confronting catastrophic threats to humanity. In this paper, I will not dispute the basic validity of the far future argument. Indeed, I agree with it, and have advanced it repeatedly in my own work  (Baum, 2009 (Baum, , 2010 Maher and Baum, 2013) . Instead, I assess the extent to which the far future argument is necessary or helpful for actually confronting the threats. In other words, what is the practical significance of the far future argument? I also propose and assess two alternative approaches to confronting the threats. One alternative emphasizes near future benefits of avoiding near future catastrophes. The other alternative emphasizes other (unrelated) benefits of actions that also help confront the threats, creating opportunities even for people who have zero care about the threats. It would be important if the threats can be confronted without the far future argument, because many people do not buy the argument. That people do not is suggested by a range of research. An extensive time discounting literature assess how much people value future costs and benefits. Most discounting studies use time scales of days to decades and focus on future benefits to oneself  (Frederick, Loewenstein, & O'Donoghue, 2002) ; these studies are of limited relevance to valuations of the far future of human civilization. One more relevant time discounting study finds that people discount lives saved 20 years later at a 25% annual rate and lives saved 100 years later at an 8% annual rate  (Johannesson & Johansson, 1996) ; extrapolating this suggests negligible concern for lives saved in the far future. Similarly,  Tonn, Conrad, & Hemrick (2006, p. 821)  find that people believe humanity should plan mainly for the upcoming 20 years or so and should plan less for time periods over 1000 years. In a study on social discounting,  Jones and Rachlin (2006)  find that people are willing to forgo more money to help close friends and family than distant acquaintances; they presumably would forgo even less for members of far future generations. Finally, there are considerations rooted in how societies today are structured. Several researchers have argued that current electoral structures favor the short-term  (Ekeli, 2005; Tonn, 1996; Wolfe, 2008) .  Similarly, Karlsson (2005)  suggests that the rise of decentralized capitalist/democratic political economies and the fall of authoritarian (notably communist) political economies has diminished major long-term planning. While none of these studies directly assess the extent to which people buy the far future argument, the studies all suggest that many people do not buy the argument to any significant degree. To the extent that efforts to confront catastrophic threats can be made synergistic with what people already care about, a lot more can be done. This would seem to be an obvious point, but it has gone largely overlooked in prior research on catastrophic threats. One exception is  Posner (2004) , who argues that some actions to reduce the risk of human extinction can be justified even if only the current generation and its immediate successor are valued. Another is  Baum (2015) , who proposes to confront the threat of catastrophic nuclear winter in terms that could appeal to nuclear weapon states; Baum calls this ''ethics with strategy''. But most of the prior research, including the studies cited above, emphasize the far future argument. This paper expands Posner's argument to further argue that some actions can be taken even for those who only care about their immediate communities or even just themselves. This paper also makes progress toward assessing the total practical significance of the far future by presenting a relatively comprehensive survey of GCRs and GCR-reducing actions. Such surveys are also scarce; one example is  Leggett (2006) , who surveys the space of GCRs to identify priorities for action. The present paper also has commonalities with  Tonn and Stiefel (2014) , who evaluate different levels of sacrifice that society should make in response to GCRs of different magnitude. The present paper also considers levels of sacrifice, but instead argues that, from a practical standpoint, it is better to start with those actions that require less sacrifice or are in other ways more desirable. Indeed, actions requiring large sacrifice may only be justifiable with reference to far future benefits. The paper is organized as follows. Section 2 briefly reviews the space of GCRs. All actions to reduce the risk must help on one or more of these so as to result in a net risk reduction. The space of GCRs likewise provides an organizing framework for subsequent sections, as summarized in Table  1 . Section 3 discusses the timing of GCRs. For catastrophes that could happen earlier, actions to avoid them will include the earlier benefits of catastrophe avoidance. Almost all GCR reduction actions have near-future GCR reduction benefits. Section 4 discusses co-benefits and mainstreaming of GCR reduction actions. Co-benefits are benefits unrelated to GCR reduction. Mainstreaming is integrating GCR reduction into established Table  1  Summary of global catastrophic risk categories (Section 2), their timing (Section 3), co-benefits and mainstreaming opportunities (Section 4), and high-cost GCR reduction actions that may only be justifiable with reference to far future benefits (Section 5). The co-benefits and mainstreaming opportunities and high-cost actions are illustrative examples, not complete listings. activities. Co-benefits and mainstreaming are both ways to facilitate GCR reduction for those who are not specifically motivated by the far future. Section 5 discusses GCR reduction actions that can only be justified in reference to the farfuture benefits of GCR reduction. While these actions will typically not be the best place to start, they can play an important role in overall GCR reduction efforts. Section 6 discusses the ways in which attention to the far future can inspire additional GCR reduction action. This includes both analytical inspiration and emotional inspiration. Section 7 concludes.
 GCR category
 The global catastrophic risks Which actions can help reduce the risk of global catastrophe depend on what the global catastrophic risks are in the first place. This section briefly overviews the risks. The risks have been described in more detail elsewhere  (Asimov, 1979; Barrett, 2007; Bostrom & C ´irkovic ´, 2008; Guterl, 2012; Jha, 2011; Leslie, 1996; Rees, 2003; Tonn & MacGregor, 2009) . While this section lists GCRs in distinct categories, the risks are often interconnected both within and across categories. For example, the emerging technology GCR of geoengineering is developed in response to the environmental change GCR of climate change, and the geoengineering risk is in turn affected by other GCRs such as large-scale violence or pandemics  (Baum, Maher, & Haqq-Misra, 2013) . Similarly, GCR reduction actions can often affect risks in multiple categories. So while the GCR reduction actions discusses in Sections 3-5 are organized in terms of the categories presented here, it should be understood that specific actions often spill across categories. All this suggests a systems approach to studying GCR.
 Environmental change By environmental change, I mean to refer to human-driven global environmental changes; natural disasters are discussed below. Climate change is perhaps the most commonly cited environmental change GCR; worst-case climate change scenarios attract considerable attention (e.g.  Sherwood & Huber, 2010) . Other environmental change GCRs could include biodiversity loss, biogeochemical flows (interference with the nitrogen and phosphorus cycles), stratospheric ozone depletion, ocean acidification, global fresh water use, land use change, chemical pollution, and atmospheric aerosol loading  (Rockstro ¨m et al., 2009a; Rockstro ¨m et al., 2009b) . While any of these phenomena could dramatically alter the global environment, it is less clear whether the impacts would be catastrophic for humanity  (Baum & Handoh, 2014; Raudsepp-Hearne et al., 2010) . For this paper, it is important that many pro-environmental actions simultaneously help across a broad set of global environmental changes, lessening the need to distinguish which changes could be catastrophic for humanity.
 Emerging technologies Several emerging technologies could cause global catastrophe, including artificial intelligence  (Bostrom, 2014; Eden, Moor, Soraker, & Steinhart, 2013) , biotechnology  (Vogel, 2013) , geoengineering  (Baum et al., 2013; Caldeira, Bala, & Cao, 2013) , and nanotechnology for atomically precise manufacturing  (Drexler, 2013) . These risks are relatively uncertain given the unprecedented nature of emerging technology, but they may constitute a significant portion of total risk.
 Large-scale violence A sufficiently large global war could be catastrophic regardless of the technologies used-for comparison, hundreds of thousands died from attacks with machetes and other unmechanized weapons in the Rwandan genocide. Weapons of mass destruction make the job much easier. Nuclear weapons can be catastrophic both through direct explosions and the indirect effects of nuclear winter  (Mills, Toon, Lee-Taylor, & Robock, 2014) . Biological weapons can also readily cause global catastrophe, in particular if they are contagious; indeed, nonstate actors or even single individuals may be able to cause global catastrophes with engineered contagions  (Nouri & Chyba, 2008; Rees, 2003) . The pressures of conflict can also lead actors to take larger risks, as occurred during World War II when the Americans proceeded with the first nuclear weapon test despite concerns that it could ignite the atmosphere, killing everyone  (Konopinski, Marvin, & Teller, 1946) . Finally, major global violence could also result from an oppressive global totalitarian government  (Caplan, 2008) .
 Pandemics Pandemics can be of natural or artificial origin, or both. Humans catch disease from the environment, in particular from other species. The development and transmission of zoonotic diseases can be enhanced by human activities including wild habitat destruction and factory farming. While it is clear that global pandemics can occur, their exact severity is a matter of ongoing analysis and debate  (Germann, Kadau, Longini, & Macken, 2006; Koblentz, 2009) .
 Natural disasters Global catastrophes can result from several natural disasters including asteroid and comet impacts  (Bucknam & Gold, 2008; Sleep & Zahnle, 1998) , supervolcano eruptions  (Driscoll, Bozzo, Gray, Robock, & Stenchikov, 2012; Rampino, Self, & Stothers, 1988) , solar storms (NRC, 2008), and gamma ray bursts  (Atri, Melott, & Karam, 2013) . While these natural disasters generally have lower probabilities, they nonetheless can be worth some effort to confront. Another natural disaster is the gradual warming of the Sun, which will (with very high probability) make Earth uninhabitable for humanity in a few billion years (O'Malley-James, Cockell, Greaves, & Raven, 2014). Other long-term astronomical risks, such as the Milky Way collision with Andromeda (increasing the rate of dangerous supernovae) and the death of all stars (removing a major energy source) play out on similar or longer time scales  (Adams, 2008) .
 Physics experiments Certain types of physics experiments have raised concerns that the experiments could go wrong, obliterating Earth and its vicinity. This notably includes high-energy particle physics experiments such as at the CERN Large Hadron Collider. Physicists evaluating this risk have argued that the risk is vanishingly small; however, they may be underestimating the risk by neglecting the possibility that their analysis is mistaken  (Ord, Hillerbrand, & Sandberg, 2010) .
 Extraterrestrial encounter It is not presently known if there is any extraterrestrial life, let alone intelligent extraterrestrial civilizations. However, if extraterrestrial civilizations exist, then the result could be catastrophic for humanity  (Baum, Haqq-Misra, & Domagal-Goldman, 2011; Michaud, 2007) . Non-civilization extraterrestrial life could also harm humanity with catastrophic contaminations  (Conley & Rummel, 2008) .
 Unknowns There may be entire categories of GCR not yet identified.
 The timing of the global catastrophic risks If a global catastrophe could occur during the near future, then there will be near-future benefits to reducing the risk. The sooner the catastrophe could occur, the larger the near-future benefits would be. In general, it will be easiest to motivate action to confront the most imminent catastrophes-hence  Posner's (2004)  argument that much GCR-reducing action can be justified even if one only cares about the present generation and the next one to come. It is thus worth examining the timing of the catastrophes.
 Specific global catastrophic risks
 Environmental change Major environmental changes are already visible, with larger changes expected on time scales of decades to tens of millennia. Climate change is among the more long-term of these, with some impacts already visible, and the worst climatic effects contained within the next 25,000 years or so.  3  Another possible long-term environmental change is an oceanic anoxic event, which is caused by phosphorus runoff and would in turn cause major die-off of marine species. An oceanic anoxic event could occur on time scales of millennia  (Handoh & Lenton, 2003) ; more localized effects of phosphorus runoff are already visible.
 Emerging technologies Dangerous biotechnology already exists, and is steadily increasing in capability. Early design work for geoengineering is already underway, with deployments suggested to occur in upcoming decades  (Keith, Parsons, & Morgan, 2010) . Experts give a significant probability to GCR-level artificial intelligence occurring within this century or next  (Baum, Goertzel, & Goertzel, 2011; Mu ¨ller & Bostrom, 2014) . Nanotechnology for atomically precise manufacturing may have similar time horizons.
 Large-scale violence Large-scale violence can happen at any time. The ongoing Ukraine crisis is a firm reminder that significant tensions linger between major nuclear weapons states. Nuclear war could even occur inadvertently, due to false alarm events that can occur at any time  (Barrett, Baum, & Hostetler, 2013) . Risks from biological weapons could increase in upcoming decades as biotechnology advances. However, overall risk from large-scale violence may be gradually declining, following a general trend toward less violence  (Pinker, 2011)  and an increasing sophistication of global peacekeeping capability  (Goldstein, 2011) .
 Pandemics Pandemics can also break out at any time. Recent outbreaks of SARS, H5N1 and H1N1 flus, MERS, and currently Ebola have so far not reached a high degree of global lethality, but they are clear reminders that the threat of pandemics persists. Advances in biotechnology can lead to increasing risk through both intentional use and mishaps, as can increasing global connectivity. On the other hand, advances in public health can reduce the risk.
 Natural disasters Many natural disasters can also occur at any time. Risk from impact events, supervolcano eruptions, solar storms, and gamma ray bursts is roughly constant over long periods of time, into the far future. NASA's near-Earth objects survey has significantly reduced estimates of the risk of large impacts occurring over the next century or so  (Harris, 2008) . Several astronomical risks, including the Sun's gradual warming, the Milky Way collision with Andromeda, and the death of all stars, are GCRs that exists exclusively in the far future.
 Physics experiments The risk from physics experiments depends critically on which physics experiments are conducted. The risk could increase as the capability to conduct experiments increases.
 Extraterrestrial encounter Humanity could encounter extraterrestrials at any time, including through ongoing searches for extraterrestrial intelligence (SETI). Some risk (especially contamination risk) comes mainly from human or robotic travel in space. Some risk comes from messaging to extraterrestrial intelligence (METI;  Haqq-Misra, Busch, Som, & Baum, 2013) . The timing of METI risk depends on the distance from Earth to the location being messaged. METI to sufficiently distant locations is another GCR that exists exclusively in the far future.
 Unknowns Unknown GCRs could occur in both the near and far future. Indeed, more future GCRs are less likely to be already identified.
 Discussion Relatively few identified GCRs exist exclusively in the far future: certain astronomical risks and METI to distant locations. For all other GCRs-and this constitutes almost all of the total identifiable risk-the catastrophes could occur in the near future. The identified risks from emerging technologies and physics experiments could only occur in the near future. The identified risks from environmental change, large-scale violence, pandemics, some natural disasters, some extraterrestrial encounter risks, and unknowns could occur in the near or far future. The preponderance of near future risks suggests that a lot of actions to reduce these risks can be done without reference to their far future benefits. On the other hand, these near future benefits may not always be enough, especially when the catastrophes would occur several decades or centuries later, as people often care little about even these earlier times. It is thus worth pursuing other means of motivating GCR reduction.
 Co-benefits and mainstreaming GCR-reducing actions Insight on how to motivate GCR reduction can be found from outside the core GCR literature, in some related literatures. The climate change mitigation community has developed the concept of co-benefits, defined as benefits besides the target goal  (Hosking, Mudu, & Dora, 2011; Miyatsuka & Zusman, n.d.) . For climate change mitigation, the target goal is greenhouse gas emissions reductions. Research assesses how communities can reduce their emissions while improving their economic development, public health, and wellbeing. The co-benefits concept readily applies GCR. Some actions to reduce GCR will also be profitable, fun, healthy, satisfying, safe, or otherwise desirable, often to those who perform the actions. Similarly, the natural disaster management community has a robust practice of mainstreaming disaster management into established goals and procedures, especially those regarding development  (Benson, 2009; Twigg & Steiner, 2002) . Disaster management actions will often only be taken when they can be integrated into established goals and procedures; otherwise, the actions will be too impractical or undesirable. For example, urban design steps to reduce a town's vulnerability to hurricane storm surge can be mainstreamed into the town's broader urban planning processes  (Frazier, Wood, & Yarnal, 2010) . GCR reduction actions can likewise also be mainstreamed into whatever people are already doing or trying to do. The GCR reduction community would be wise to adopt the approaches of co-benefits and mainstreaming. Doing so requires an understanding of the co-benefits that can come from various GCR-reducing actions and the relevant established goals and procedures. For many of these actions-in particular those with sufficiently large co-benefits and wellestablished goals and procedures-reducing GCR can be a nice ancillary benefit of actions that might as well be taken anyway. For these actions, no concern for the far future is needed; often, no concern beyond one's immediate community is needed. These actions require the least sacrifice (indeed, it is a sacrifice not to take these actions) and likewise will often be the easiest actions to promote. This begs the question of which GCR-reducing actions have significant co-benefits and mainstreaming opportunities.
 . Environmental change Environmental change is largely driven by a wide variety of basic activities, including food consumption, transport, real estate development, and natural resource usage. More environmentally friendly actions can often be justified for nonenvironmental reasons. A recent study by McKinsey  (Enkvist, Naucle ´r, & Rosander, 2007)  found that many greenhouse gas emission reductions would result in net monetary benefits for those who reduce these emissions, especially in the realm of energy efficiency in buildings and transport. Happiness research has found that people rate their daily commute as being among their least happy activities  (Layard, 2003) . Public health research links high-meat diets with obesity and other health problems  (Pan et al., 2012) . Buildings, transport, and food are meanwhile three of the most environmentally important sectors  (Metz et al., 2007; Steinfeld et al., 2006; USEIA, 2011) . If significant changes in these sectors can be achieved for nonenvironmental reasons, the environmental benefit could be quite large.
 Emerging technologies It is often beneficial to develop regulations for multiple technologies at the same time, due to similarities between the technologies and the regulations  (Kuzma & Priest, 2010; Wilson, 2013a) . Concerns about other technologies can thus motivate general technology regulation, which provides a framework for mainstreaming the regulation of emerging technologies GCRs. In addition, some actions specific to certain emerging technologies can have co-benefits. For example, one proposed solution to artificial intelligence risk is to design the AI to be ''friendly'' to humanity. In addition to not causing a catastrophe, such an AI could help with other societal problems  (Muehlhauser & Bostrom, 2014) . If such an AI can be achieved with sufficient confidence, then this could be an attractive action even for those who are not concerned about AI risk.
 Large-scale violence Achieving peace avoids violence at all scales and also brings a variety of co-benefits. One co-benefit is economic growththe so-called ''peace dividend''  (Knight, Loayza, & Villanueva, 1996; Ward & Davis, 1992) . Another co-benefit is psychological. Recent research finds that conflict is often driven by humiliation, and likewise that giving people a sense of dignity can help  (Lindner, 2006; Stern, 2003) . Finally, other research suggests that reducing domestic violence against women could lead to less interstate war  (Hudson, Ballif-Spanvill, Caprioli, & Emmett, 2012) . Emphasizing these co-benefits could justify much action to reduce large-scale violence. Another worthy point of focus is violent nonstate actors, which continue to receive extensive attention in the wake of the 9-11 attacks. While nonstate actors may not be able to cause violence large enough to result in global catastrophe, 4 actions to confront them may have co-benefits and mainstreaming opportunities for large-scale violence. For example, the annual Nuclear Security Summits initiated by US President Barack Obama aim to prevent nonstate actors from acquiring nuclear weapons, but they also strengthen norms against nuclear weapon use more generally.
 Pandemics As noted above, there is some debate about how severe pandemics could be, including whether they would impact the far future. To a large extent, this debate is irrelevant. Regardless of how severe pandemics would be, there already exists a significant global public health infrastructure that responds to pandemics of all sizes. Improving this infrastructure can further improve the response. The case for improving this infrastructure is strengthened by the possibility of catastrophic pandemics, but the case is not dependent on this possibility  (McKibbin & Sidorenko, 2006) .
 Natural disasters The GCR literature has proposed certain measures to increase society's resilience to a wide range of global catastrophes, including natural disasters. These measures include food stockpiles , underground refuges  (Jebari, 2014) , and space colonies and refuges  (Abrams et al., 2007; Shapiro, 2009) . These measures also tend to increase society's resilience to smaller catastrophes. Indeed, many actions taken to prepare for smaller catastrophes also benefit GCR reduction. In addition, while space colonies and refuges have been criticized for their high cost relative to other means of reducing global catastrophic risk  (Baum, 2009; Sandberg, Matheny, & C ´irkovic ´, 2008) , some space missions are already underway or in planning for a variety of other reasons, including science, political prestige, and economic opportunity (e.g. in asteroid mining). Space colonies or refuges could be mainstreamed into these missions  (Baum, Denkenberger, & Haqq-Misra, 2014) .
 Physics experiments Physics experiments are a curious case, because the relevant experiments are quite expensive (hundreds of millions to billions of dollars) and the social benefits somewhat limited. As  Parson (2007, p. 155)  puts it, ''this research is remote from practical application and serves largely to indulge national pride and the intellectual passion of a tiny elite group''. Arguably, the co-benefits of reducing physics experiment risk include the money saved by not doing the experiments, and by tasking the money to a worthier cause, analogous to the peace dividend, though this is likely to be a controversial view among those who value the physics experiments.
 Extraterrestrial encounter Protection against extraterrestrial contamination has the co-benefit of protecting extraterrestrial environments from contamination by humans, which is of significant scientific value  (Conley & Rummel, 2008) . The costs of SETI and METI are small relative to the big physics experiments, so while there are dollar savings to realize from skipping them, these are less of an issue. Perhaps the extraterrestrial-risk-reducing action with the most co-benefits would be research and public education into what the risks could be. Discussions of ETI are very popular, as seen in the extensive popular media and entertainment attention to ETI.
 Unknowns Actions likely to reduce unknown GCRs will typically be generic actions that also help reduce other GCRs or even smaller risks, such as building refuges  (Jebari, 2014)  and stockpiling resources .
 Discussion Many GCR-reducing actions, covering the full breadth of GCRs, have sizable co-benefits, and can also be mainstreamed into existing activities. Many of these actions will often be desirable even without reference to GCR, let alone to the far future benefits of GCR reduction. These ''easy'' actions will typically be the lowest hanging fruit, the easiest GCR reductions to promote. They offer a sensible starting point for those seeking to reduce GCR.
 Actions with significant cost Ideally, all GCR-reducing actions would have low costs and large co-benefits, such that it would be easy to persuade people to take the actions, and such that the totality of GCR could be reduced with minimal burden to those taking the actions and to society at large. As discussed above, many such actions exist. However, this is not the case for all GCR-reducing actions. Some of these other actions require considerable sacrifice, especially the most aggressive GCR-reduction efforts. Tonn and Stiefel's (2014) levels of societal actions are instructive here. The levels range from doing nothing to an extreme war footing in which society is organized specifically to reduce GCR. Actions requiring more sacrifice, especially those at or near the level of extreme war footing, might only be justifiable with reference to the far-future benefits. While these actions will typically not be the lowest hanging fruit, they could be important components of an overall portfolio of GCR-reducing actions.
 Specific global catastrophic risks
 Environmental change The most aggressive pro-environmental actions include public policies like a high carbon tax, personal behaviors requiring great inconvenience and sacrifice, and restructuring the entire global industrial economy away from fossil fuels and other pollutants. To achieve a larger reduction in environmental change GCR, some of these more aggressive actions may be needed. That these more aggressive actions may only be justifiable with reference to far-future benefits is a core point from debates about discounting in environmental policy (Nordhaus, 2008).
 Emerging technologies One way to reduce emerging technology GCR is to simply abstain from developing the technologies, i.e., to relinquish them  (Joy, 2000) . However, if these technologies do not cause catastrophe, they sometimes come with great benefits: geoengineering can avoid the worst effects of climate change; AI can solve a variety of social problems; biotechnology can help cure disease. Thus relinquishing the technologies can require a large sacrifice  (Baum, 2014) . This sacrifice may sometimes only be justifiable given the far-future benefits of GCR reduction.
 Large-scale violence While nuclear weapons can cause great harm, they are also often attributed with helping maintain peace, through the doctrine of nuclear deterrence: countries hesitate to attack each other for fear of being destroyed in nuclear retaliation. There are questions about the efficacy of nuclear deterrence  (Wilson, 2013b)  and there are proposals to achieve deterrence with out large nuclear arsenals  (Baum, 2015) . However, a common view posits that nuclear deterrence is necessary until international relations are peaceful enough for a world without nuclear weapons  (Obama, 2009) . Following this logic, immediate nuclear disarmament might reduce GCR, but it might also increase the preponderance of smaller conflicts and other geopolitical instabilities. Depending on the details, immediate nuclear disarmament might only be justifiable with reference to the far future.
 Pandemics One aggressive action to reduce pandemics risk would be aggressive quarantine, such as blockading the major islands of Indonesia, Japan, the United Kingdom, and other countries. Travel restrictions could keep populations in these places safe. During a sufficiently severe outbreak, populations in these places could even request to be blockaded. A safer but costlier and less desirable policy would blockade them at first alert of a possible outbreak, or even keep them blockaded on a permanent basis. Doing so might lower GCR, but might only be justifiable with reference to the far future.
 Natural disasters One proposed aggressive action for natural disasters could be to drill the ground around potential supervolcanoes to extract the heat, although the technological feasibility of this proposal has not yet been established.  5  This could be a very costly project, but, if it works, it could also reduce supervolcanoes GCR. The project would come with a co-benefit of geothermal energy, but this is likely not nearly enough to justify the expense. Another possibility is advanced surfaceindependent refuges, which could protect against a variety of GCRs, including many of the natural disasters, but again could come at great expense  Beckstead, 2014) .
 Physics experiments and extraterrestrial encounter I am not aware of any actions to reduce near-future GCR from physics experiments and extraterrestrial encounter that have significant cost and can only be justified with reference to far-future benefits. To the contrary, many actions to reduce these risks save money (Section 4.1). Protection against contamination does have a cost, and shutting METI programs down could cost the public a source of popular entertainment. On the other hand, the shut down itself could also create an entertaining controversy. Regardless, the costs involved are not large.
 Unknowns One action that might only be justifiable with reference to far future benefits is a far-future version of the ''extraterrestrial time capsule'' proposed in . These capsules contain artifacts of benefit to catastrophe survivors for a range of known and unknown catastrophe scenarios. The capsules are launched into space in trajectories designed to return to Earth at some future date.  suggest a return date 100 years into the future, but it may be possible (and expensive) to have return dates in the far future.
 Discussion For those who wish to keep humanity highly safe from catastrophe, there are actions that can only be justified with reference to the far-future benefits of GCR reduction. While these actions are typically not the best place to start, they can offer additional GCR reductions beyond what the easier actions offer. Given the enormous far-future benefits of GCR reduction, arguably these actions merit consideration. However, hopefully GCR can be essentially eliminated without resorting to these actions. If these actions are necessary, it will likewise be necessary to appeal to the importance of the far future.
 Far future as inspiration The paper thus far has focused on how to avoid appeals to the far future argument, in recognition of the fact that many people are not motivated by what will benefit the far future. But some GCR reduction actions can only be justified with reference to far future benefits. Additionally, some people are motivated to benefit the far future. Other people could be too. Tapping the inspirational power of the far future can enable more GCR reduction. There are at least two ways that the far future can inspire action: analytical and emotional. Both are consistent with the far future argument, but the argument is typically inspired by analytical considerations. The analytical inspiration is found in works analyzing how to maximize the good or achieve related objectives. Most of the scholarly works invoking the far future argument are of this sort.  6  Such ideas have the potential to resonate not just with other scholars, but with people in other professions as well, and also the lay public. Thus there can be some value to disseminating analysis about the importance of the far future and its relation to GCR. Analytical inspiration can also come from analyzing specific actions in terms of their far-future importance. Such analysis can help promote these actions, even if the actions could be justified without reference to the far future. However, the analysis should be careful to connect with actual decision makers, and not just evaluate hypothetically optimal actions that no one ever takes. For example, there has been now multiple decades of research analyzing what the optimal carbon tax should be (for an early work, see  Nordhaus, 1992 ), yet throughout this period, for most of the world, the actual carbon tax has been zero. Analytical inspiration has its limits. Research effort may be more productively spent on what policies and other actions people are actually willing to implement. The other far future inspiration is emotional. The destruction of human civilization can itself be a wrenching emotional idea. In The Fate of the Earth, Jonathan Schell writes ''The thought of cutting off life's flow, of amputating this future, is so shocking, so alien to nature, and so contradictory to life's impulse that we can scarcely entertain it before turning away in revulsion and disbelief''  (Schell, 1982 (Schell, /2000 ). In addition, there is a certain beauty to the idea of helping shape the entire arch of the narrative of humanity, or even the universe itself. People often find a sense of purpose and meaning in contributing to something bigger than themselves-and it does not get any bigger than this. Carl  Sagan's (1994)  Pale Blue Dot and James  Martin's (2007)  The Meaning of the 21st Century both capture this well, painting vivid pictures of the special place of humanity in the universe and the special opportunities people today have to make a difference of potentially cosmic significance. This perspective says that humanity faces great challenges. It says that if these challenges are successfully met, then humanity can go on to some amazing achievements. It is a worthy perspective for integrating the far future into our lives, not just for our day-to-day actions but also for how we understand ourselves as human beings alive today. This may be worth something in its own right, but it can also have a practical value in motivating additional actions to confront catastrophic threats to humanity.
 Conclusion The far future argument is sound. The goal of helping the far future is a very worthy one, and helping the far future often means helping reduce the risk of those global catastrophes that could diminish the far-future success of human civilization. However, in practical terms, reducing this risk will not always require attention to its far-future significance. This is important because many people are not motivated to help the far future, but they could nonetheless be motivated to take actions that reduce GCR and in turn help the far future. They may do this because the actions reduce the risk of near-future GCRs, or because the actions have co-benefits unrelated to GCRs and can be mainstreamed into established activities. This paper surveys GCRs and GCR-reducing actions in terms of how much these actions require support for the far future argument for confronting catastrophic threats to humanity. The analysis suggests that a large portion of total GCR, probably a large majority, can be reduced without reference to the far future and with reference to what people already care about, be it the near future or even more parochial concerns. These actions will often be the best to promote, achieving the largest GCR reduction relative to effort spent. On the other hand, some significant GCR reducing actions (especially those requiring large sacrifice) can only be justified with reference to their far-future benefits. For these actions in particular, it is important to emphasize how the far future can inspire action. Several priorities for future research are apparent. Quantitative GCR analysis could help identify which actions best reduce GCR and also what portion of GCR can be reduced without reference to the far future. Analysis covering the breadth of GCRs would be especially helpful. Social scientific research could study how to effectively engage stakeholders so as to leverage co-benefits and mainstream GCR reductions into existing programs. Social scientific research could also examine how to effectively tap the inspirational power of the far future, especially for emotional inspiration, which has received limited prior attention. Progress in these research areas could go a long way toward identifying how to, in practice, achieve large GCR reductions. The overall message of this paper is that helping the far future requires attention to which specific actions can help the far future and likewise to what can motivate these actions. The actions are not necessarily motivated by their far-future impact. This is fine. The far future does not care why people acted to help it-the far future only cares that it was helped. And people taking these actions will rarely mind that their actions also help the far future. Most people will probably view this as at least a nice ancillary benefit. Additionally, people will appreciate that those promoting the far future have taken the courtesy to consider what they care about and fit the far future into that. It can be disrespectful and counterproductive to expect people to drop everything they are doing just because some research concluded that the far future is more important. This means that those who seek to promote actions to benefit the far future must engage on an interpersonal level with the people who will take these actions, to understand what these people care about and how far-future-benefiting actions can fit in. This is an important task to pursue, given the enormity of what human civilization can accomplish from now into the far future. Futures
			 The 25,000 year figure is derived from Archer and Ganopolski (2005) , Fig.3C, which shows a rapid temperature spike that declines most of the way back to current temperatures within 25,000 years and then remains at similar temperatures for another 500,000 years. However, this refers specifically to climatic effects; the human effects could persist longer, especially if the climate change causes a civilization-ending global catastrophe. S.D.Baum / Futures 72 (2015) 86-96
			 Nuclear terrorism would likely be too small to cause a far-future-impacting global catastrophe, unless it catalyzed a large-scale interstate nuclear war (Ayson, 2010) . Biological terrorism could more readily cause a global catastrophe, as discussed above.S.D.Baum / Futures 72 (2015) 86-96
			 S.D.Baum / Futures 72 (2015) 86-96
			 An idea to this effect is briefly discussed inLeggett (2006, p. 794). 6 See citations in Footnote 1. S.D. Baum / Futures 72 (2015) 86-96

Abstract: others. 2 This definition of the far future is most explicitly stated in Beckstead (2013). In contrast, psychology and cognitive science research commonly defines ''far future'' in timescales of years (e.g.


Victor Galaz
Miguel A Centeno
Peter W Callahan
Amar Causevic
Thayer Patterson
Irina Brass
Seth Baum
Darryl Farber
Joern Fischer
David Garcia
Timon Mcphearson
Daniel Jimenez
Brian King
Paul Larcey
Karen Levy

Abstract: Automated decision making and predictive analytics through artificial intelligence, in combination with rapid progress in technologies such as sensor technology and robotics are likely to change the way individuals, communities, governments and private actors perceive and respond to climate and ecological change. Methods based on various forms of artificial intelligence are already today being applied in a number of research fields related to climate change and environmental monitoring. Investments into applications of these technologies in agriculture, forestry and the extraction of marine resources also seem to be increasing rapidly. Despite a growing interest in, and deployment of AI-technologies in domains critical for sustainability, few have explored possible systemic risks in depth. This article offers a global overview of the progress of such technologies in sectors with high impact potential for sustainability like farming, forestry and the extraction of marine resources. We also identify possible systemic risks in these domains including a) algorithmic bias and allocative harms; b) unequal access and benefits; c) cascading failures and external disruptions, and d) trade-offs between efficiency and resilience. We explore these emerging risks, identify critical questions, and discuss the limitations of current governance mechanisms in addressing AI sustainability risks in these sectors.