The Value Learning Problem

Introduction Standard texts in AI safety and ethics, such as  Weld and Etzioni [1994]  or  Anderson and Anderson [2011] , generally focus on autonomous systems with reasoning abilities that are complementary and not strictly superior to those of humans. Relatively little attention is given to future AI systems that may be "superintelligent" in the sense of  Bostrom [2014] , i.e., systems "much smarter than the best human brains in practically every field, including scientific creativity, general wisdom, and social skills." Our discussion will place a greater focus on methods and frameworks for designing robust and beneficial smarter-than-human AI systems, bracketing questions about whether such systems would have moral standing of their own. Smarter-than-human AI systems are likely to introduce a number of new safety challenges. First, bad behavior by smarter-than-human systems can have larger and more lasting consequences; an antisocial adult is more dangerous than an antisocial child, even if the adult is as physically weak as a child. Whereas low-intelligence systems can be tested and patched over many iterations, Bostrom argues that even small errors in the first superintelligent systems could have extinction-level consequences  [Bostrom, 2014] . The possible development of such systems raises the stakes for AI safety work. Second, systems that can strictly outperform humans cognitively have less to gain from integrating into existing economies and communities.  Hall [2007]  has argued: The economic law of comparative advantage states that cooperation between individuals of differing capabilities remains mutually beneficial.  [ . . . ]  In other words, even if AIs become much more productive than we are, it will remain to their advantage to trade with us and to ours to trade with them. As noted by Benson-Tilsen and Soares [forthcoming 2016], however, rational trade presupposes that agents expect more gains from trade than from coercion. Non-human species have various "comparative advantages" over humans, but humans generally exploit non-humans through force. Similar patterns can be observed in the history of human war and conquest. Whereas agents at similar capability levels have incentives to compromise, collaborate, and trade, agents with strong power advantages over others can have incentives to simply take what they want. The upshot of this is that engineering a functioning society of powerful autonomous AI systems and humans requires that those AI systems be prosocial. The point is an abstract one, but it has important practical consequences: rational agents' interests do not align automatically, particularly when they have very different goals and capabilities. Third, superhumanly creative and adaptive systems may arrive at what  Bostrom [2014, chap. 8 ] calls "perverse instantiations" of their programmed goals.  Wiener [1960]  calls this the "Sorcerer's Apprentice" problem, after the fable of an apprentice whose enchanted broom follows instructions' letter but not their spirit. The novelty here is not that programs can exhibit incorrect or counter-intuitive behavior, but that software agents smart enough to understand natural language may still base their decisions on misrepresentations of their programmers' intent. The idea of superintelligent agents monomaniacally pursuing "dumb"-seeming goals may sound odd, but it follows from the observation of  Bostrom and Yudkowsky [2014, chap. 7 ] that AI capabilities and goals are logically independent. 1 Humans can fully comprehend that their "designer" (evolution) had a particular "goal" (reproduction) in mind for sex, without thereby feeling compelled to forsake contraception. Instilling one's tastes or moral values into an heir isn't impossible, but it also doesn't happen automatically. Lastly,  Bostrom and Yudkowsky [2014]  point out that smarter-than-human systems may become better than humans at moral reasoning. Without a systematic understanding of how perverse instantiations differ from moral progress, how can we distinguish moral genius in highly intelligent machines from moral depravity? Given the potential long-term impact of advanced AI systems, it would be prudent to investigate whether early research progress is possible on any of these fronts. In this paper we give a preliminary, informal survey of several research directions that we think may help address the above four concerns, beginning by arguing for indirect approaches to specifying human values in AI agents. We describe a promising approach to indirect value specification, value learning, and consider still more indirect approaches based on modeling actual and potential states of human operators.
 Valuable Goals Cannot Be Directly Specified We argued above that highly capable autonomous systems could have disastrous effects if their values are misspecified. Still, this leaves open the possibility that specifying correct values is easy, or (more plausibly) that it presents no special difficulties over and above the challenge of building a smarterthan-human AI system. A number of researchers have voiced the intuition that some simple programmed goal would suffice for making superintelligent systems robustly beneficial.  Hibbard [2001] , for example, suggested training a simple learning system to recognize positive human emotions from facial expressions, voice tones, and body language. Hibbard then proposed that machines of much greater capability-perhaps even superintelligent machinescould be programmed to execute actions predicted to lead to futures with as many "positive human emotions" as possible, as evaluated by the original simple learning system. This proposal has some intuitive appeal-wouldn't such a system always act to make humans happy?-until one considers the Sorcerer's Apprentice. We have a particular set of associations in mind when we speak of "positive human emotions," but the simple learner would almost surely have learned a different and simpler concept, such as "surface features correlating with positive human emotions in the training data." This simpler concept almost surely does not have its maximum at a point which Hibbard would consider to contain lots of positive human emotions. The maximum is much more likely to occur in (for example) scenarios that contain an enormous number of tiny human-shaped animatronics acting out positive human emotions. Thus, a powerful learning system that takes actions according to how well the simple learner would rank them is liable to spend time and resources creating animatronics rather than spending time and resources making humans happy. Indeed,  Hibbard [2012]  himself comes to the conclusion that his proposal fails to exclude the possibility that lifelike animatronic replicas of happy people could be counted as exhibiting "positive emotions." As another example, Schmidhuber  [2007]  proposes that creativity, curiosity, and a desire for discovery and beauty can be instilled by creating systems that maximize a different simple measure: "create action sequences that extend the observation history and yield previously unknown / unpredictable but quickly learnable algorithmic regularity or compressibility." However, while it is quite plausible that human creativity and discovery are related to the act of compressing observation, an agent following Schmidhuber's goal would not behave in intuitively curious and creative ways. One simple way to meet Schmidhuber's desideratum, for example, is to appropriate resources and construct artifacts that generate cryptographic secrets, then present the agent with a long and complex series of observations encoded from highly regular data, and then reveal the secret to the agent, thereby allowing the agent to gain enormous compression on its past sensory data. An agent following Schmidhuber's goal is much more likely to build artifacts of this form than it is to pursue anything resembling human creativity. The system may not take this action in particular, but it will take actions that generate at least that much compression of its sensory data, and as a result, the system is unlikely to be prosocial. Building an agent to do something which (in humans) correlates with the desired behavior does not necessarily result in a system that acts like a human. The general lesson we draw from cases like these is that most goals that are simple to specify will not capture all the contextual complexities of real-world human values and objectives  [Yudkowsky, 2011] . Moral psychologists and moral philosophers aren't locked in decades-and centuries-long debates about the right codifications of ethics because they're missing the obvious. Rather, such debates persist for the simple reason that morality is complicated. People want lots of things, in very particular ways, and their desires are context-sensitive. Imagine a simplified state space of possibilities that vary in count (how many happy human-shaped objects exist), in the size of the average happy human-shaped object, and in the average moral worth of happy human-shaped objects. Human experience has occurred in a small region of this space, where almost all human-shaped objects emitting what looks like happiness are ≈ 2-meter-sized humans with moral weight. But the highest scores on the count axis occur in tandem with low size, and the smallest possible systems that can mimic outward signs of emotion are of low moral worth. In linear programming, it is a theorem that the maximum of an objective function occurs on a vertex of the space. (Sometimes the maximum will be on an edge, including its vertices.) For intuitively similar reasons, the optimal solution to a goal tends to occur on a vertex (or edge, or hyperface) of the possibility space. Hibbard's goal does not contain any information about size or moral worth, and so agents pursuing this goal only consider size and moral worth insofar as they pertain to pushing toward the hyperface of maximum count. To quote : A system that is optimizing a function of n variables, where the objective depends on a subset of size k < n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. The Sorcerer's Apprentice problem arises when systems' programmed goals do not contain information about all relevant dimensions along which observations can vary. The agent has been directed towards the wrong hyperface of the possibility space.  2  When confronted with this type of failure, many have an impulse to patch the flawed goals. If Hibbard's system would make smiling animatronics, then find ways to require that the emotions come from actual humans; if the system would then put humans in a drugged stupor in order to make them smile, forbid it from using drugs; and so on. Such constraints cut off particular means by which the system can get a higher count, but they don't address the underlying problem that the system is still maximizing count. If one causal pathway is forbidden, then the system will follow the nearest non-forbidden causal path-e.g., mechanically manipulating the pleasure centers of human brains. It isn't feasible to patch every goal; nor is it safe to patch as many as come to mind and assume that there are no unforeseen perverse instantiations. Intuitively, we would like to direct the intelligence of highly advanced systems to solving some of this problem on our behalf, and we would like such systems to attend to our likely intentions even when our formal and informal representations of our intentions are flawed. The notion of the operator's "intentions," however, is unlikely to lend itself to clean formal specification. By what methods, then, could an intelligent machine be constructed to reliably learn what to value and to act as its operators intended?
 Inductive Value Learning Correctly specifying a formal criterion for recognizing a cat in a video stream by hand is difficult, if not impossible. This does not mean, however, that cat recognition is hopeless; it means that a level of indirection is required. An image recognition system can be constructed and trained to recognize cats. We propose that the value learning problem be approached by similarly indirect means. Inductive value learning via labeled training data raises a number of difficulties. A visual recognition system classifies images; an inductive value learning system classifies outcomes. What are outcomes? What format would a value-learning data set come in? Imagine a highly intelligent system that uses large amounts of data to construct a causal model of its universe. Imagine also that this world-model can be used to reason about the likely outcomes of the agent's available actions, that the system has some method for rating outcomes, and that it executes the action leading to the most highly rated outcome. In order for the system to inductively learn what to value, the system must be designed so that when certain "training" observations are made (or specially-demarcated updates to its world-model occur), labeled training data extracted from the observation or update alters the method by which the system ranks various potential outcomes. This simple model highlights a central concern and two open questions relevant to inductive value learning.
 Corrigibility Imagine that some of an agent's available actions allow it to modify itself, and that it currently assigns high utility to outcomes containing high numbers of animatronic replicas of humans. It may be the case that, according to the system's world-model, all of the following hold: (1) if more training data is received, those high-rated outcomes will have their ratings adjusted downwards; (2) after the ratings are adjusted, the system will achieve outcomes that have fewer cheap animatronics; and (3) there are actions available which remove the inductive value learning framework. In this situation, a sufficiently capable system would favor actions that disable its value learning framework. It would not necessarily consider its own process of learning our values a good thing, any more than humans must approve of psychological disorders they possess. One could try to construct protected sections of code to prevent the value learning framework from being modified, but these constraints would be difficult to trust if the system is more clever than its designers when it comes to exploiting loopholes. A robustly safe initial system would need to be constructed in such a way that actions which remove the value learning framework are poorly rated even if they are available. Some preliminary efforts toward describing a system with this property have been discussed under the name corrigibility by  Soares and Fallenstein [2015] , but no complete proposals currently exist.
 Ontology Identification The representations used in a highly intelligent agent's worldmodel may change over time. A fully trustworthy value learning system would need to not only classify potential outcomes according to their value, but persist in doing so correctly even when its understanding of the space of outcomes undergoes a major change. Consider a programmer that wants to train a system to pursue a very simple goal: produce diamond. The programmers have an atomic model of physics, and they generate training data labeled according to the number of carbon atoms covalently bound to four other carbon atoms in that training outcome. For this training data to be used, the classification algorithm needs to identify the atoms in a potential outcome considered by the system. In this toy example, we can assume that the programmers look at the structure of the initial worldmodel and hard-code a tool for identifying the atoms within. What happens, then, if the system develops a nuclear model of physics, in which the ontology of the universe now contains primitive protons, neutrons, and electrons instead of primitive atoms? The system might fail to identify any carbon atoms in the new world-model, making the system indifferent between all outcomes in the dominant hypothesis. Its actions would then be dominated by any tiny remaining probabilities that it is in a universe where fundamental carbon atoms are hiding somewhere. This is clearly undesirable. Ideally, a scientific learner should be able to infer that nuclei containing six protons are the true carbon atoms, much as humans have done. The difficulty lies in formalizing this process. To design a system that classifies potential outcomes according to how much diamond is in them, some mechanism is needed for identifying the intended ontology of the training data within the potential outcomes as currently modeled by the AI. This is the ontology identification problem introduced by de Blanc  [2011]  and further discussed by  Soares [2015] . This problem is not a traditional focus of machine learning work. When our only concern is that systems form better world-models, then an argument can be made that the nuts and bolts are less important. As long as the system's new world-model better predicts the data than its old world-model, the question of whether diamonds or atoms are "really represented" in either model isn't obviously significant. When the system needs to consistently pursue certain outcomes, however, it matters that the system's internal dynamics preserve (or improve) its representation of which outcomes are desirable, independent of how helpful its representations are for prediction. The problem of making correct choices is not reducible to the problem of making accurate predictions. Inductive value learning requires the construction of an outcome-classifier from value-labeled training data, but it also requires some method for identifying, inside the states or potential states described in its world-model, the referents of the labels in the training data. This could perhaps be done during the course of inductive value learning. The system's methods for inferring a causal world-model from sense data could perhaps be repurposed to infer a description of what has been labeled. If the system adopts a better world-model, it could then re-interpret its training data to re-bind the value labels. This looks like a promising line of research, but it seems to us to require new insights before it is close to being formalizable, let alone usable in practice. In particular, we suspect that ontology identification will require a better understanding of algorithms that construct multi-level world-models from sense data.
 Ambiguity Identification Reinforcement learning can be thought of as a method for sidestepping these difficulties with value learning. Rather than designing systems to learn which outcomes are desirable, one creates a proxy for desirable outcomes: a reward function specified in terms of observations. By controlling rewards via a reward signal, the operator can then judiciously guide the learner toward desired behaviors. Indirect proxies for desired outcomes, however, face many of the same Sorcerer's Apprentice difficulties. Maximizing how often an operator transmits a reward signal is distinct from the problem of maximizing the operator's satisfaction with outcomes; these goals may coincide in testing environments and yet diverge in new environments-e.g., once the learner has an opportunity to manipulate and deceive its operator or otherwise hijack its reward channel  [Bostrom, 2014, chap. 12] . For further discussion, see  Soares [2015] . Superintelligent systems that achieve valuable real-world outcomes may need goals specified in terms of desirable outcomes, rather than rewards specified in terms of observations. If so, then we will need some robust way of ensuring that the system learns our goals, as opposed to superficially similar goals. When training a recognition system, producing satisfactory training data is often a difficult task. There is a classic parable of machine learning (told by, e.g.,  Dreyfus and Dreyfus [1992] ) of an algorithm intended to classify whether or not pictures of woods contained a tank concealed between the trees. Pictures of empty woods were taken one day; pictures with concealed tanks were taken the next. The classifier identified the latter set with great accuracy, and tested extremely well on the portion of the data that had been withheld from training. However, the system performed poorly on new images. It turned out that the first set of pictures had been taken on a sunny day, while the second set had been taken on a cloudy day. The classifier was not identifying tanks; it was identifying image brightness! The same mistake is possible when constructing a training data set for inductive value learning. In value learning, however, such mistakes may be more difficult to notice and more consequential. Consider a training set that successfully represents real-world cases of happy human beings (labeled with high ratings) and real-world cases of pointless human suffering (rated poorly). The simplest generalization from this data may, again, be that human-shaped-things-proclaiminghappiness are of great value, even if these are animatronics imitating happiness. It seems plausible that someone training an inductive value learner could neglect to include a sufficiently wide variety of animatronics mimicking happiness and labeled as low-value. How many other obvious-in-retrospect pitfalls are hiding in our blind spots? A training set covering all relevant dimensions that we can think of may yet exclude relevant dimensions. A robustly safe value learner would need to be able to identify new plausiblyrelevant dimensions along which no training data is provided, and query the operators about these ambiguities. This is the kind of modification that would help in actually solving the value learning problem, as opposed to working around it. At the same time, this is the kind of modification that could take advantage of machines' increased capabilities as the field of AI advances. Formalizing this idea is a key open problem. Given a data set which classifies outcomes in terms of some world-model, how can dimensions along which the data set gives little information be identified? One way to approach the problem is to study how humans learn concepts from sparse data, as discussed by  Tenenbaum et al. [2011]  and Sotala  [2015] . Alternatively, it may be possible to find some compact criterion for identifying ambiguities in a simpler fashion. In both cases, further research could prove fruitful.
 Modeling Intent The problem of ambiguity identification may call for methods beyond the inductive learning of value from training data. An intelligent system with a sufficiently refined model of humans may already have the data needed, provided that the right question is asked, to deduce that humans are more likely to care about whether happy-looking human-shaped things have brains than about the nearby breeze. The trouble would be designing the system to use this information in exactly the right way. Picture a system that builds multi-level environment models from sensory data and learns its values inductively. One could then specially demarcate some part of the model as the "model of the operator," define some explicit rules for extracting a model of the operator's preferences from the model of the operator (in terms of possible outcomes), and adjust the ratings on various outcomes in accordance with the model of the operator's preferences. This would be a system which attempts to learn and follow another agent's intentions, as opposed to learning from labeled training data-a "do what I mean" (DWIM) architecture. The inverse reinforcement learning (IRL) techniques of Ng and  Russell [2000]  can be viewed as a DWIM approach, in which an agent attempts to identify and maximize the reward function of some other agent in the environment. However, existing IRL formalizations do not capture the full problem; the preferences of humans cannot necessarily be captured in terms of observations alone. For example, a system, upon observing its operator lose at a game of chess, should not conclude that its operator wanted to lose at chess, even if the system can clearly see where the operator "decided" to make a bad move instead of a good one. Or imagine a human operator who has a friend that must be put into hiding. The learner may either take the friend to safety, or abandon the friend in a dangerous location and use the resources saved in this way to improve the operator's life. If the system reports that the friend is safe in both cases, and the human operator trusts the system, then the latter observation history may be preferred by the operator. However, the latter outcome would definitely not be preferred by most people if they had complete knowledge of the outcomes. Human preferences are complex, multi-faceted, and often contradictory. Safely extracting preferences from a model of a human would be no easy task. Problems of ontology identification recur here: the framework for extracting preferences and affecting outcome ratings needs to be robust to drastic changes in the learner's model of the operator. The specialcase identification of the "operator model" must survive as the system goes from modeling the operator as a simple reward function to modeling the operator as a fuzzy, ever-changing part of reality built out of biological cells-which are made of atoms, which arise from quantum fields. DWIM architectures must avoid a number of other hazards. Suppose the system learns that its operator model affects its outcome ratings, and the system has available to it actions that affect the operator. Actions which manipulate the operator to make their preferences easier to fulfill may then be highly rated, as they lead to highly-rated outcomes (where the system achieves the operator's now-easy goals). Solving this problem is not so simple as forbidding the system from affecting the operator; any query made by the system to the operator in order to resolve some ambiguity will affect the operator in some way. A DWIM architecture requires significant additional complexity on top of inductive value learning: the agent's goaladjusting learning system no longer simply classifies outcomes; it must also model humans and extract human prefer-ences about human-modeled outcomes, and translate between human-modeled future outcomes and future outcomes as modeled by the system. The hope is that this complexity purchases a system that potentially achieves full and direct coverage of the complexity of human value, without relying on the abilities of the programmers to hand-code exceptions for every edge case or compose exactly the right training set. Further investigations into inverse reinforcement learning or other methods of constructing satisfactory initial operator models may be a good place to start studying the plausibility of DWIM architectures.
 Extrapolating Volition A DWIM architecture may be sufficient when constructing a system that reliably pursues "concrete" goals (such as "cure cancer and then await instruction"), but it may not be sufficient for more complex or sophisticated goals where the operators themselves do not know what they intend-for example, "Do what I would want, if I had more knowledge and more time to think." None of the frameworks discussed so far seem powerful enough to specify philosophical ideas like the "ideal advisor theory" of  Rosati [1995]  or the "reflective equilibrium" of  Rawls [1971] . Here, even "indirect" approaches to making robust and beneficial AI systems run aground of actively debated questions in moral philosophy. One possible approach to resolving normative uncertainty (e.g., about what the operators would want if they were wiser or better people) would be to build a DWIM system that takes a model of a human operator and extrapolates it in the direction of e.g. Rawls' reflective equilibrium. For example, the extrapolation might predict what the operator would decide if they knew everything the system knows, or if they had considered many possible moral arguments  [Bostrom, 2014, chap. 13 ]. However, a high-powered system searching for moral arguments that would put the operators into a reflectively stable state (as a computational expedient to fully simulating the operators' process of reflection) introduces a new set of potential pitfalls. A high-powered search for the most persuasive moral arguments that elicit retrospective approval of moral changes might find arguments that induce psychotic breakdowns or religious conversions. The system should be constrained to search for only "valid" moral arguments, but defining what counts as a valid moral argument is itself a major area of normative uncertainty and disagreement. In this domain, querying for ambiguities is difficult. In everyday practice, an argument that is persuasive to smart and skeptical humans is often valid, but a superintelligent search for persuasive arguments may well discover invalid but extremely persuasive arguments. It is difficult to identify technical approaches to indirect normativity that are tractable today, although there have been a few initial forays.  Christiano [2014]  informally proposes one mechanism by which a system could perhaps safely extrapolate the volition of its operator.  MacAskill [2014]  has given an extensive report on "meta-normativity," touching upon many different philosophical aspects of the difficulties of resolving normative uncertainty. This is an area where further philo-sophical study may make it clearer how to begin approaching the associated long-run engineering problems.
 Discussion Just as human intelligence has allowed us to develop tools and strategies by which we can control our environment, so too could superintelligent systems develop tools and strategies more powerful than our own, and gain correspondingly greater control over future outcomes  [Bostrom, 2014, chap. 6 ]. Although it is not clear how long the development of smarterthan-human systems will take, or what approaches in AI or other disciplines may prove most relevant to developing such systems, early efforts in this area are justified by its importance and neglectedness. In the introduction to this paper, we discussed four different ways in which the potential development of superintelligent machines changes the task of AI safety and ethics work. Addressing all these concerns does not seem easy. Designs for AI systems that are intended to become superintelligent will need to be corrigible in the sense of  Soares and Fallenstein [2015] , i.e., willing to assist their operators in attempted corrections. The systems will need some method for learning and adopting prosocial preferences, in light of the fact that we cannot expect arbitrary rational actors to exhibit prosocial behavior in the face of large power disparities. Operators will require methods for robustly communicating their intentions to the system, if Sorcerer's Apprentice scenarios are to be avoided. And eventually, explicit methodologies for resolving normative uncertainty may be required. This paper has given a cursory overview of a number of potential lines of research for AI value specification. We discuss these ideas in part to give an overview of plausible approaches to the concerns outlined above, and also because these are topics that seem amenable to research starting sooner rather than later, even in the face of great uncertainty about the particular architectures of future AI systems. It is difficult to know which lines of research will pan out, and we hope that this survey inspires research along a number of new paths, so that we have a firm theoretical grasp of how systems could reliably and safely learn our values in principle before it comes time to build systems that must do so in practice. 			  Bostrom's "orthogonality thesis" can be treated as an application ofHume's [1739]  observation that natural-language "is" and "ought" claims are independent.
			 Instead of trying to direct the system toward exactly the right hyperface, one might try to create a "limited optimization" system that doesn't push so hard in whatever direction it moves. This seems like a promising research avenue, but is beyond the scope of this paper.

Abstract: Autonomous AI systems' programmed goals can easily fall short of programmers' intentions. Even a machine intelligent enough to understand its designers' intentions would not necessarily act as intended. We discuss early ideas on how one might design smarter-than-human AI systems that can inductively learn what to value from labeled training data, and highlight questions about the construction of systems that model and act upon their operators' preferences.


David Fridovich-Keil
Andrea Bajcsy
Jaime F Fisac
Sylvia L Herbert
Steven Wang
Anca D Dragan
Claire J Tomlin

Confidence-aware motion prediction for real-time collision avoidance

Introduction Motion planning serves a key role in robotics, enabling robots to automatically compute trajectories that achieve the specified objectives while avoiding unwanted collisions. In many situations of practical interest, such as autonomous driving and unmanned aerial vehicle (UAV) navigation, it is important that motion planning account not just for the current state of the environment, but also for its predicted future state. Often, certain objects in the environment may move in active, complex patterns that cannot be readily predicted using straightforward physics models; we shall refer to such complex moving objects as agents. Examples of agents considered in this paper include pedestrians and human-driven vehicles. Predicting the future state of these agents is generally a difficult problem. Some of the key challenges include unclear and varying intents of other agents, mismatches between dynamics models and reality, incomplete sensor information, and interaction effects. One popular approach to addressing the challenge of a priori unknown agent intent is to use rule-based or data-driven algorithms to predict individual trajectories for each agent, as in  Schmerling et al. (2017) . Alternatively,  Ziebart et al. (2009) ,  Bandyopadhyay et al. (2013) , and  Kochenderfer et al. (2010)  explicitly predict an agent's full state distribution over time; this representation may be better suited to capturing uncertainty in an agent's dynamics and the environment itself.  Wang et al. (2019)  and  Fisac et al. (2018b)  pose the prediction problem game-theoretically to model coupled human-robot interaction effects explicitly. Unfortunately, a significant problem still remains: if an agent suddenly moves in a way that is not predicted, or not assigned sufficient probability, the robot may not react appropriately. For example, in Figure  1  a pedestrian is walking around an obstacle that the robot, a quadcopter, cannot detect. To the robot, such behavior may be assigned very low probability, which could lead the robot to plan a dangerous trajectory. In this particular example, this inaccuracy caused the quadcopter to collide with the pedestrian (Figure  1 , left). To prepare for this eventuality, we introduce the idea of confidence-aware prediction. We argue that, in addition to predicting the future state of an agent, it is also crucial for a robot to assess the quality of the mechanism by which it is generating those predictions. That is, a robot should reason about how confident it is in its predictions of other agents before attempting to plan future motion. For computational efficiency, the quadcopter uses a simplified model of pedestrian dynamics and decision-making. Thus equipped, it generates a time-varying probability distribution over the future state of the pedestrian, and plans trajectories to a pre-specified goal that maintain a low probability of collision. Figure  1 (right) illustrates how this approach works in practice. The quadcopter maintains a Bayesian belief over its prediction confidence. As soon as the pedestrian moves in a way that was assigned low probability by the predictive model, the quadcopter adjusts its belief about the accuracy of that model. Consequently, it is less certain about what the pedestrian will do in the future. This leads the quadcopter's onboard motion planner, which attempts to find efficient trajectories with low probability of collision, to generate more cautious, and perhaps less efficient, motion plans. In order to improve the robustness of generated motion plans, we employ the recent FaSTrack framework from  Herbert et al. (2017)  for fast and safe motion planning and tracking. FaSTrack quantifies the maximum possible tracking error between a high-order dynamical model of the physical robot and the (potentially lower-order) dynamical model used by its motion planner. Solving an offline Hamilton-Jacobi reachability problem yields a guaranteed tracking error bound and the corresponding safety controller. These may be used by an out-of-the-box real-time motion planning algorithm to facilitate motion plans with strong runtime collision-avoidance guarantees. The remainder of this paper is organized as follows. Section 2 places this work in the context of existing literature in human motion modeling and prediction, as well as robust motion planning. Section 3 frames the prediction and planning problems more formally, and introduces a running example used throughout the paper. Section 4 presents our main contribution: confidence-aware predictions. Section 5 showcases confidence-aware predictions in operation in several examples. Section 6 describes the application of the robust motion planning framework from FaSTrack to this setting, in which predictions are probabilistic. Section 7 explores a connection between our approach and reachability theory. Section 8 presents experimental results from a hardware demonstration. Finally, Section 9 concludes with a discussion of some of the limitations of our work and how they might be addressed in specific applications, as well as suggestions for future research.   1 . When planning around humans, accurate predictions of human motion (visualized here in pink and blue, representing high and low probability, respectively) are an essential prerequisite for safety. Unfortunately, these approaches may fail to explain all observed motion at runtime (e.g., human avoids unmodeled spill on the ground), leading to inaccurate predictions, and potentially, collisions (left). Our method addresses this by updating its predictive model confidence in real time (right), leading to more conservative motion planning in circumstances when predictions are known to be suspect.
 Prior work 2
 Human modeling and prediction One common approach for predicting human actions is to collect data from real-world scenarios and train a machine learning model via supervised learning. Such techniques use the human's current state, and potentially her prior state and action history, to predict future actions directly.  Amor et al. (2014) ,  Ding et al. (2011) ,  Koppula and Saxena (2013) ,  Lasota and Shah (2015) , and  Hawkins et al. (2013)  demonstrated the effectiveness of this approach for inference and planning around human arm motion. In addition,  Hawkins et al. (2013)  focused on multi-step tasks such as assembly, and  Schmerling et al. (2017)  and  Driggs-Campbell et al. (2018)  addressed the prediction problem for human drivers. Rather than predicting actions directly, an alternative is for the robot to model the human as a rational agent seeking to maximize an unknown objective function. The human's actions up to a particular time may be viewed as Bayesian evidence from which the robot may infer the parameters of that objective. Assuming that the human seeks to maximize this objective in the future, the robot can predict her future movements (e.g.,  Bai et al., 2015; Baker et al., 2007; Ng and Russell, 2000; Ziebart et al., 2009) . In this paper, we build on this work by introducing a principled online technique for estimating confidence in such a learned model of human motion.
 Safe robot motion planning Once armed with a predictive model of the human motion, the robot may leverage motion planning methods that plan around uncertain moving obstacles and generate real-time dynamically feasible and safe trajectories. To avoid moving obstacles in real time, robots typically employ reactive and/or path-based methods. Reactive methods directly map sensor readings into control, with no memory involved (e.g.,  Belkhouche, 2009) . Path-based methods such as rapidly-exploring random trees from  Karaman and Frazzoli (2011)  and A* from  Hart et al. (1968)  find simple kinematic paths through space and, if necessary, time. These path-based methods of planning are advantageous in terms of efficiency, yet, while they have in some cases been combined with probabilistically moving obstacles as in  Aoude et al. (2013)  and  Ziebart et al. (2009) , they do not consider the endogenous dynamics of the robot or exogenous disturbances such as wind. As a result, the robot may deviate from the planned path and potentially collide with obstacles. It is common for these plans to try to avoid obstacles by a heuristic margin of error.  Herbert et al. (2017) and Fridovich-Keil et al. (2018)  proposed FaSTrack, a recent algorithm that provides a guaranteed tracking error margin and corresponding errorfeedback controller for dynamic systems tracking a generic planner in the presence of bounded external disturbance. Our work builds upon FaSTrack to create an algorithm that can safely and dynamically navigate around uncertain moving obstacles in real time.
 Problem setup We consider a single mobile robot operating in a shared space with a single human agent (e.g., a pedestrian or human-driven car). For simplicity, we presume that the robot has full knowledge of its own state and that of the human, although both would require online estimation in practice. As we present each formal component of this problem, we will provide a concrete illustration using a running example in which a quadcopter is navigating around a pedestrian.
 Dynamical system models and safety We will model the motion of both the human and the robot as the evolution of two dynamical systems. Let the state of the human be x H 2 R n H , where n H is the dimension of the human state space. We similarly define the robot's state, for planning purposes, as x R 2 R n R . In general, these states could represent the positions and velocities of a mobile robot and a human in a shared environment, the kinematic configurations of a human and a robotic manipulator in a common workspace, or the positions, orientations, and velocities of human-driven and autonomous vehicles in an intersection. We express the evolution of these states over time as a family of ordinary differential equations: _ x H = f H (x H , u H ), _ x R = f R (x R , u R ) ð1Þ where u H 2 R m H and u R 2 R m R are the control actions of the human and robot, respectively. Running example: We introduce a running example for illustration purposes throughout the paper. In this example we consider a small quadcopter that needs to fly to goal location g R 2 R 3 in a room where a pedestrian is walking. For the purposes of planning, the quadcopter's 3D state is given by its position in space x R = ½p x , p y , p z , with velocity controls assumed decoupled in each spatial direction, up to v R = 0:25 m/s. The human can only move by walking and therefore her state is given by planar coordinates x H = ½h x , h y evolving as _ x H = ½v H cos u H , v H sin u H . Intuitively, we model the human as moving with a fixed speed and controlling their heading angle. At any given time, the human is assumed to either move at a leisurely walking speed (v H '1 m/s) or remain still (v H '0). Ultimately, the robot needs to plan and execute an efficient trajectory to a pre-specified goal state (g R ), without colliding with the human. We define the keep-out set K & R n H × R n R as the set of joint robot-human states to be avoided (for example, because they imply physical collisions). To avoid reaching this set, the robot must reason about the human's future motion when constructing its own motion plan. Running example: In our quadcopter-avoidingpedestrian example, K consists of joint robot-human states in which the quadcopter is flying within a square of side length l = 0:3 m centered around the human's location, while at any altitude, as well as any joint states in which the robot is outside the environment bounds defined as a box with a square base of side L = 3:66 m and height H = 2 m, regardless of the human's state.
 Robust robot control Provided an objective and a dynamics model, the robot must generate a motion plan that avoids the keep-out set K. Unfortunately, this safety requirement is difficult to meet during operation for two main reasons. 1. Model mismatch. The dynamical system model f R will never be a perfect representation of the real robot. This mismatch could lead to unintended collision. 2. Disturbances. Even with a perfect dynamics model, there may be unobserved, external ''disturbance'' inputs such as wind or friction. Without accounting for these disturbances, the system is not guaranteed to avoid K, even if the planned trajectory is pointwise collision-free. To account for modeling error and external disturbances, we could in principle design a higher-fidelity dynamical model directly in a robust motion planning framework. Unfortunately, however, real-time trajectory optimization in high dimensions can be computationally burdensome, particularly when we also require some notion of robustness to external disturbance. Ideally, we would like to enjoy the computational benefits of planning with a lower-fidelity model while enforcing the safety constraints induced by the higher-fidelity model. To characterize this model mismatch, we consider a higher fidelity and typically higher-order dynamical representation of the robot, with state representation s R 2 R n S . This dynamical model will also explicitly account for external disturbances as unknown bounded inputs, distinct from control inputs. In order to map between this higher-fidelity ''tracking'' state s R and the lower-fidelity ''planning'' state x R , we shall assume a known projection operator p : R n S ! R n R . Fortunately, we can plan in the lower-dimensional state space at runtime, and guarantee robust collision avoidance via an offline reachability analysis that quantifies the effects of model mismatch and external disturbance. This framework, called FaSTrack and first proposed by  Herbert et al. (2017) , is described in further detail in Section 6. Running example: We model our quadcopter with the following flight dynamics (in the near-hover regime, at zero yaw with respect to a global coordinate frame): _ p x _ p y _ p z 2 4 3 5 = v x v y v z 2 4 3 5 , _ v x _ v y _ v z 2 4 3 5 = a g tan u u Àa g tan u f u T À a g 2 4 3 5 ð2Þ where ½p x , p y , p z is the quadcopter's position in space and ½v x , v y , v z is its velocity expressed in the fixed global frame. We model its control inputs as thrust acceleration u T and attitude angles (roll u f and pitch u u ), and denote the acceleration due to gravity as a g . The quadcopter's motion planner generates nominal kinematic trajectories in the lowerdimensional ½p x , p y , p z position state space. Therefore, we have a linear projection map p(s R ) = ½I 3 , 0 3 s R , that is, x R retains the position variables in s R and discards the velocities.
 Predictive human model In order to predict the human's future motion, the robot uses its internal model of human dynamics, f H . Under this modeling assumption, the human's future trajectory depends upon the choice of control input over time, u H ( Á ). Extensive work in econometrics and cognitive science, such as that of  Von Neumann and Morgenstern (1945) ,  Luce (1959), and Baker et al. (2007) , has shown that human behavior (that is, u H ) can be well modeled by utility-driven optimization. Thus, the robot models the human as optimizing a reward function, r H (x H , u H ; u), that depends on the human's state and action, as well as a set of parameters u. This reward function could be a linear combination of features as in many inverse optimal control implementations (where the goal or feature weighting u must be learned, either online or offline), or more generally learned through function approximators such as deep neural networks, where u are the trained weights as in  Finn et al. (2016) . We assume that the robot has a suitable human reward function r H , either learned offline from prior human demonstrations or otherwise encoded by the system designers. Thus, endowed with r H , the robot can model the human's choice of control action as a probability distribution over actions conditioned on state. Under maximumentropy assumptions  (Ziebart et al., 2008)  inspired by noisy-rationality decision-making models  (Baker et al., 2007) , the robot models the human as more likely to choose (discrete) actions u H with high expected utility, in this case the state-action value (or Q-value): P(u H jx H ; b, u) = e bQ H (x H , u H ;u) P ũ e bQ H (x H , ũ;u) ð3Þ We use a temporally and spatially discretized version of human dynamics, fH . These discrete-time dynamics may be found by integrating f H over a fixed time step of Dt with fixed control u H over the interval. Section 5 provides further details on this discretization. Running example: The quadcopter's model of the human assumes the human intends to reach some target location g H 2 R 2 in a straight line. The human's reward function is given by the distance traveled over time step Dt, i.e., r H (x H , u H ; g H ) = À v H Dt, and human trajectories are constrained to terminate at g H . The state-action value, parameterized by u = g H , captures the optimal cost of reaching g H from x H when initially applying u H for a duration Dt: Q H (x H , u H ; g H ) = À v H DtÀ k x H + v H Dt½cos u H , sin u H > À g H k 2 : Often, the coefficient b is termed the rationality coefficient, because it quantifies the degree to which the robot expects the human's choice of control to align with its model of utility. For example, taking b # 0 yields a model of a human who appears ''irrational,'' choosing actions uniformly at random and completely ignoring the modeled utility. At the other extreme, taking b " ' corresponds to a ''perfectly rational'' human, whose actions exactly optimize the modeled reward function. As we will see in Section 4, b can also be viewed as a measure of the robot's confidence in the predictive accuracy of Q H . Note that Q H (x H , u H ; u) only depends on the human state and action and not on the robot's. Thus far, we have intentionally neglected discussion of human-robot interaction effects. These effects are notoriously difficult to model, and the community has devoted a significant effort to building and validating a variety of models (e.g.,  Sadigh et al., 2016; Trautman and Krause, 2010) . In that spirit, we could have chosen to model human actions u H as dependent upon robot state x R in (3), and likewise defined Q H to depend upon x R . This extended formulation is sufficiently general as to encompass all possible (Markov) interaction models. However, in this work we explicitly do not model these interactions; indeed, one of the most important virtues of our approach is its robustness to precisely these sorts of modeling errors.
 Probabilistically safe motion planning Ideally, the robot's motion planner should generate trajectories that reach a desired goal state efficiently, while maintaining safety. More specifically, in this context ''safety'' indicates that the physical system will never enter the keepout set K during operation, despite human motion and external disturbances. That is, we would like to guarantee that (p(s R ), x H ) 6 2 K for all time. To make this type of strong, deterministic, a priori safety guarantee requires the robot to avoid the set of all human states x H which could possibly be occupied at a particular time, i.e., the human's forward reachable set. If the robot can find trajectories that are safe for any possible human trajectory then there is no need to predict the human's next action. Unfortunately, the forward reachable set of the human often encompasses such a large volume of the workspace that it is impossible for the robot to find a guaranteed safe trajectory to the goal state. This motivates refining our notion of prediction: rather than reasoning about all the places where the human could be, the robot can instead reason about how likely the human is to be at each location. This probabilistic reasoning provides a guide for planning robot trajectories with a quantitative degree of safety assurance. Our probabilistic model of human control input (3) coupled with dynamics model f H allows us to compute a probability distribution over human states for every future time. By relaxing our conception of safety to consider only collisions that might occur with sufficient probability P th , we dramatically reduce the effective volume of this set of future states to avoid. In practice, P th should be chosen carefully by a system designer in order to trade off overall collision probability with conservativeness in motion planning. The proposed approach in this paper follows two central steps to provide a quantifiable, high-confidence collision avoidance guarantee for the robot's motion around the human. In Section 4 we present our proposed Bayesian framework for reasoning about the uncertainty inherent in a model's prediction of human behavior. Based on this inference, we demonstrate how to generate a real-time probabilistic prediction of the human's motion over time. Next, in Section 6, we extend a state-of-the-art, provably safe, real-time robotic motion planner to incorporate our time-varying probabilistic human prediction.
 Confidence-aware human motion prediction Any approach to human motion prediction short of computing a full forward reachable set must, explicitly or implicitly, reflect a model of human decision-making. In this work, we make that model explicit by assuming that the human chooses control actions in a Markovian fashion according to the probability distribution (3). Other work in the literature, such as that of  Schmerling et al. (2017) , aims to learn a generative probabilistic model for human trajectories; implicitly, this training procedure distills a model of human decision-making. Whether explicit or implicit, these models are by nature imperfect and liable to make inaccurate predictions eventually. One benefit of using an explicit model of human decision-making, such as (3), is that we may reason directly and succinctly about its performance online. In particular, the entropy of the human control distribution in (3) is a decreasing function of the parameter b. High values of b place more probability mass on highutility control actions u H , whereas low values of b spread the probability mass more evenly between different control inputs, regardless of their modeled utility Q H . Therefore, b naturally quantifies how well the human's motion is expected to agree with the notion of optimality encoded in Q H . The commonly used term ''rationality coefficient,'' however, seems to imply that discrepancies between the two indicate a failure on the human's part to make the ''correct'' decisions, as encoded by the modeled utility. Instead, we argue that these inevitable disagreements are primarily a result of the model's inability to fully capture the human's behavior. Thus, instead of conceiving of b as a rationality measure, we believe that b can be given a more pragmatic interpretation related to the accuracy with which the robot's model of the human is able to explain the human's motion. Consistently, in this paper, we refer to b as model confidence. An important related observation following from this interpretation of b is that the predictive accuracy of a human model is likely to change over time. For example, the human may change their mind unexpectedly, or react suddenly to some aspect of the environment that the robot is unaware of. Therefore, we shall model b as an unobserved, time-varying parameter. Estimating it in real time provides us with a direct, quantitative summary of the degree to which the utility model Q H explains the human's current motion. To do this, we maintain a Bayesian belief about the possible values of b. Initially, we begin with a uniform prior over b and over time this distribution evolves given measurements of the human's state and actions.
 Real-time inference of model confidence We reason about the model confidence b as a hidden state in a hidden Markov model (HMM) framework. The robot starts with a prior belief b 0 À over the initial value of b. In this work, we use a uniform prior, although that is not strictly necessary. At each discrete time step k 2 f0, 1, 2, . . .g, it will have some belief about model confidence b k À (b). 3 After observing a human action u k H , the robot will update its belief to b k + by applying Bayes' rule. The hidden state may evolve between subsequent time steps, accounting for the important fact that the predictive accuracy of the human model may change over time as unmodeled factors in the human's behavior become more or less relevant. As, by definition, we do not have access to a model of these factors, we use a naive ''e-static'' transition model: at each time k, b may, with some probability e, be re-sampled from the initial distribution b 0 À , and otherwise retains its previous value. We define the belief over the next value of b (denoted by b 0 ) as an expectation of the conditional probability P(b 0 jb), i.e., b k À (b 0 ) :¼ E b;b kÀ1 + ½P(b 0 jb). Concretely, this expectation may be computed as b k À (b 0 ) = (1 À e)b kÀ1 + (b 0 ) + eb 0 À (b 0 ) ð4Þ By measuring the evolution of the human's state x H over time, we assume that, at every time step k, the robot is able to observe the human's control input u k H . This observed control may be used as evidence to update the robot's belief b k À about b over time via a Bayesian update: b k + (b) = P(u k H jx k H ; b, u)b k À (b) P b P(u k H jx k H ; b, u)b k À ( b) ð5Þ with b k + (b) :¼ P(bjx 0:k H , u 0:k H ) for k 2 f0, 1, . . .g, and P(u k H jx k H ; b, u) given by (3). It is critical to be able to perform this update rapidly to facilitate real-time operation; this would be difficult in the original continuous hypothesis space b 2 ½0, '), or even in a large discrete set. Fortunately, our software examples in Section 5 and hardware demonstration in Section 8 suggest that maintaining a Bayesian belief over a relatively small set of N b = 5 discrete values of b distributed on a log scale achieves significant improvement relative to using a fixed value. The ''e-static'' transition model leads to the desirable consequence that old observations of the human's actions have a smaller influence on the current model confidence distribution than recent observations. In fact, if no new observations are made, successively applying time updates asymptotically contracts the belief towards the initial distribution, that is, b k À ( Á ) ! b 0 À ( Á ). The choice of parameter e effectively controls the rate of this contraction, with higher e leading to more rapid contraction.
 Human motion prediction Equipped with a belief over b at time step k, we are now able to propagate the human's state distribution forward to any future time via the well-known Kolmogorov forward equations, recursively. In particular, suppose that we know the probability that the human is in each state x k H at some future time step k. We know that (according to our utility model) the probability of the human choosing control u k H in state x k H is given by (3). Accounting for the otherwise deterministic dynamics model fH , we obtain the following expression for the human's state distribution at the following time step k + 1: P(x k + 1 H ; b, u) = X x k H , u k H P(x k + 1 H jx k H , u k H ; b, u) Á P(u k H jx k H ; b, u)P(x k H ; b, u) ð6Þ for a particular choice of b. Marginalizing over b according to our belief at the current step time k, we obtain the overall occupancy probability distribution at each future time step k: P(x k H ; u) = E b;b k P(x k H ; b, u) ð7Þ Note that (  6 ) is expressed more generally than is strictly required. Indeed, because the only randomness in dynamics model fH originates from the human's choice of control input u H , we have P( x k + 1 H jx k H , u k H ; b, u) = 1fx k + 1 H = fH (x k H , u k H )g.
 Model confidence with auxiliary parameter identification Thus far, we have tacitly assumed that the only unknown parameter in the human utility model (  3 ) is the model confidence, b. However, often one or more of the auxiliary parameters u are also unknown. These auxiliary parameters could encode one or more human goal states or intents, or other characteristics of the human's utility, such as her preference for avoiding parts of the environment. Further, much like model confidence, they may change over time. In principle, it is possible to maintain a Bayesian belief over b and u jointly. The Bayesian update for the hidden state (b, u) is then given by ) the prior at time step k. This approach can be practical for parameters taking finitely many values from a small, discrete set, e.g., possible distinct modes for a human driver (distracted, cautious, aggressive). However, for certain scenarios or approaches it may not be practical to maintain a full Bayesian belief on the parameters u. In such cases, it is reasonable to replace the belief over u with a point estimate u, such as the maximum likelihood estimator or the mean, and substitute that estimate into (6). Depending on the complexity of the resulting maximum likelihood estimation problem, it may or may not be computationally feasible to update the parameter estimate u at each time step. Fortunately, even when it is computationally expensive to estimate u, we can leverage our model confidence as an indicator of when reestimating these parameters may be most useful. That is, when model confidence degrades that may indicate poor estimates of u. b k + (b, u) = P(u k H jx k H ; b, u)b k À (b, u) P b, ũ P(u k H jx k H ; b, ũ)b k À ( b, ũ) ð8Þ
 Prediction examples We illustrate these inference steps with two sets of examples: our running pedestrian example and a simple model of a car.
 Pedestrian model (running example) So far, we have presented a running example of a quadcopter avoiding a human. We use a deliberately simple, purely kinematic model of continuous-time human motion: _ x H = _ h x _ h y ! = v H cos u H v H sin u H ! ð9Þ However, as discussed in Section 3.3, the proposed prediction method operates in discrete time (and space). The discrete dynamics corresponding to (9) are given by x k + 1 H À x k H [x H (t + Dt) À x H (t) = v H Dt cos u H (t) v H Dt sin u H (t) ! ð10Þ for a time discretization of Dt.
 Dubins car model To emphasize the generality of our method, we present similar results for a different application domain: autonomous driving. We will model a human-driven vehicle as a dynamical system whose state x H evolves as _ x H = _ h x _ h y _ h f 2 4 3 5 = v H cos h f v H sin h f u H 2 4 3 5 ð11Þ Observe that, while (11) appears very similar to (9), in this Dubins car example the angle of motion is a state, not a control input. We discretize these dynamics by integrating (11) from t to t + Dt, assuming a constant control input u H : x k + 1 H À x k H [x H (t + Dt) À x H (t) = v H u H (t) sin (h f (t) + u H (t)Dt) À sin (h f (t)) À Á À v H u H (t) cos (h f (t) + u H (t)Dt) À cos (h f (t)) À Á u H Dt 2 6 4 3 7 5 For a specific goal position g = ½g x , g y , the Q-value corresponding to state-action pair (x H , u H ) and reward function r H (x H , u H ) = À v H Dt (until the goal is reached) may be found by solving a shortest path problem offline.
 Accurate model First, we consider a scenario in which the robot has full knowledge of the human's goal, and the human moves along the shortest path from a start location to this known goal state. Thus, human motion is well-explained by Q H . The first row of Figure  2  illustrates the probability distributions our method predicts for the pedestrian's future state at different times. Initially, the predictions generated by our Bayesian confidence-inference approach (right) appear similar to those generated by the low model confidence predictor (left). However, our method rapidly discovers that Q H is an accurate description of the pedestrian's motion and generates predictions that match the high model confidence predictor (center). The data used in this example was collected by tracking the motion of a real person walking in a motion capture arena. See Section 8 for further details. Likewise, the first row of Figure  3  shows similar results for a human-driven Dubins car model (in simulation) at an intersection. Here, traffic laws provide a strong prior on the human's potential goal states. As shown, our method of Bayesian model confidence inference quickly infers the correct goal and learns that the human driver is acting in accordance with its model Q H . The resulting predictions are substantially similar to the high-b predictor. The data used in this example was simulated by controlling a Dubins car model along a pre-specified trajectory.
 Unmodeled obstacle Often, robots do not have fully specified models of the environment. Here, we showcase the resilience of our approach to unmodeled obstacles that the human must avoid. In this scenario, the human has the same start and goal as in the accurate model case, except that there is an obstacle along the way. The robot is unaware of this obstacle, however, which means that in its vicinity the human's motion is not well-explained by Q H , and b(b) ought to place more probability mass on higher values of b. The second rows of Figure  2  and Figure  3  illustrate this type of situation for the pedestrian and Dubins car, respectively. In Figure  2 , the pedestrian walks to an a priori known goal location and avoids an unmodeled spill on the ground. Analogously, in Figure  3  the car swerves to avoid a large pothole. By inferring model confidence online, our approach generates higher-variance predictions of future state, but only in the vicinity of these unmodeled obstacles. At other times throughout the episode when Q H is more accurate, our approach produces predictions more in line with the high model confidence predictor.
 Unmodeled goal In most realistic human-robot encounters, even if the robot does have an accurate environment map and observes all obstacles, it is unlikely for it to be aware of all human goals. We test our approach's resilience to unknown human goals by constructing a scenario in which the human moves between both known and unknown goals. The third row of Figure  2  illustrates this situation for the pedestrian example. Here, the pedestrian first moves to one known goal position, then to another, and finally back to the start which was not a modeled goal location. The first two legs of this trajectory are consistent with the robot's model of goal-oriented motion, though accurate prediction does require the predictor to infer which goal the pedestrian is walking toward. However, when the pedestrian returns to the start, her motion appears inconsistent with Q H , skewing the robot's belief over b toward zero. Similarly, in the third row of Figure  3  we consider a situation in which a car makes an unexpected turn onto an unmapped access road. As soon as the driver initiates the turn, our predictor rapidly learns to distrust its internal model Q H and shift its belief over b upward.
 Safe probabilistic planning and tracking Given probabilistic predictions of the human's future motion, the robot must plan efficient trajectories that avoid collision with high probability. In order to reason robustly about this probability of future collision, we must account for potential tracking errors incurred by the real system as it follows planned trajectories. To this end, we build on the recent FaSTrack framework of  Herbert et al. (2017) , which provides control-theoretic robust safety certificates in the presence of deterministic obstacles, and extend it to achieve approximate probabilistic collision-avoidance.
 Background: fast planning, safe tracking Recall that x R is the robot's state for the purposes of motion planning, and that s R encodes a higher-fidelity, potentially higher-dimensional notion of state (with associated dynamics). The recently proposed FaSTrack framework from  Herbert et al. (2017)  uses Hamilton-Jacobi reachability analysis to quantify the worst-case tracking performance of the s R -system as it follows trajectories generated by the x R -system. For further reading on reachability analysis refer to  Evans and Souganidis (1984) ,  Mitchell et al. (2005) , and  Bansal et al. (2017) . A byproduct of this FaSTrack analysis is an error feedback controller that the s R system can use to achieve this worst-case tracking error. The tracking error bound may be given to one of many off-the-shelf real-time motion planning algorithms operating in x R -space in order to guarantee real-time collision avoidance by the s Rsystem. Formally, FaSTrack precomputes an optimal tracking controller, as well as a corresponding compact set E in the robot's planning state space, such that (p(s R (t))À x R, ref (t)) 2 E for any reference trajectory proposed by the lower-fidelity planner. This bound E is a trajectory tracking certificate that can be passed to an online planning algorithm for real-time safety verification: the dynamical robot is guaranteed to always be somewhere within the bound relative to the current planned reference point x R, ref (t). This tracking error bound may sometimes be expressed analytically; otherwise, it may be computed numerically offline using level set methods (e.g.,  Mitchell, 2009) . Equipped with E, the planner can generate safe plans online by ensuring that the entire tracking error bound around the nominal state remains collision-free throughout the trajectory. Efficiently checking these E-augmented trajectories for collisions with known obstacles is critical for real-time performance. Note that the planner only needs to know E (which is computed offline) and otherwise requires no explicit understanding of the high-fidelity model. Running example: As dynamics (2) are decoupled in the three spatial directions, the bound E computed by FaSTrack is an axis-aligned box of dimensions E x × E y × E z . For further details refer to Fridovich-Keil et al. (  2018 ).
 Robust tracking, probabilistic safety Unfortunately, planning algorithms for collision checking against deterministic obstacles cannot be readily applied to our problem. Instead, a trajectory's collision check should return the probability that it might lead to a collision. Based on this probability, the planning algorithm can discriminate between trajectories that are sufficiently safe and those that are not. As discussed in Section 3.4, a safe online motion planner invoked at time t should continually check the probability that, at any future time t, (p(s R (t)), x H (t)) 2 K. The tracking error bound guarantee from FaSTrack allows us to conduct worst-case analysis on collisions given a human state x H . Concretely, if no point in the Minkowski sum fx R + Eg is in the collision set with x H , we can guarantee that the robot is not in collision with the human. The probability of a collision event for any point x R (t) along a candidate trajectory is then P coll (x R (t)) :¼ P((x R , x H ) 2 K) ð12Þ Assuming worst-case tracking error bound E, this quantity can be upper-bounded by the total probability that x H (t) will be in collision with any of the possible robot states xR 2 fx R (t) + Eg. For each robot planning state x R 2 R n R we define the set of human states in potential collision with the robot: H E (x R ) :¼ fx H 2 R n H : 9x R 2 fx R + Eg, (x R , xH ) 2 Kg ð13Þ Running example: Given K and E, H E (x R ) is the set of human positions within the rectangle of dimensions (l + E x ) × (l + E y ) centered on ½p x , p y . A human anywhere in this rectangle could be in collision with the quadcopter. The following result follows directly from the definition of the tracking error bound and a union bound. Proposition 1. The probability of a robot with worst-case tracking error E colliding with the human at any trajectory point x R (t) is bounded above by the probability mass of x H (t) contained within H E (x R (t)). We consider discrete-time motion plans. The probability of collision along any such trajectory from current time step k to final step k + K is upper-bounded by P k:k + K coll ł P k:k + K coll :¼ 1 À Y k + K k = k P(x k H 6 2 H E (x k R )jx k H 6 2 H E (x s R ), k ł s\k) ð14Þ Evaluating the right-hand side of (  14 ) exactly requires reasoning about the joint distribution of human states over all time steps and its conditional relationship on whether collision has yet occurred. This is equivalent to maintaining a probability distribution over the exponentially large space of trajectories x k:k + K H that the human might follow. As The International Journal of Robotics Research 39(2-3) motion planning occurs in real time, we shall resort to a heuristic approximation of (  14 ). One approach to approximating (  14 ) is to assume that the event x k 1 H 6 2 H E (x k 1 R ) is independent of x k 2 H 6 2 H E (x k 2 R ), for all k 1 6 ¼ k 2 . This independence assumption is equivalent to removing the conditioning in (  14 ). Unfortunately, this approximation is excessively pessimistic; if there is no collision at time step k, then collision is also unlikely at time step k + 1 because both human and robot trajectories are continuous. In fact, for sufficiently small time discretization Dt and nonzero collision probabilities at each time step, the total collision probability resulting from an independence assumption would approach 1 exponentially fast in the number of time steps K. We shall refine this approximation by finding a tight lower bound on the right-hand side of (  14 ). Because collision events are correlated in time, we first consider replacing each conditional probability P( x k H 6 2 H E (x k R )jx s H 6 2 H E (x s R ), k ł s\k) by 1 for all k 2 fk + 1, . . . , k + Kg. This effectively lower bounds P k:k + K coll by the worst-case probability of collision at the current time step k: P k:k + K coll ø 1 À P(x k H 6 2 H E (x k R )) = P(x k H 2 H E (x k R )) ð15Þ This bound is extremely loose in general, because it completely ignores the possibility of future collision. However, note that probabilities in the product in (  14 ) may be conditioned in any particular order (not necessarily chronological). This commutativity allows us to generate K À k + 1 lower bounds of the form P k:k + K coll ø P(x k H 2 H E (x k R )) for k 2 fk, . . . , k + Kg. Taking the tightest of all of these bounds, we can obtain an informative, yet quickly computable, approximator for the sought probability: P k:k + K coll ø max k2fk:k + Kg P(x k H 2 H E (x k R ))'P k:k + K coll ð16Þ To summarize, the left inequality in (  16 ) lower-bounds P k:k + K coll with the greatest marginal collision probability at any point in the trajectory. On the right-hand side of (  16 ), we take this greatest marginal collision probability as an approximator of the actual probability of collision over the entire trajectory. In effect, we shall approximate P k:k + K coll with a tight lower bound of an upper bound. While this type of approximation may err on the side of optimism, we note that both the robot's ability to replan over time and the fact that the left-hand side of (  16 ) is an upper bound on total trajectory collision probability mitigate this potentially underestimated risk.
 Safe online planning under uncertain human predictions This approximation of collision probability allows the robot to discriminate between valid and invalid candidate trajectories during motion planning. Using the prediction methodology proposed in Section 4, we may quickly generate, at every time t, the marginal probabilities in (  16 ) at each future time k 2 fk, . . . , k + Kg, based on past observations at times 0, . . . , k. The planner then computes the instantaneous probability of collision P(x k H 2 H E (x k R )) by integrating P(x t H jx 0:k H ) over H E (x k R ) , and rejects the candidate point x k R if this probability exceeds P th . Note that for graph-based planners that consider candidate trajectories by generating a graph of time-stamped states, rejecting a candidate edge from this graph is equivalent to rejecting all further trajectories that would contain that edge. This early rejection rule is consistent with the proposed approximation (16) of P k:k + K coll while preventing unnecessary exploration of candidate trajectories that would ultimately be deemed unsafe. Throughout operation, the robot follows each planned trajectory using the error feedback controller provided by FaSTrack, which ensures that the robot's high-fidelity state representation s R and the lower-fidelity state used for planning, x R , differ by no more than the tracking error bound E. This planning and tracking procedure continues until the robot reaches its desired goal state. Running example: Our quadcopter is now required to navigate to a target position shown in Figure  4  without colliding with the human. Our proposed algorithm successfully avoids collisions at all times, replanning to leave greater separation from the human whenever her motion departs from the model. In contrast, robot planning with fixed model confidence is either overly conservative at the expense of time and performance or overly aggressive at the expense of safety.
 Connections to reachability analysis In this section, we present an alternative, complementary analysis of the overall safety properties of the proposed approach to prediction and motion planning. This discussion is grounded in the language of reachability theory and worst-case analysis of human motion.
 Forward reachable set Throughout this section, we frequently refer to the human's time-indexed forward reachable set. We define this set formally in the following. Definition 1. (Forward reachable set) For a dynamical system _ x = f (x, u) with state trajectories given by the function j(x(0), t, u( Á ))ex(t), the forward reachable set FRS(x, t) of a state x after time t has elapsed is FRS(x, t) :¼ fx 0 : 9u( Á ), x 0 = j(x, t, u( Á ))g That is, a state x 0 is in the forward reachable set of x after time t if it is reachable via some applied control signal u( Á ). Remark 1. (Recovery of FRS) For P th = 0 and any finite b, the set of states assigned probability greater than P th is identical to the forward reachable set, up to discretization errors. This is visualized for low, high, and Bayesian model confidence in Figure  5 .
 A sufficient condition for the safety of individual trajectories In Section 6.2, we construct an approximation to the probability of collision along a trajectory, which we use during motion planning to avoid potentially dangerous states. To make this guarantee of collision avoidance for a motion plan even stronger, it would suffice to ensure that the robot never comes too close to the human's forward reachable set. More precisely, a planned trajectory is safe if fx R (t) + Eg \ FRS(x H , t) = ;, for every state x R (t) along a motion plan generated when the human was at state x H . The proof of this statement follows directly from the properties of the tracking error bound E described in Section 6. While this condition may seem appealing, it is in fact highly restrictive. The requirement of avoiding the full forward reachable set is not always possible in confined spaces; indeed, this was our original motivation for wanting to predict human motion (see Section 3.4). However, despite this shortcoming, the logic behind this sufficient condition for safety provides insight into the effectiveness of our framework.
 Recovering the forward reachable set Though it will not constitute a formal safety guarantee, we analyze the empirical safety properties of our approach by examining how our predicted state distributions over time relate to forward reachable sets. During operation, our belief over model confidence b evolves to match the degree to which the utility model Q H explains recent human motion. The ''time constant'' governing the speed of this evolution may be tuned by the system designer to be arbitrarily fast by choosing the parameter e to be small, as discussed in Section 4.1. Thus, we may safely assume that b(b) places high probability mass on small values of b as soon as the robot observes human motion that is not well explained by Q H . Figure  6  shows the sets of states with ''high enough'' (.P th ) predicted probability mass overlaid on the human's forward reachable set at time t, which is a circle of radius v H t centered on x H for the dynamics in our running example. When b is high (10), we observe that virtually all of the probability mass is concentrated in a small number of states in the direction of motion predicted by our utility model. When b is low (0:05) we observe that the set of states assigned probability above our collision threshold P th occupies a much larger fraction of the reachable set. A typical belief b(b) recorded at a moment when the human was roughly moving according to Q H yields an intermediate set of states. Figure  7  illustrates the evolution of these sets of states over time, for the unmodeled obstacle example of Section 5.4 in which a pedestrian avoids a spill. Each row corresponds to the predicted state distribution at a particular point in time. Within a row, each column shows the reachable set and the set of states assigned occupancy Interestingly, as the Bayesian model confidence decreases, which occurs when the pedestrian turns to avoid the spill at t'6 s, the predicted state distribution assigns high probability to a relatively large set of states, but unlike the low-b predictor that set of states is oriented toward the known goal. Of course, had b(b) placed even more probability mass on lower values of b then the Bayesian confidence predictor would converge to the low confidence one. In addition, we observe that, within each row as the prediction horizon increases, the area contained within the forward reachable set increases and the fraction of that area contained within the predicted sets decreases. This phenomenon is a direct consequence of our choice of threshold P th . Had we chosen a smaller threshold value, a larger fraction of the forward reachable set would have been occupied by the lower-b predictors. This observation may be viewed prescriptively. Recalling the sufficient condition for safety of planned trajectories from Section 7.2, if the robot replans every T replan seconds, we may interpret the fraction of FRS( Á , t + T replan ) assigned occupancy probability greater than P th by the low-confidence predictor as a rough indicator of the safety of an individual motion plan, robust to worst-case human movement. As this fraction tends toward unity, the robot is more and more likely to be safe. However, for any P th .0, this fraction approaches zero for T replan " '. This immediately suggests that, if we wish to replan every T replan seconds, we can achieve a particular level of safety as measured by this fraction by choosing an appropriate threshold P th . In summary, confidence-aware predictions rapidly place high-probability mass on low values of b whenever human motion is not well-explained by utility model Q H . Whenever this happens, the resulting predictions encompass a larger fraction of the forward reachable set, and in the limit that P th # 0 we recover the forward reachable set exactly. The larger this fraction, the more closely our approach satisfies the sufficient condition for safety presented in Section 7.2.
 Hardware demonstration We implemented confidence-aware human motion prediction (Section 4) and integrated it into a real-time, safe probabilistic motion planner (Section 6), all within the Robot Operating System (ROS) software framework of  Quigley et al. (2009) . To demonstrate the efficacy of our methods, we tested our work for the quadcopter-avoiding-pedestrian example used for illustration throughout this paper. Human trajectories were recorded as (x, y) positions on the ground plane at roughly 235 Hz by an OptiTrack infrared motion capture system, and we used a Crazyflie 2.0 micro-quadcopter, also tracked by the OptiTrack system. 4 Figure  4  illustrates the unmodeled obstacle case from Section 5.4, in which the pedestrian turns to avoid a spill on the ground. Using a low model confidence results in motion plans that suddenly and excessively deviate from the ideal straight-line path when the pedestrian turns to avoid the spill. By contrast, the high-confidence predictor consistently predicts that the pedestrian will walk in a straight line to the goal even when they turn; this almost leads to collision, as shown in detail in Figure  8 . Our proposed approach for Bayesian model confidence initially assigns high confidence and predicts that the pedestrian will walk straight to the goal, but when they turn to avoid the spill, the predictions become less confident. This causes the quadcopter to make a minor course correction, shown in further detail in Figure  9 .
 Conclusion When robots operate in complex environments in concert with other agents, safety often depends upon the robot's ability to predict the agents' future actions. While this prediction problem may be tractable in some cases, it can be extremely difficult for agents such as people who act with intent. In this paper, we introduce the idea of confidenceaware prediction as a natural coping mechanism for predicting the future actions of intent-driven agents. Our approach uses each measurement of the human's state to reason about the accuracy of its internal model of human decision-making. This reasoning about model confidence is expressed compactly as a Bayesian filter over the possible values of a single parameter, b, which controls the entropy of the robot's model of the human's choice of action. In effect, whenever the human's motion is not wellexplained by this model, the robot predicts that the human could occupy a larger volume of the state space.  We couple this notion of confidence-aware prediction with a reachability-based robust motion planning algorithm, FaSTrack, which quantifies the robot's ability to track a planned reference trajectory. Using this maximum tracking error allows us to bound an approximation of the probability of collision along planned trajectories. In addition, we present a deeper connection between confidence-aware prediction and forward reachable sets, which provides an alternative explanation of the safety of our approach. We demonstrate the proposed methodology on a ROS-based quadcopter testbed in a motion capture arena.
 Limitations There are several important limitations of this work, which we summarize and discuss in the following. 9.1.1. State discretization. As presented, our approach to prediction requires a discrete representation of the human's state space. This can be tractable for the relatively simple dynamical models of human motion we consider in this work. Fortunately, one of the strongest attributes of confidenceaware prediction is that it affords a certain degree of robustness to modeling errors by design. Still, our approach is effectively limited to low-order dynamical models. 9.1.2. FaSTrack complexity. FaSTrack provides a strong safety guarantee vis-a `-vis the maximum tracking error that could ever exist between a higher-fidelity dynamical model of the robot and a lower-order model used for motion planning. Unfortunately, the computational complexity of finding this maximum tracking error and the corresponding safety controller scales exponentially with the dimension of The subsequent motion plans may not be safe; here, poor prediction quality leads to a collision. the high-fidelity model. In some cases, these dynamics are decomposable and analytic solutions exist (e.g.,  Chen et al., 2018; Fridovich-Keil et al., 2018) , and in other cases conservative approximations may be effective (e.g.,  Chen et al., 2016; Royo et al., 2018) . 9.1.3. Boltzmann distributional assumption. We model the human's choice of control input at each time step as an independent, random draw from a Boltzmann distribution (3). This distributional assumption is motivated from the literature in cognitive science and econometrics and is increasingly common in robotics, yet it may not be accurate in all cases. Maintaining an up-to-date model confidence belief b(b) can certainly mitigate this inaccuracy, but only at the cost of making excessively conservative predictions. 9.1.4. Safety certification. Our analysis in Section 7 makes connections to forward reachability in an effort to understand the safety properties of our system. As shown, whenever our confidence-aware prediction method detects poor model performance it quickly yields predictions that approximate the human's forward reachable set. Although this approximation is not perfect, and hence we cannot provide a strong safety certificate, the connection to reachability is in some sense prescriptive. That is, it can be used to guide the choice of collision probability threshold P th and replanning frequency. However, even if we could provide a strong guarantee of collision avoidance for a particular motion plan, that would not, in general, guarantee that future motion plans would be recursively safe. This recursive property is much more general and, unsurprisingly, more difficult to satisfy.
 Future directions Future work will aim to address each of these shortcomings. We are also interested in extending our methodology for the multi-robot, multi-human setting; our preliminary results are reported by  Bajcsy et al. (2018) . In addition, we believe that our model confidence inference approach could be integrated with other commonly used probabilistic prediction methods besides the Boltzmann utility model. Finally, we are excited to test our work in hardware in other application spaces, such as manipulation and driving. Fig. Fig.1. When planning around humans, accurate predictions of human motion (visualized here in pink and blue, representing high and low probability, respectively) are an essential prerequisite for safety. Unfortunately, these approaches may fail to explain all observed motion at runtime (e.g., human avoids unmodeled spill on the ground), leading to inaccurate predictions, and potentially, collisions (left). Our method addresses this by updating its predictive model confidence in real time (right), leading to more conservative motion planning in circumstances when predictions are known to be suspect.
 with b k + (b, u) :¼ P(b, ujx 0:k H , u 0:k H ) the running posterior and b k À (b, u) :¼ P(b, ujx 0:kÀ1 H , u 0:kÀ1 H
 Fig. 2 . 2 Fig. 2. Snapshots of pedestrian trajectory and probabilistic model predictions. Top row: Pedestrian moves from the bottom right to a goal marked as a red circle. Middle row: Pedestrian changes course to avoid a spill on the floor. Bottom row: Pedestrian moves to one known goal, then to another, then to a third which the robot has not modeled. The first two columns show predictions for low and high model confidence; the third column shows the predictions using our Bayesian model confidence. For all pedestrian videos, see https://youtu.be/lh_E9rW-MJo.
 Fig. 3 . 3 Fig. 3. Snapshots of Dubins car and probabilistic predictions. Top row: Car moves straight ahead toward one of two known goals (red arrows), staying in its lane. Middle row: Car suddenly swerves to the left to avoid a pothole. Bottom row: Car turns to the right, away from the only known goal. The left and center columns show results for low and high confidence predictors, respectively, and the right column shows our approach using Bayesian inferred model confidence. For all Dubins car videos, see https://youtu.be/sAJKNnP42fQ.
 Fig. 4 . 4 Fig. 4. Scenario from the middle row of Figure 2 visualized with robot's trajectory. When b is low and the robot is not confident, it makes large deviations from its path to accommodate the human. When b is high, the robot refuses to change course and comes dangerously close to the human. With inferred model confidence, the robot balances safety and efficiency with a slight deviation around the human.
 Fig. 5 . 5 Fig. 5. The human (black dot) is moving west towards a goal. Visualized are the predicted state distributions for 1 second into the future when using low, high, and Bayesian model confidence. Higher-saturation indicates higher likelihood of occupancy. The dashed circle represents the pedestrian's 1 second forward reachable set.
 Fig. 6 . 6 Fig.6. Visualization of the states with probability greater than or equal to the collision threshold, P th = 0:01. The human's forward reachable set includes the set of states assigned probability greater than P th . We show these ''high probability'' predicted states for predictors with fixed low and high b, as well as our Bayesian-inferred b.
 Fig. 7 . 7 Fig.7. The human (black dot) is walking towards the known goal (red dot) but has to avoid an unmodeled coffee spill on the ground. Here we show the snapshots of the predictions at various future times (columns) as the human walks around in real time (rows). The visualized states have probability greater than or equal to P th = 0:01. Each panel displays the human prediction under low confidence (in yellow), high confidence (in dark purple), and Bayesian confidence (colored as per the most likely b value), as well as the forward reachable set. The human's actual trajectory is shown in red.
 Fig. 8 . 8 Fig. 8. Predicting with fixed-b (in this case, b = 20) can yield highly inaccurate predictions (and worse, confidently inaccurate ones).The subsequent motion plans may not be safe; here, poor prediction quality leads to a collision.
 Fig. 9 . 9 Fig. 9. Inferring b leads to predicted state distributions whose entropy increases whenever the utility model Q H fails to explain observed human motion. The resulting predictions are more robust to modeling errors, resulting in safer motion plans. Here, the quadcopter successfully avoids the pedestrian even when they turn unexpectedly.
			 The International Journal of Robotics Research 39(2-3)

Abstract: One of the most difficult challenges in robot motion planning is to account for the behavior of other moving agents, such as humans. Commonly, practitioners employ predictive models to reason about where other agents are going to move. Though there has been much recent work in building predictive models, no model is ever perfect: an agent can always move unexpectedly, in a way that is not predicted or not assigned sufficient probability. In such cases, the robot may plan trajectories that appear safe but, in fact, lead to collision. Rather than trust a model's predictions blindly, we propose that the robot should use the model's current predictive accuracy to inform the degree of confidence in its future predictions. This model confidence inference allows us to generate probabilistic motion predictions that exploit modeled structure when the structure successfully explains human motion, and degrade gracefully whenever the human moves unexpectedly. We accomplish this by maintaining a Bayesian belief over a single parameter that governs the variance of our human motion model. We couple this prediction algorithm with a recently proposed robust motion planner and controller to guide the construction of robot trajectories that are, to a good approximation, collision-free with a high, user-specified probability. We provide extensive analysis of the combined approach and its overall safety properties by establishing a connection to reachability analysis, and conclude with a hardware demonstration in which a small quadcopter operates safely in the same space as a human pedestrian.