Zana Buçinca
Phoebe Lin
Krzysztof Z Gajos
Elena L 2020 Glassman

Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems

INTRODUCTION Because people and AI-powered systems have complementary strengths, many expected that human+AI teams would perform better on decision-making tasks than either people or AIs alone  [1, 21, 22] . However, there is mounting evidence that human+AI teams often perform worse than AIs alone  [16, 17, 28, 34] . We hypothesize that this mismatch between our field's aspirations and the current reality can be attributed, in part, to several pragmatic decisions we frequently make in our research practice. Specifically, although our aspiration is formulated at the level of sociotechnical systems , i.e., human+AI teams working together to make complex decisions, we often make one of two possible critical mistakes: (1) Rather than evaluating how well the human+AI team performs together on a decision-making task, we evaluate by using proxy tasks, how accurately a human can predict the decision or decision boundaries of the AI  [13, 27, 29, 34] .  (2)  We rely on subjective measures of trust and preference, e.g.,  [35, 36, 44] , instead of objective measures of performance. We consider each of these two concerns in turn. First, evaluations that use proxy tasks force study participants to pay attention to the AI and the accompanying explanationssomething that they are unlikely to do when performing a realistic decision-making task. Cognitive science provides compelling evidence that people treat cognition like any other form of labor  [24]  and favor less demanding forms of cognition, i.e., heuristics over analytical thinking, even in high stakes contexts like medical diagnosis  [31] . Therefore, we hypothesize that user performance and preference on proxy tasks may not accurately predict their performance and preference on the actual decision-making tasks where their cognitive focus is elsewhere and they can choose whether and how much to attend to the AI. Second, subjective measures such as trust and preference have been embraced as the focal point for the evaluation of explainable systems  [35, 36, 44] , but we hypothesize that subjective measures may also be poor predictors of the ultimate performance of people performing realistic decision-making tasks while supported by explainable AI-powered systems. Preference and trust are important facets of explainable AI systems: they may predict users' intent to attend to the AI and its explanations in realistic tasks settings and adhere to the system's recommendations. However, the goal of explainable interfaces should be instilling in users the right amount of trust  [10, 32, 33] . This remains a remarkable challenge, as on one end of the trust spectrum users might over-rely on the system and remain oblivious of its errors, whereas on the other end they might exhibit self-reliance and ignore the system's correct recommendations. Furthermore, evaluating an AI's decision, its explanation of that decision, and incorporating that information into the decisionmaking process requires cognitive effort and the existing evidence suggests that preference does not predict performance on cognitive tasks  [8, 12, 37] . To evaluate these two hypotheses, we conducted two online experiments and one in-person study of an AI-powered decision support system for a nutrition-related decision-making task. In one online study we used a proxy task, in which participants were asked to predict the AI's recommendations given the explanations produced by the explainable AI system. In the second online study, participants completed an actual decision-making task: actually making decisions assisted by the same explainable AI system as in the first study. In both studies, we measured participants' objective performance and collected subjective measures of trust, preference, mental demand, and understanding. In the in-person study, we used a think-aloud method to gain insights into how people reason while making decisions assisted by an explainable AI system. In each study, we presented participants with two substantially distinct explanation types eliciting either deductive or inductive reasoning. The results of these studies indicate that (1) subjective measures from the proxy task do not generalize to the actual decision-making task, and (2) when using actual decision-making tasks, subjective results do not predict objective performance results. Specifically, participants trusted and preferred inductive explanations in the proxy task, whereas they trusted and preferred the deductive explanations in the actual task. Second, in the actual decision-making task, participants recognized AI errors better with inductive explanations, yet they preferred and trusted the deductive explanations more. The in-person think-aloud study revealed insights about why participants preferred and trusted one explanation type over another, but we found that by thinking aloud during an actual decision-making task, participants may be induced to exert additional cognitive effort, and behave differently than they would during an actual decision-making task when they are, more realistically, not thinking aloud. In summary, we show that the results of evaluating explainable AI systems using proxy tasks may not predict the results of evaluations using actual decision-making tasks. Users also do not necessarily perform better with systems that they prefer and trust more. To draw correct conclusions from empirical studies, explainable AI researchers should be wary of evaluation pitfalls, such as proxy tasks and subjective measures. Thus, as we recognize that explainable AI technology forms part of sociotechnical systems, and as we increasingly use these technologies in high-stakes scenarios, our evaluation methodologies need to reliably demonstrate how the entire sociotechnical systems (i.e., human+AI teams) will perform on real tasks.
 RELATED WORK 2.1 Decision-making and Decision Support Systems Decision-making is a fundamental cognitive process that allows humans to choose one option or course of action from among a set of alternatives  [42, 43, 45] . Since it is an undertaking that requires cognitive effort, people often employ mental shortcuts, or heuristics, when making decisions  [40] . These heuristics save time and effort, and frequently lead to good outcomes, but in some situations they result in cognitive biases that systematically lead to poor decisions (see, e.g.,  [4] ). To help people make good decision reliably, computer-based Decision Support Systems (DSS) have been used across numerous disciplines (e.g., management  [15] , medicine  [20] , justice  [47] ). While DSS have been around for a long time, they are now increasingly being deployed because the recent advancements in AI enabled these systems to achieve high accuracy. But since humans are the final arbiters in decisions made with DSS, the overall sociotechincal system's accuracy depends both on the system's accuracy and on the humans and their underlying cognitive processes. Research shows that even when supported by a DSS, people are prone to insert bias into the decision-making process  [16] . One approach for mitigating cognitive biases in decision-making is to use cognitive forcing strategies, which introduce self-awareness and self-monitoring of decision-making  [7] . Although not universally effective  [38] , these strategies have shown promising results as they improve decision-making performance, both if the human is assisted  [17, 34]  or is not assisted by a DSS  [31] . To illustrate, Green & Chen  [17]  showed that across different AI-assisted decision-making treatments, humans performed best when they had to make the preliminary decision on their own first before being shown the system recommendation (which forced them to engage analytically with the system's recommendation and explanation if their own preliminary decision differed from that offered by the system). Even though conceptual frameworks that consider cognitive processes in decision-making with DSS have been proposed recently  [41] , further research is needed to thoroughly investigate how to incorporate DSS into human decision-making and the effect of cognitive processes while making system-assisted decisions.
 Evaluating AI-Powered Decision Support Systems Motivated by the growing number of studies in interpretable and explainable AI-powered decision support systems, researchers have called for more rigorous evaluation of explainable systems  [9, 14, 19] . Notably, Doshi-Velez & Kim  [9]  proposed a taxonomy for evaluation of explainable AI systems, composed of the following categories: application grounded evaluation (i.e., domain experts evaluated on actual tasks), human grounded evaluation (i.e., lay humans evaluated on simplified tasks) and functionally grounded evaluation (i.e., no humans, proxy tasks). To put our work into context, our definition of the actual task falls into application grounded evaluation, where people for whom the system is intended (i.e., not necessarily experts) are evaluated on the intended task. Whereas, the the proxy task is closer to human grounded evaluation but addresses both domain experts and lay people evaluated on simplified tasks, such as the simulation of model's prediction given an input and an explanation. Studies using actual tasks evaluate the performance of human and the system, as a whole, on the decision-making task  [3, 17, 23, 46] . In these studies, participants are told to focus on making good decisions and it is up to them to decide whether and how to use the AI's assistance to accomplish the task. In contrast, studies that use proxy tasks evaluate how well users are able to simulate the model's decisions  [6, 13, 27, 34]  or decision boundaries  [29] . In such studies, participants are specifically instructed to pay attention to the AI. These studies evaluate the human's mental model of the system when the human is actively attending to the system's predictions and explanations, but do not necessarily evaluate how well the human is able to perform real decision-making tasks with the system. For example, to identify which factors make a model more interpretable, Lage et al. ask participants to simulate the interpretable model's predictions  [27] . In addition to the evaluation task, the choice of evaluation metrics is a critical one for the correct evaluation of intelligent systems  [2] . In explainable AI literature, subjective measures, such as user trust and experience, have been largely embraced as the focal point for the evaluation of explainable systems  [35, 36, 44, 48] . Hoffman et al.  [19]  proposed metrics for explainable systems that are grounded in the subjective evaluation of a system (e.g., user satisfaction, trust, and understanding). These may take the form of questionnaires on attitude and confidence in the system  [18]  and helpfulness of the system  [5, 26] . However, while these measures are informative, evidence suggests they do not necessarily predict user's performance with the system. For example, Green & Chen  [16]  discovered that self-reported measures could be misleading, since participant's confidence in their performance was negatively associated with their actual performance. Similarly, Lai & Tan  [28]  found that humans cannot accurately estimate their own performance. More closely related to our findings, Poursabzi-Sangdeh et al.  [34]  observed that even though participants were significantly more confident on the predictions of one model over the other, their decisions did not reflect the stated confidence. Furthermore, Lakkaraju & Bastani  [30]  demonstrated that participants trusted the same underlying biased model almost 10 times more when they were presented with misleading explanations compared to the truthful explanations that revealed the model's bias. These findings indicate that not only are subjective measures poor predictors of performance, but they can easily be manipulated and lead users to adhere to biased or malicious systems.
 EXPERIMENTS We conducted experiments with two different evaluation tasks and explanation designs to test the following hypotheses: H1: Results of widely accepted proxy tasks, where the user is asked to explicitly engage with the explanations, may not predict the results of realistic settings where the user's focus is on the actual decision-making task. H2: Subjective measures, such as self-reported trust and preference with respect to different explanation designs, may not predict the ultimate human+AI performance.
 Proxy Task 3.1.1 Task Description. We designed the task around nutrition because it is generally accessible and plausibly useful in explainable AI applications for a general audience. Participants were shown a series of 24 images of different plates of food. The ground truth of the percent fat content was also shown to them as a fact. Participants were then asked: "What will the AI decide?" given that the AI must decide "Is X% or more of the nutrients on this plate fat?". As illustrated in Figure  1 , each image was accompanied by explanations generated by the simulated AI. The participants chose which decision they thought the AI would make given the explanations and the ground truth. We designed two types of explanations, eliciting either inductive or deductive reasoning. In inductive reasoning, one infers general patters from specific observations. Thus, for the inductive explanations, we created example-based explanations that required participants to recognize the ingredients that contributed to fat content and draw their own conclusion about the given image. As shown in Figures 1a, the inductive explanations began with "Here are examples of plates that the AI knows the fat content of and categorizes as similar to the one above. " Participants then saw four additional images of plates of food. In deductive reasoning, in contrast, one starts with general rules and reaches a conclusion with respect to a specific situation. Thus, for the deductive explanations, we provided the general rules that the simulated AI applied to generate its recommendations. For example, in Figure  1b , the deductive explanation begins with "Here are ingredients the AI knows the fat content of and recognized as main nutrients:" followed by a list of ingredients. We chose a within-subjects study design, where for one half of the study session, participants saw inductive explanations and, for the other half of the study session, they saw deductive explanations. The order in which the two types of explanations were seen was counterbalanced. Each AI had an overall accuracy of 75%, which meant that in 25% of the cases the simulated AI misclassified the image or misrecognized ingredients (e.g., Figure  1b ). The order of the specific food images was randomized, but all participants encountered the AI errors at the same positions. We fixed the errors at questions 4, 7, 11, 16, 22 and 23, though which food the error was associated to was randomized. We included the ground truth of the fat content of plates of food, because the main aim of the proxy task was to measure whether the user builds correct mental models of the AI and not to assess the actual nutrition expertise of the participant. 3.1.2 Procedure. This study was conducted online, using Amazon Mechanical Turk. Participants were first presented with brief information about the study and an informed consent form. Next, participants completed the main part of the study, in which they answered 24 nutrition-related questions, divided into two blocks of 12 questions. They saw inductive explanations in one block and the deductive explanations in the other. The order of explanations was randomized across participants. Participants completed mid-study and end-study questionnaires so that they would provide a separate assessment for each of the two explanation types. They were also asked to directly compare their experiences with the two simulated AIs in a questionnaire at the end of the study. 3.1.3 Participants. We recruited 200 participants via Amazon Mechanical Turk (AMT). Participation was limited to adults in the US. Of the total 200 participants, 183 were retained for final analyses, while 17 were excluded based on their answers to two commonsense questions included in the questionnaires (i.e., What color is the sky?). The study lasted 7 minutes on average. Each worker was paid 2 USD. 3.1.4 Design and Analysis. This was a within-subjects design. The within-subjects factor was explanation type -inductive or deductive. We collected the following measures: • Performance: Percentage of correct predictions of AI's decisions • Appropriateness: Participants responded to the statement "The AI based its decision on appropriate examples/ingredients. " with either 0=No or 1=Yes (after every question) • Trust: Participants responded to the statement "I trust this AI to assess the fat content of food. " on a 5-point Likert scale from 1=Strongly disagree to 5=Strongly agree (at the end of each block) • Mental demand: Participants answered the question "How mentally demanding was understanding how this AI makes decisions?" on a 5-point Likert scale from 1=Very low to 5=Very high (every four questions) • Comparison between the two explanation types: Participants were asked at the end of the study to choose one AI over another on trust, preference, and mental demand. We used repeated measures ANOVA for within-subjects analyses and the binomial test for the comparison questions.
 Actual Decision-making Task 3.2.1 Task description. The actual decision-making task had a similar set up to the proxy task. Participants were shown the same series of 24 images of different plates of food, but were asked their own decision whether the percent fat content of nutrients on the plate is higher than a certain percentage. As illustrated in Figure  2 , each image was accompanied by an answer recommended by a simulated AI, and an explanation provided by that AI. We introduced two more conditions to serve as baselines in the actual decision-making task depicted in Figure  3 . There were three between-subjects conditions in this study: 1. the no-AI baseline (where no recommendations or explanations were provided), 2. the no-explanation baseline (where a recommendation was provided by a simulated AI, but no explanation was given), and 3. the main condition in which both recommendations and explanations were provided. In this last condition, two withinsubjects sub-conditions were present: for one half of the study participants saw inductive explanations and for the other they saw deductive explanations. The order in which the two types of explanations were seen was counterbalanced. In the no-AI baseline, participants were not asked any of the questions relating to the performance of the AI. The explanations in this task differed only slightly from the explanations in the proxy task, because they indicated the AI's recommendation. Inductive explanations started with: "Here are examples of plates that the AI categorizes as similar to the one above and do (not) have X% or more fat. " followed by four examples of images. Similarly, deductive explanations stated: "Here are ingredients the AI recognized as main nutrients which do (not) make up X% or more fat on this plate:" followed by a list of ingredients. 3.2.2 Procedure. The procedure was the same as for the proxy task. The study was conducted online, using the Amazon Mechanical Turk. Participants were first presented with a brief information about the study and an informed consent form. Next, participants completed the main part of the study, in which they answered 24 nutrition-related questions, divided into two blocks of 12 questions. All participants also completed a questionnaire at the end of the study, providing subjective assessments of the system they interacted with. Participants who were presented with AI-generated recommendations accompanied by explanations also completed a mid-study questionnaire (so that they would provide separate assessment for each of the two explanation types) and they were also asked to directly compare their experiences with the two simulated AIs at the end of the study.
 Participants. We recruited 113 participants via Amazon Mechanical Turk (AMT). Participation was limited to adults in the US. Of the total 113 participants, 102 were retained for final analyses, while 11 were excluded based on their answers to two commonsense questions included in the pre-activity and post-activity questionnaires (i.e., "What color is the sky?"). The task lasted 10 minutes on average. Each worker was paid 5 USD per task.
 Design and Analysis . This was a mixed between-and withinsubjects design. As stated before, the three between-subjects conditions were: 1. the no-AI baseline; 2. the no-explanation baseline, in which the AI-generated recommendations were provided but no explanations; 3. the main condition, in which both the AI-generated recommendations and explanations were provided. The withinsubjects factor was explanation type (inductive or deductive) and it was applied only for participants who were presented with AIgenerated recommendations with explanations. We collected the following measures: • Performance: Percentage of correct answers (overall for each AI, and specifically for questions when AI presented incorrect explanations) • Understanding: Participants responded to the statement "I understand how the AI made this recommendation." on a 5point Likert scale from 1=Strongly disagree to 5=Strongly agree (after every question) • Trust: Participants responded to the statement "I trust this AI to assess the fat content of food. " on a 5-point Likert scale from 1=Strongly disagree to 5=Strongly agree (every four questions) • Helpfulness: Participants responded to the statement "This AI helped me assess the percent fat content." on a 5-point Likert scale from 1=Strongly disagree to 5=Strongly agree (at the end of each block) • Comparison between the two explanation types: Participants were asked at the end of the study to choose one AI over another on trust, preference, understanding and helpfulness. We used analysis of variance (ANOVA) for between-subjects analyses and repeated measures ANOVA for within-subjects analyses. We used the binomial test for the comparison questions.
 RESULTS
 Proxy Task Results The explanation type had a significant effect on participants' trust and preference in the system. Participants trusted the AI more when presented with inductive explanations (M = 3.55), rather than deductive explanations (M = 3.40, F 1,182 = 5.37, p = .02). Asked to compare the two AIs, most of the participants stated they trusted more the inductive AI (58%, p = .04). When asked the hypothetical question: "If you were asked to evaluate fat content of plates of food, which AI would you prefer to interact with more?", again most of the participants (62%) chose the inductive AI over the deductive AI (p = .001). The inductive AI was also rated significantly higher (M = 0.83) than the deductive AI (M = 0.79) in terms of the appropriateness of examples (ingredients for the deductive condition) on which the AI based its decision (F (1, 182) = 13.68, p = 0.0003). When the AI presented incorrect examples/ingredients, there was no significant difference among the inductive (M = 0.47) and deductive (M = 0.50) conditions (F (1, 182) = 1.02, p = .31, n.s.). We observed no significant difference in overall performance when participants were presented with inductive (M = 0.64) or deductive explanations (M = 0.64, F (1, 182) = 0.0009, n.s.). When either AI presented incorrect explanations, although the average performance dropped for both inductive (M = 0.40) and deductive (M = 0.41) conditions, there was also no significant difference among them (F (1, 182) = .03, n.s.). In terms of mental demand, there was a significant effect of the explanation type. Participants rated the deductive AI (M = 2.94) as more mentally demanding than the inductive AI (M = 2.79, F (1, 182) = 7.75, p = .0006). The effect was noticed also when they were asked: "Which AI required more thinking while choosing which decision it would make?", with 61% of participants choosing deductive over inductive (p = .005).
 Actual Decision-making Task Results 18 participants were randomized into the no-AI condition, 19 into the AI with no explanation condition, and 65 were presented with AI recommendations supported by explanations. We observed a significant main effect of the presence of explanations on participants' trust in the AI's ability to assess the fat content of food. Participants who saw either kind of explanation, trusted the AI more (M = 3.56) than those who received AI recommendations, but no explanations (M = 3.17, F 1,483 = 11.28, p = .0008). Further, there was a significant main effect of the explanation type on participants' trust: participants trusted the AI when they received deductive explanations more (M = 3.68) than when they received inductive explanations (M = 3.44, F 1,64 = 5.96, p = .01). When asked which of the two AIs they trusted more, most participants (65%) said that they trusted the AI that provided deductive explanations more than the one that provided inductive explanations (p = .02). Participants also found the AI significantly more helpful when explanations were present (M = 3.78) than when no explanations were offered (M = 3.26, F 1,147 = 4.88, p = .03). Further, participants reported that they found deductive explanations more helpful (M = 3.92) than inductive ones (M = 3.65) and this difference was marginally significant (F 1,64 = 3.66, p = .06). When asked which of the two AIs they found more helpful, most participants (68%) chose the AI that provided deductive explanations (p = .006). Participants also reported that they understood how the AI made its recommendations better when explanations were present (M = 3.84) than when no explanations were provided (M = 3.67, F 1,2014 = 6.89p = .009). There was no difference in the perceived level of understanding between the two explanation types (F 1,64 = 0.44, p = .51). Asked about their overall preference, most participants (63%) preferred the AI that provided deductive explanations over the AI that provided inductive explanations (p = .05). In terms of actual performance on the task, participants who received AI recommendations (with or without explanations) provided a significantly larger fraction of accurate answers (M = 0.72) than those who did not receive AI recommendations (M = 0.46, F 1,2446 = 118.07, p < .0001). Explanations further improved overall performance: participants who saw explanations of AI recommendations had a significantly higher proportion of correct answers (M = 0.74) than participants who did not receive explanations of AI recommendations (M = 0.68, F 1,2014 = 5.10, p = .02) (depicted in Figure  4a ). There was no significant difference between the two explanation types in terms of overall performance (F 1,64 = 0.44, n.s.). However, we observed a significant interaction between explanation type and the correctness of AI recommendations (F 2,2013 = 15.03p < .0001). When the AI made correct recommendations, participants performed similarly when they saw inductive (M = .78) and deductive (M = .81) explanations (F 1,64 = 1.13, n.s.). When the AI made incorrect recommendations, however, participants were significantly more accurate when they saw inductive (M = 0.63) than deductive (M = 0.48) explanation (F 1,64 = 7.02, p = .01) (depicted in Figure  4b ). To ensure the results of our studies were not random, we replicated both experiments with almost identical setup and obtained the same main results (in terms of significance) reported in this section.
 QUALITATIVE STUDY Through the qualitative study, we explored the user reasoning and sought to gain insight into the discrepancy between subjective measures and performance. We asked participants to think aloud during an in-person study in order to understand how and why people perceive AI the way they do, in addition to what factors go into making decisions when assisted by an AI.
 Task The same task design was used in this study as in the actual decisionmaking task, except that all participants were presented with the main condition (where both recommendations and explanations were provided). As in the actual decision-making task, each participant saw both inductive and deductive explanations.
 Procedure Upon arriving to the lab, participants were presented with an informed consent form, including agreeing to being screen-and audiorecorded, and instructions on the task. Afterwards, the steps in this study were similar to those in the actual decision-making task, except that we added the think-aloud method  [11] : as participants completed the task, they were asked to verbalize their thought process as they made each decision. At the end of the task, there was a semi-structured interview, during which participants briefly discussed how they believed the two AIs were making their recommendations and why they did or did not trust them. Participants also discussed if and why they preferred one AI over the other.
 Participants We recruited 11 participants via community-wide emailing lists (8 female, 3 male, age range 23-29, M = 24.86, SD = 2.11). Participants were primarily graduate students with backgrounds from design, biomedical engineering, and education. Participants had varying levels of experience with AI and machine learning, ranging from 0-5 years of experience.
 Design and Analysis We transcribed the think-aloud comments and the post-task interviews. Transcripts were coded and analyzed for patterns using an inductive approach  [39] . We focused on comments about (1) how the AI made its recommendations; (2) trust in the AI; (3) erroneous recommendations; (4) why people preferred one explanation type over the other. From a careful reading of the transcripts, we discuss some of the themes and trends that emerged from the data.
 Results Preference of one explanation type over another. Eight out of the 11 participants preferred the inductive explanations. Participants who preferred inductive explanations perceived the four images as data. One participant stated that "Because [the AI] showed similar pictures, I knew that it had data backing it up" (P3). On the other hand, participants who preferred deductive explanations perceived the listing of ingredients to be reliable, and that "if the AI recognized that it's steak, then I would think, Oh the AI knows more about steak fat than I do, so I'm going to trust that since it identified the object correctly. " (P6). In our observations, we found that the way participants used the explanations was different depending on the explanation type. With inductive explanations, one participant often first made their own judgement before looking at the recommendation, and then used the recommendation to confirm their own judgement. In a cake example, one participant said, "So I feel it probably does have more than 30% because it's cake, and that's cream cheese. But these are all similar to that, and the AI also says that it does have more than 30% fat, so I agree" (P2). With deductive explanations, participants evaluated the explanations and recommendation more before making any decision. In the same cake example, a different participant said, "There are [the AI recognizes] nuts, cream cheese, and cake. That seems to make sense. Nuts are high in fat, so is dairy, so I agree with that. " (P6). Cognitive Demand. At the end of the study, participants were asked which AI was easier to understand. Ten out of 11 participants felt the inductive explanations were easier to understand than the deductive explanations. Several participants stated that the deductive explanations forced them to think more, and that generally they spent more time making a decision with deductive explanations. One participant said, for example, "I feel like with this one I have to think a bit more and and rely on my own experiences with food to see or understand to gauge what's fatty. " (P2). Errors and Over-reliance. Nine out of 11 participants claimed to trust the inductive explanations more. We intentionally introduced erroneous recommendations because we expected participants to utilize them to calibrate their mental model of the AI. When participants understood the error and believed the error was reasonable for an AI to make, they expressed less distrust in subsequent questions. However, when participants perceived the error to be inconsistent with other errors, their trust in subsequent recommendations was hurt much more. For example, one participant stated, "I think the AI makes the recommendation based on shape and color. But in some other dessert examples, it was able to identify the dessert as a dessert. So I wasn't sure why it was so difficult to understand this particular item" (P5). We found that there was also some observable correlation between explanation type and trust. Many participants claimed it was easier to identify errors from the inductive explanations, yet agreed with erroneous recommendations from inductive explanations more. In some of those instances, participants either did not realize the main food image was different from the other four or felt the main food image was similar enough though not exact. Lastly, one participant stated the inductive explanations were easier to understand because "you can visually see exactly why it would come to its decision,", but for deductive explanations "you can see what it's detecting but not why" (P8), and yet this participant also stated that the deductive explanations seemed more trustworthy. Impact of the Think-Aloud method on participant behavior. In this study, we asked participants to perform the actual decision-making task and we expected to observe similar results to those obtained in the previous experiment when using the actual tasks. Yet, in this study, 8 out of the 11 participants preferred the inductive explanations and 10 out of 11 participants felt the inductive explanations were easier to understand than the deductive explanations. These results are comparable to the results we obtained in the previous experiment when we used the proxy task rather than the actual task. We believe that the use of the think-aloud method may have impacted participants' behavior in this study. Specifically, because participants were instructed to verbalize their thoughts, they were more likely to engage in analytical thinking when considering the AI recommendations and explanations than they were in the previous experiment with the actual tasks, where their focus was primarily on making decisions. It is possible that while the think-aloud method is part of standard research practice for evaluating interfaces, it is itself a form of cognitive forcing intervention  [7] , which impacts how people perform on cognitively-demanding tasks such as interacting with an explainable AI system on decision-making tasks. The act of talking about the explanations led participants to devote more of their attention and cognition to the explanations, and thus made them behave more similarly to participants in working with the proxy task rather than those working with the actual task.
 DISCUSSION In this study, we investigated two hypotheses regarding the evaluation of AI-powered explainable systems: • H1: Results of widely accepted proxy tasks, where the user is asked to explicitly engage with the explanations, may not predict the results of realistic settings where the user's focus is on the actual decision-making task. • H2: Subjective measures, such as self-reported trust and preference with respect to different explanation designs, may not predict the ultimate human+AI performance. We examined these hypotheses in the context of a nutrition-related decision-making task, by designing two distinct evaluation tasks and two distinct explanation designs. The first task was a proxy task, where the users had to simulate the AI's decision by examining the explanations. The second task was the more realistic, actual decision-making task, where the user had to make their own decisions about the nutritional content of meals assisted by AIgenerated recommendations and explanations. Each of the tasks had two parts, where users interacted with substantially different explanation styles-inductive and deductive. In the experiment with the proxy task, participants preferred and trusted the AI that used inductive explanations significantly more. They also reported that the AI that used inductive explanations based its decision on more accurate examples on average than the AI that used deductive explanations. When asked "If you were asked to evaluate fat content of plates of food, which AI would you prefer to interact with more?"), the majority of participants chose the AI that provided inductive explanations. In contrast with the proxy task experiment, in the experiment with the actual decision-making task, participants rated the AI with deductive explanations as their preferred AI, and viewed it as more trustworthy and more helpful compared to the AI that used inductive explanations. The contrast in terms of performance measures was less pronounced. When attempting proxy tasks, participants demonstrated nearly identical accuracy regardless of explanation type. However, when attempting actual decision-making tasks and the AI provided an incorrect recommendation, participants ignored that incorrect recommendation and provided the correct answer significantly more often when they had access to inductive, not deductive, explanations for the AI's recommendation. These contradictory results produced by the two experiments indicate that results of evaluations that use proxy tasks may not correspond to results on actual tasks, thus supporting H1. This may be because in the proxy task the users cannot complete the task without engaging analytically with the explanations. Whereas, in the actual decision-making task, the user's primary goal is to make the most accurate decisions about the nutritional content of meals; she chooses whether and how deeply she engages with the AI's recommendations and explanations. This finding has implications for the explainable AI community, as there is a current trend to use proxy tasks to evaluate user mental models of the AI-powered systems, with the implicit assumption that the results will translate to the realistic settings where users make decisions about an actual task while assisted by an AI. We tested H2 on the actual decision-making task. The results show that participants preferred, trusted and found the AI with deductive explanations more helpful than the AI that used inductive explanations. Yet, they performed significantly better with the AI that used inductive explanations when the AI made erroneous recommendations. Therefore, H2 is also supported. This finding suggests that the design decisions for explainable interfaces should not be made by relying solely on user experience and subjective measures. Subjective measures of trust and preference are, of course, valuable and informative, but they should be used to complement rather than replace performance measures. Our research demonstrated that results from studies that use proxy tasks may not predict results from studies that use realistic tasks. Our results also demonstrated that user preference may not predict their performance. However, we recognize that evaluating novel AI advances through human subjects experiments that involve realistic tasks is expensive in terms of time and resources, and may negatively impact the pace of innovation in the field. Therefore, future research needs to uncover why these differences exist so that we can develop low burden evaluation techniques that correctly predict the outcomes of deploying a system in a realistic setting. We believe that the reason why explainable AI systems are sensitive to the difference between proxy task and actual task evaluation designs is that different AI explanation strategies require different kinds and amounts of cognition from the users (like our inductive and deductive explanations). However, people are reluctant to exert cognitive effort  [24, 25]  unless they are motivated or forced to do so. They also make substantially different decisions depending on whether they choose to exert cognitive effort or not  [12, 37] . In actual decision-making situations, people often choose not to engage in effortful analytical thinking, even in high-stakes situations like medical diagnosis  [31] . Meanwhile, proxy tasks force participants to explicitly pay attention to the behavior of the AI and the explanations produced. Thus, results observed when participants interact with proxy tasks do not accurately predict people's behavior in many realistic settings. In our study, participants who interacted with the proxy task felt that the deductive explanations required significantly more thinking than the inductive explanations. Therefore, in the proxy task where the participants were obliged to exert cognitive effort to evaluate the explanations, they said they preferred and trusted the less cognitively demanding explanations more, the inductive explanations. In contrast, in the actual task the participants could complete the task even without engaging with the explanations. Thus, we suspect that in the deductive condition participants perceived the explanations as too mentally demanding, and chose to over-rely on the AI's recommendation, just to avoid cognitive effort of examining those explanations. They also might have perceived the AI that provided deductive explanations as more competent because it required more thinking. One implication of our analysis is that the effectiveness of explainable AI systems can be substantially impacted by the design of the interaction (rather than just the algorithms or explanations). For example, a recent study showed that a simple cognitive forcing strategy (having participants make their own preliminary decision before being shown the AI's decision) resulted in much higher accuracy of the final decisions made by human+AI teams than any strategy that did not involve cognitive forcing  [17] . Inadvertently, we uncovered an additional potential pitfall for evaluating explainable AI systems. As the results of our qualitative study demonstrated, the use of the think-aloud method-a standard technique for evaluating interactive systems-can also substantially impact how participants allocate their mental effort. Because participants were asked to think aloud, we suspect that they exerted additional cognitive effort to engage with the explanations and analyze their reasoning behind their decisions. Together, these results indicate that cognitive effort is an important aspect of explanation design and its evaluation. Explanations high in cognitive demand might be ignored by the users while simple explanations might not convey the appropriate amount of evidence that is needed to make informed decisions. At the same time, traditional methods of probing users' minds while using explainable interfaces should also be re-evaluated. By taking into account the cognitive effort and cognitive processes that are employed during the evaluation of the explanations, we might generate explainable interfaces that optimize the performance of the sociotechnical (hu-man+AI) system as a whole. Such interfaces would instill trust, and make the user aware of the system's errors.
 CONCLUSION To achieve the aspiration of human+AI teams that complement oneanother and perform better than either the human or the AI alone, researchers need to be cautious about their pragmatic decisions. In this study, through online experiments and an in-person study, we showed how several assumptions researchers make about the evaluation of the explainable AI systems for decision-making tasks could lead to misleading results. First, choosing proxy tasks for the evaluation of explainable AI systems shifts the user's focus toward the AI, so the obtained results might not correspond to results of the user completing the actual decision-making task while assisted by the AI. In fact, our results indicate that users trust and prefer one explanation design (i.e. inductive) more in the proxy task, while they trust and prefer the other explanation design (i.e. deductive) more in the actual decision-making task. Second, the subjective evaluation of explainable systems with measures such as trust and preference may not correspond to the ultimate user performance with the system. We found that people trusted and preferred the AI with deductive explanations more, but recognized AI errors better with the inductive explanations. Lastly, our results suggest that think-aloud studies may not convey how people make decisions with explainable systems in realistic settings. The results from the think-aloud in-person study, which used the actual task design, aligned more with the results we obtained in the proxy task. These findings suggest that to draw correct conclusions about their experiments, explainable AI researchers should be wary of the explainable systems' evaluation pitfalls and design their evaluation accordingly. Particularly, the correct and holistic evaluation of explainable AI interfaces as sociotechnical systems is of paramount importance, as they are increasingly being deployed in critical decision-making domains with grave repercussions. Figure 1 : 1 Figure 1: The proxy task. Illustration of the simulated AI system participants interacted with: (a) is an example of an inductive explanation with appropriate examples. (b) is an example of a deductive explanation with misrecognized ingredients, where the simulated AI misrecognized apples and beets as avocados and bacon.
 Figure 2 :Figure 3 : 23 Figure 2: The actual task. Illustration of the simulated AI system participants interacted with. (a) is an example of incorrect recommendations with inductive explanations. Contrasting the query image with the explanations reveals that the simulated AI misrecognized churros with chocolate as sweet potato fries with BBQ sauce. (b) is an example of correct recommendation with deductive explanations.
 Figure 4 : 4 Figure 4: Performance in the actual decision-making task. (a) depicts the mean of performance among no-AI, no-Explanations and with-Explanations (overall) conditions. (b) depicts the mean of performance among inductive and deductive conditions, when the AI recommendation is correct and erroneous. Error bars indicate one standard error.
 Figure 5 : 5 Figure 5: Subjective evaluations in terms of trust and preference of the two AIs. Red and blue depict the percent of participants that chose inductive and deductive AI, respectively. (a) proxy task (b) actual decision-making task.

Abstract:
Explainable artificially intelligent (XAI) systems form part of sociotechnical systems, e.g., human+AI teams tasked with making decisions. Yet, current XAI systems are rarely evaluated by measuring the performance of human+AI teams on actual decision-making tasks. We conducted two online experiments and one in-person think-aloud study to evaluate two currently common techniques for evaluating XAI systems: (1) using proxy, artificial tasks such as how well humans predict the AI's decision from the given explanations, and (2) using subjective measures of trust and preference as predictors of actual performance. The results of our experiments demonstrate that evaluations with proxy tasks did not predict the results of the evaluations with the actual decision-making tasks. Further, the subjective measures on evaluations with actual decision-making tasks did not predict the objective performance on those same tasks. Our results suggest that by employing misleading evaluation methods, our field may be inadvertently slowing its progress toward developing human+AI teams that can reliably perform better than humans or AIs alone.
 CCS CONCEPTS • Human-centered computing → Interaction design; Empirical studies in interaction design.


The State of Research in Existential Risk

Existential and Global Catastrophic Risk An existential risk is one that threatens the premature extinction of Earth-originating intelligent life, or the permanent and drastic destruction of its potential for desirable future development  (Bostrom, 2002) . For most practical purposes, this refers to developments that might wipe out humanity, or lock us into a situation we (or other intelligent life on earth) cannot recover from, such as a major and permanent global civilizational collapse.  1  The Centre for the Study of Existential Risk, among other organisations within the existential risk research community, also includes a focus on global catastrophic risks. Here, a common definition is that used by the Global Catastrophic Risk Institute: Global catastrophic risk is the risk of events large enough to significantly harm or even destroy human civilization at the global scale" ((http://gcrinstitute.org/concept). For this definition of global catastrophic risk, human extinction risks would be a subset of a broader set of global catastrophic risks. However, precise definitions for these classes of risk are less important than having a shared sense of the magnitude of scope of events or developments under consideration, and the key characteristics of such events or developments. The study of existential and global catastrophic risk restricts us to events or trends that might lead to a full civilizational collapse. It rules out localised catastrophes, and global-scale events that would represent a tragedy but that would not impact our civilisation in the longer-term -unless these events were likely to play a key role in more severe and permanent cascades and collapses. Events as historically significant as Chernobyl, Hurricane Katrina, the Ebola outbreak, most of our wars in the 20 th century, and even the Spanish influenza would fail to constitute global catastrophes or existential threats. For this research community, their primary relevance is in what they can tell us about more severe possibilities within the relevant risk categories. On the other hand, various near-misses during the Cold War  (Lewis et al, 2014)  might plausibly have led to a global thermonuclear war with global catastrophic consequences, and thus this topic is firmly in scope. A clear example of an existential risk is the risk of an asteroid impact on the scale of that which wiped out the dinosaurs 66 million years ago  (Schulte et al, 2010) . An example of a global catastrophic risk that is not existential might include a large-scale pandemic disease outbreak  (Palmer et al, 2017; Millett & Snyder-Beattie, 2017) . Research and expert opinion indicates that it is unlikely that a natural pandemic outbreak could wipe out all humans across the globe given our distribution, immune variation, and other factors; and in most plausible scenarios global recovery seems likely in the longer term. It is also important to consider factors that increase stress or resilience on a global scale -events or developments that in of themselves would not be a global catastrophe, but might make it more or less likely for a global or existential catastrophe to occur. Climate change has the potential to result in global catastrophic consequences at the more severe end of the possibility spectrum -e.g. 5 degrees and up  (Wager & Weitzman, 2016) . But we might also consider less severe climate change as a stressor, as it could be expected to lead to major droughts and famines and other resource shortages, mass migration, geopolitical tension that could result in local or global war, and so forth. It could also lead to international conflict, for example over the use of controversial mitigation techniques such as sulphate aerosol geoengineering technologies. More generally, many of the specific risks we will look at need to be placed in the context of a world with a rising population, rising resource footprint, more extreme weather events, increasing pressures on ecosystem services, a changing physical and electronic infrastructure, and changing geopolitical pressures -and a world with a range of technologies more powerful than any we've had in previous centuries.
 Topics of focus within the existential and global catastrophic risk research community Risks of particular focus within the existential risk community include those relating to our interaction with the global environment (for example, extreme climate change; globally catastrophic biodiversity loss; and risks from natural pandemic outbreaks). They also include risks from scientific and technological advances, such as risks from engineered pandemics and future advances in synthetic biology and other biotechnologies; nuclear winter; and risks from future advances in, and applications of, artificial intelligence. Some focus is given to risks from asteroids and supervolcanoes, but these have been less highly prioritised to date, due to the low frequency of occurrence of the relevant events; the last global catastrophe-level asteroid impact event was the impact that created the Chicxulub Crater 66 million years ago, and the last likely global catastrophe-level supervolcano is thought to have been the Toba eruption 70,000 years ago  (Ambrose, 2000) . The community also has research groups working on global systemic risks, interactions and cascades between risks, risks from future technologies such as neurotechnologies, advances in nanotechnology and other technologies. Researchers also focus on broader or more cross-cutting themes, such as governance challenges associated with global catastrophic risks, ethical considerations relating to the value of future generations, analysis of the costs and value of reducing existential and global catastrophic risk relative to other global priorities, foresight, horizon-scanning and road-mapping exercises for risks and relevant sciences and technologies, and methodological issues relating to reasoning about extreme events under great uncertainty. For most individual global catastrophic and existential risks (with the possible exception of risk from advanced artificial intelligence), there are individual research communities working on these topics; albeit sometimes focusing more so on risks at a lower end of the spectrum. Some have been doing so for decades. For example, there are many centres doing valuable work on nuclear non-proliferation and security, pandemic preparedness and surveillance, and bioweapon governance -research leaders from these communities are represented at the Garrick Colloquium. There are numerous centres working on different aspects of climate change, biodiversity loss and other environmental risks and resource-related challenges. There are a number of excellent groups working on global systemic risks. NASA, the Planetary Defense community and others work on asteroid scanning and mitigation strategies. Furthermore there are a range of centres in academia, think tanks, government and elsewhere working on more cross-cutting issues such as risk governance and international security, foresight and scenario planning, and so forth. At a level below global catastrophic risk, there is excellent work on risk modelling being done in, and associated with, the reinsurance industry. The existential and global catastrophic risk research community collaborates with, and draws on, research from these communities extensively. It aims to complement such work by looking at existential and global catastrophic risks as a class of risks, with the aim of: • Identifying the particular challenges associated with risks of this magnitude -whether they be scientific, analytic, ethical, or to do with governance, coordination, planning, or perception. • Identifying risk areas where insufficient attention has been paid to the most extreme scenarios. • Examining how these different risks, and other global developments, may interact with each other. • Identifying previously unidentified or potential future risks • Trying to distinguish which extreme scenarios are plausible and worthy of further work, even if they may be low probability, as opposed to those that can be dismissed as science fiction.
 Recent growth of this community There has been a lot of recent growth within the existential risk research community.
 What can we learn from this research, and what can we do with that knowledge? It is worth considering the insights we can gain from studying risks of this magnitude that may not be gleaned from bodies of work analysing and mitigating risk more broadly, or risks of a lower magnitude. One source of value is simply to be able to rule out scientifically implausible scenarios, in order to better focus our attention on existing threats and plausible future threats. This is of particular value for low-probability, extreme impact events, especially those that may be entirely novel. Many researchers argue the value of understanding existential risk on the grounds of their long-term moral significance. If we can demonstrate that certain threats have the potential for human extinction, or permanent collapse, then it is argued that the importance of mitigating them increases dramatically, due to the fact that they would not only harm current generations, but wipe out the potential for a huge number of future lives. There may be specific actions we could take that are designed to mitigate the most severe versions of these threats in particular. These might include establishing a very strict ban on specific types of virus research or bioweapons, or on research on AI systems that could both come up with a realistic model of the world and engage in recursive self-improvement. They could also include seed banks, shelters and alternative foods, so that even in many otherwise extinction-level events we might increase the odds of continuation of the species. In addition, there may be strategies we would avoid adopting unless we had reasonable cause to believe we were headed for a world in which a particular type of global catastrophe were likely. For example, it would be extremely foolish and unconstructive to call for a ban on artificial intelligence research at this point. At some point in the future however, a body of evidence might indicate that certain developments with catastrophic potential are likely within several years. This then might be reason to consider a temporary moratorium on progress to enhance the capability of AI systems, while various containment and safety measures were being explored. This is not without precedent, even at lower-levels of risk: the US White House issued a temporary moratorium on gain-of-function influenza research several years ago to allow time for more risk-benefit analysis  (Lipsitch & Inglesby, 2014) . Or consider a world in which evidence indicated we were headed for climate change of 5 degrees or over, in the absence of drastic action. In these circumstances we might consider deploying technologies, such as sulphate aerosol geoengineering, which we might be hesitant to deploy under less severe scenarios  (Crutzen, 2006) . Our reservations might be based on concerns over risks posed by the intervention itself, thorny global governance challenges that the intervention presents, or questions over public acceptance. These may be sound reservations that would rule out these strategies in all but the most exceptional of circumstances. A research community developing indicators that we may be approaching unusually dangerous global circumstances, and developing strategies for last-ditch solutions, may be of considerable value given the risks we may face in the coming century -even if many of these strategies are never needed or deployed. Lastly, by studying the particular scientific, governance, ethical, and communication issues that arise when confronting one global catastrophic risk, we can learn valuable lessons to draw on for future challenges. For example, climate change in some ways exemplifies a lot of the issues that make existential and global catastrophic risk especially challenging. There is still a lot of scientific uncertainty over timelines, probability and pathways to the most severe impacts. The scale and impact are difficult to grasp, impossible to see, and the most severe consequences will fall on future generations. We still don't have broad public acceptance of the science, especially in the US. It involves countries around the world coordinating, each making near-term sacrifices in favour of the longer-term future. Yet the Paris Agreement was extremely encouraging. It involved 194 countries committing to make sacrifices in the interests of future generations in the face of these challenges. Despite the setback of the US's recent announcement of intent to withdraw, this remains an important achievement and a critical step towards progress on climate change. By learning from the successes and failures of this process, as well as from the history of nuclear non-proliferation and international diplomacy, norms and conventions prohibiting the use of biological weapons, and other global processes to manage and mitigate global risk, we can be better prepared for the future.
 Key challenges in existential and global catastrophic risk There is a large body of expertise from the risk sciences that is of relevance to the study of existential and global catastrophic risk. However, a number of characteristics make these risks particularly challenging to analyse using normal risk analysis approaches. These include the difficulty in estimating the probability and expected impact of rare or even unprecedented events, where there may be sparse data to draw on; and the changing nature of some risks, in particular those associated with rapidly developing technologies (especially those interacting with a rapidly developing infrastructure). For some global catastrophic risks, probability is relatively straightforward to quantify. For asteroid impacts, for example, we can look to sources of evidence such as the earth's fossil record, patterns of impact craters on the Moon and on Mars, datasets of asteroids passing our field of vision, and use these to make a reasonable estimate of the frequency with which we might expect an asteroid of a given size to hit the earth. The pathways by which an asteroid impact would result in global catastrophe are also relatively straightforward; therefore we can estimate the expected global harm expected from asteroids of different sizes. However, a similar analysis is much less straightforward for other risks. This can be illustrated by the example of catastrophic climate change. For a start, there is great uncertainty about the sensitivity of the earth system to the effects of our activities. The possibility of severe climate change is affected by factors including to but not limited to the potential for methane release from beneath the melting arctic permafrost, and from the seabed; how much CO2 the deep ocean can absorb; the possibility of collapse of Antarctic and Greenland ice sheets; the possibility that the gulf stream might halt. The most worrying scenarios involve a combination of these factors driving each other as part of a positive feedback loop. While ongoing research will help us understand these factors and their interactions more clearly, it is very difficult to assign a meaningful probability to an outcome such as >5 degree climate change in the 21 st century under a certain emission scenario. It may be that attempting to assign strict probabilities is the wrong approach; an alternative would be to develop frameworks of 'safe operating thresholds' with wide error bounds, exemplified by the 'Planetary boundaries' framework put forward by Johan Rockstrom and colleagues at the Stockholm Resilience Centre  (Steffen et al, 2015) . The expected harm from long-term climate change will also depend very heavily on the extent to which various mitigation and adaptation strategies are adopted, and how successful they are. Similarly, in the case of global pandemics, we we can ask scientific questions about the possibility of the 'perfect virus' with high health impact, high infectivity, and long incubation time. However, the scale and severity of impact will be predicated by many other factors -movement of humans or other vectors, capabilities of health services, the response of the population, and more. It is plausible that the bulk of the damage might not even be caused by the virus, but instead by a broad infrastructure collapse as emergency services and hospitals are overwhelmed, just-in-time food delivery is disrupted, and other systems underpinning societal order collapse. However, none of these challenges are insurmountable. Modelling and analysis of factors that contribute to these risks can deepen our scientific understanding. This can help us establish estimates for probability and impact, and in some cases rule out concerns entirely. Where past examples are sparse, there is value in drawing on counterfactual examples of 'near misses', as described by Gordon Woo  (Woo, 2016)  and others. Design and analysis of scenarios can help in identifying key considerations and interactions. This may help us identify key interventions that reduce risk significantly, even if in instances where it is difficult to assign tight probabilities to events. Other risk analysis techniques, and interventions to mitigate risks at scales smaller that global catastrophe level, also provide a range of useful insights for the analysis of global catastrophic risks, as shown by the work of GCRI and others. By studying global catastrophic risks as a class of risks, we can identify shared characteristics of global catastrophe events, which may help us identify common strategies that aid us in becoming more resilient as a species against a broad set of risks. For example, a number of global catastrophic events (global nuclear war, supervolcano eruption, asteroid impact) would result in large amounts of particulate matter being ejected into the atmosphere, resulting in a disruption of photosynthesis  (Maher and Baum, 2013) . The development of alternative foods that are not dependent on sunlight, and strategies to scale up production of these food sources rapidly, would be robust in the face of a broad range of catastrophe events  (Denkenberger and Pearse, 2014) . The maintenance of permanent seed banks, manned shelters suitable for lengthy use, and information vaults represent similar safeguards. Similar strategies, useful for reduction of a broad range of risks, are likely to be feasible at the level of national and international governance  (Farquhar et al, 2017; Cotton-Barrett et al, 2016) .
 Case Study: Potential future global risks from artificial intelligence In this final section, I aim to present this research community's work over the last decade on potential future risks from artificial intelligence as a case study on how the field has made progress on a particularly novel area of concern, focusing on the broad strategies involved. This is complemented by a more technical analysis of AI risk provided by Seth Baum. The theoretical nature of the risk: Early work, carried out mainly by the Machine Intelligence Research Institute and the Future of Humanity Institute, aimed to explore the possibility of artificial general intelligence of greater-than-human-ability. This was coupled with the aim of exploring and characterising the theoretical risk associated that such a development could pose, drawing on input from experts in computer science and other fields. Rather than focus on difficult-to-characterise questions like consciousness, sentience, and evil intentions, the work instead focused on foundational issues like: -The difficulty of designing safe goals for very capable, powerful systems able to take a very wide range of actions in a wide range of environments, where the actions could have a wide range of consequences. -Certain predicted behaviours that might be expected from a powerful optimizing agent, such as a drive to acquire additional resources, a drive to avoid being switched off prior to completion of its goal, or a drive to improve the system's own capability. -The theoretical possibility and limits of recursive self-improvement. This refers to the possibility that a sufficiently capable system may be able to surpass human programmers in its ability to design the next generation of systems. It is hypothesised that this could in turn result in a 'chain reaction' of performance improvement  (Yampolskiy, 2015) , rapidly leading to a system far beyond human capability in most cognitive domains. -The challenge that for most of the relevant design and control processes, it may be necessary to solve a range of technical and theoretical issues ahead of time, before certain critical thresholds in capability are reached. Beyond these thresholds, it may be much more difficult to intervene effectively due to the level of capability and autonomy of subsequent iterations of the system. It is worth noting that in principle, it is possible that systems may be developed which have a much greater ability to engage in science, engineering, and manufacture, and a greater ability to manipulate the global environment than we humans have. This level of power raises concerns of global catastrophic or existential risk, unless very carefully developed. These programmes of research were then published in a book, Superintelligence  (Bostrom, 2014) , which received a lot of attention, as well as in many further academic papers. These lines of argument are by no means uncontroversial -many experts disagree on various points. Most experts consider artificial general intelligence to be decades out of reach as a scientific milestone, and some expect hundreds of years of progress to be needed  (Grace et al, 2017) . Some are sceptical about the hypothesis that rapid progress, enabled by the engagement of the AI systems themselves in the research and development process, could occur once a certain level of capability and generality is reached (e.g. see  Walsh, 2016 ). Yet others are sceptical that such systems would be likely to demonstrate the traits of agency, autonomy and goal-driven behaviour that may make the actions of such systems difficult to predict or intervene on. Some experts hold that these is a limit to how much meaningful work can be done at this point in time to ensure the safety and stability of future systems, given the limits that can be meaningfully predicted about the theoretical underpinnings and engineering design of these future systems, as well as the limits of our knowledge regarding the nature of intelligence. Others have raised the concern that a focus on risks from powerful future systems may distract focus from more near-term risks and opportunities associated with the current state of the technology. However, a growing body of experts in AI consider many of these concerns plausible and worthy of further study (e.g. see  Dafoe and Russell, 2016) . While the level of capability warranting global catastrophic concerns is still decades away or longer, ongoing research on the matter is warranted by the magnitude of the challenge, and the range of technical and governance questions in need of study in advance of such developments. It is the view of this author that these concerns should not be used as a rationale to propose slowing down progress on the technology at this point in time. Nor should this research take the place of necessary and valuable work on near-term opportunities and challenges posed by AI, but rather should complement it. Broader scientific engagement: The next steps for the community were to engage more deeply with the scientific artificial intelligence community in academia and industry-to explore, discuss and debate these arguments, as well as other risks that may be associated with AI. In 2015, a landmark conference was held in Puerto Rico, with representatives of the leading companies working explicitly towards a vision of general AI, alongside experts in governance, law, economics, risk and other relevant fields. The conference resulted in an open letter calling for more research on AI that was safe, robust, and beneficial, and for ongoing attention to issues relating to the longer term. The letter was signed by research leaders in artificial intelligence across industry and academia, as well as leading experts in a range of different fields, and was accompanied by a paper outlining research priorities, and a grants programme to support relevant work. The conference has been followed by a programme of activities to foster collaboration with more of the machine learning community both near-and long-term issues relating to safe design and risk. This has including a series of workshops organised by CSER and others at the major machine learning conferences (ICML, NIPS, IJCAI, DALI). T In 2017 FLI organised a follow-on to the Puerto Rico conference in Asilomar, resulting in the endorsement by many leading researchers of a set of principles for the long-term development of AI.
 Technical research: In parallel, much of the work in the last two years has focused on translating some of the more foundational questions raised by early work at FHI and MIRI and elsewhere into crisp technical research problems that can be worked on today. This includes approaches involving fundamental mathematical frameworks for agent decision-making and behaviour, as well as research programmes exploring how some of the behaviours that would be of concern in long-term systems may manifest in the near-term systems we are building currently. A number of research agendas have been published in the last year, and several of the leading companies focused on general AI now have safety teams exploring these issues. In addition, there has been substantial growth in projects mapping progress and trajectories in AI in domains relevant to both narrow and general artificial intelligence. Global engagement: Within the existential risk community, more thought is now going to the particular political and governance challenges that may emerge as we move closer towards more powerful and general AI systems, with programmes starting up at FHI, CSER, the Centre for the Future of Intelligence. Long-term impacts and risks have risen on the agenda for governments in Europe and the United States, for example being mentioned as worthy of further study in the US Office for Science and Technology Policy's recent report on preparing for the future of artificial intelligence. One priority that has emerged is the need for greater global engagement, particularly with research leaders and other stakeholders in China, but also India, Japan, and emerging hubs in Africa. Any global conversation around the future of artificial intelligence, and potential global benefits risks associated with it, needs to have global representation. From a pragmatic point of view, China is on course to emerge as a scientific leader and agenda-setter in AI research over the coming decade. If at some point in several decades we truly do approach a level of technological breakthrough with global risk consequences, we are unlikely to be able to achieve a safe transition without a strong level of global cooperation and trust, which is going to require a lot of dedicated work to achieve. Now is a good time to start laying the groundwork.
 Conclusion We are entering a century in which humanity will be confronted with unprecedented threats to global civilisation. Some of these may result from the manner in which our footprint as a species strains our global environment, such as the impacts of climate change and biodiversity loss. Some may result from the development and deployment of increasingly powerful technologies, whether due to malevolent use or unintended consequences. The challenge posed by the analysis and mitigation of these threats requires an interdisciplinary approach: a community that can draw on the best expertise from the risk sciences, as well as the expertise of scientists, law and governance specialists, ethicists, and others. It also requires a community that can draw on expertise on different types and sources of risk, and consider both lessons that can be applied across risks, and the interactions that are likely to occur between different global developments. Many of the most useful tools in global risk analysis have been drawn from the risk science literature, and deeper collaborations between the existential risk community and the risk science community are likely to be increasingly important in the years to come. The first dedicated centre established was arguably Nick Bostrom's Future of Humanity Institute (FHI; https://www.fhi.ox.ac.uk/) in Oxford in 2003. The FHI has focused on cross-cutting global catastrophic risk analysis, philosophical analysis on the global importance of reducing existential risk, and in the last decade has placed its strongest focus on characterising potential risks from artificial general intelligence and superintelligence; a lot of this has been in collaboration with the Machine Intelligence Research Institute in Berkeley. The Global Catastrophic Risk Institute (GCRI;The Centre for the Study of Existential Risk (CSER; http://cser.org) was founded in 2012 by Martin Rees, Huw Price, and Jaan Tallinn, although its first research grants were secured, and first postdoctoral researchers hired, more recently in late 2015 and 2016. We now have a research team beginning work on biological threats, extreme climate change, ecological tipping points, catastrophic risks related to future advances in artificial intelligence, and analysis of emerging technologies such as geoengineering. We also have postdocs working on more cross-cutting themes such as horizonscanning and foresight for extreme risk, responsible innovation in risky sciences and technologies, population growth and resource use.The Future of Life Institute (https://futureoflife.org/) was founded in 2014, focusing on artificial intelligence, climate change, risks from biotechnology, and risks from nuclear weapons. It has organised two highly successful conferences on the future of artificial intelligence and potential risks it might bring, resulting in a widely signed and shared open letter on the responsible development of AI, a grants programme to support work on AI safety, and a set of principles aimed at promoting the beneficial development and application of AI within the research community and more broadly. It has also organised a conference on nuclear war, and has engaged in activities to encourage divestment from nuclear weapons.Several more recent initiatives are underway within academia, including at Stockholm, Warwick (United Kingdom), and Australia National University, and other world-leading risk centres such as the Garrick Institute (UCLA) are increasingly including global catastrophic risk within their remit, indicating that a diverse range of new expertise will be brought to bear on these topics in coming years. http://gcrinstitute.org/) in the US was founded in 2011, focusing on risks including bioweapons, nuclear war, artificial intelligence, and natural events, and employing risk analysis methodology. Under Seth Baum and Tony Barrett's leadership, it has played a key role in establishing links between the existential risk community and experts in these fields.
			 Electronic copy available at: https://ssrn.com/abstract=3446663

Abstract: In the last fifteen years there has been substantial growth in research on existential risk -the category of risks that threaten human extinction, or the permanent and drastic reduction of humanity's future potential. A number of new organisations focused explicitly on existential and global catastrophic risk have been founded in recent years, complementing the long-standing work of existing centres focused on specific risk areas such as nuclear war, biosecurity, climate change and systemic risk. This paper provides a brief overview of the emergence of this new research community, and provides a case study on the community's research on potential risks posed by future developments in artificial intelligence. There exists the opportunity for powerful collaboration between the new approaches and perspectives provided by the existential risk research community, and the expertise and tools developed by the risk sciences for risks of various magnitudes. However, there are a number of key characteristics of existential and global catastrophic risks, such as their magnitude, and their rare or unprecedented nature, that are likely to make them particularly challenging to submit to standard risk analysis, and will require new and specialised approaches. 1 It is worth noting that Bostrom also considers some more unusual possibilities, such as the emergence of permanent totalitarianism