Jose Hernández-Orallo
Fernando Martínez-Plumed
Shahar Avin
Jess Whittlestone
Seán Ó H Éigeartaigh

AI Paradigms and AI Safety: Mapping Artefacts and Techniques to Safety Issues

INTRODUCTION As in any other scientific or engineering discipline, many AI researchers work within a well-established paradigm, with some standard objects of study, problems to solve, and associated formalisms and terminology. Understanding past, current and future AI paradigms is an important source of insight for funding agencies, policymakers, and AI researchers themselves, because it shapes how we think about what problems AI research is aiming to solve, what methods are required to solve them, and what the wider implications of progress might be. We believe that thinking clearly about AI paradigms is particularly crucial for AI safety research: an increasingly important area concerned with understanding and preventing possible risks and harmful impacts in the design and deployment of AI systems. For the purposes of this paper, we define AI safety broadly, to include both risks from AI systems for which the source of risk is accidental, ranging from unpredictable systems to negligent use, and non-accidental risks such as those stemming from malicious use or adversarial attacks (which might sometimes also be referred to as AI security risks.) This includes risks with many different types of consequences: including human, environmental, and economic consequences. AI safety is becoming particularly important as AI is increasingly used to automate tasks that involve interaction with the world. A characteristic example is AI being used in many components of self-driving cars such as perception, reasoning and action. Research in AI safety has typically analysed a specific risk or issue under a particular existing paradigm, such as interruptibility for reinforcement learning agents  [33] , adversarial attacks on deep learning systems  [43] , or fake media produced with GANs  [21] . Poor choices or misinterpretation of current and future paradigms may result in AI safety research focusing in the wrong places: analysing scenarios that will not take place in the future, or that will at best manifest in completely different ways. For example, concerns about adversarial attacks on current deep learning methods have led to many papers proposing methods to defend against malicious perturbations, but it is not clear that these actually relate to plausible security concerns  [16] . Though adversarial examples help us understand how brittle current deep learning methods can be, the kinds of attacks that some safety research is concerned with can only arise in contrived interactions, and make strong assumptions about the goal, knowledge, and action space of the attacker. In contrast, other risks have been ignored because they are outside of the current paradigm: for instance, attacks which go beyond an "independent" or functional interpretation of a neural network, and instead use the "information from previous frames to generate perturbations on later frames"  [41] . Thinking about different paradigms is also important for clearly assessing safety considerations and risks in concrete real-world applications. For instance, the risks of using AI in vehicles will be perceived differently depending on whether we expect AI to be assisting or replacing human drivers, whether a self-driving vehicle is considered as a single autonomous agent, or whether we consider the whole traffic system as a swarm of interacting agents. Even as research becomes oriented towards future risks (some years or even decades away) the different safety issues associated with different AI paradigms have not been explicitly addressed in the literature  [6, 2, 23, 14] . As AI research is fast evolving and likely to result in increasingly powerful systems in future, it is crucial for AI safety to explore issues associated with a broader range of possible AI paradigms, and to explicitly discuss what assumptions about paradigms and safety issues are being made in each research paper or project. This will not only enable the research community to identify the effects of less-explored paradigms on safety concerns, but could also increase awareness among AI developers that the paradigms they work in have consequences for safety, potentially highlighting approaches that can eliminate or reduce safety risks. In this paper, we present a structured approach for thinking about paradigms in AI and use this as the basis for empirical analysis of how AI safety issues have been explored in the research literature so far. We begin by defining what we mean by 'paradigms' in AI more precisely, drawing on literature in the philosophy of science to distinguish two different types of paradigms in AI: artefacts and techniques. Drawing on existing research and the expertise of several AI and AI safety researchers, we outline a preliminary taxonomy of fourteen different AI techniques and ten different AI artefacts. We then discuss how these different techniques and artefacts relate to AI safety issues. We use AAAI's 'AI topics' database to conduct a grounded empirical analysis of the historical evolution of these different paradigms, and the safety issues associated with them. Our analysis identifies a number of gaps in AI safety research where certain combinations of techniques, artefacts and safety issues need to be addressed. We conclude the paper by discussing implications for future research in AI safety.
 DEFINING PARADIGMS IN AI Before discussing paradigms in AI and how they relate to safety issues, we need a clearer account of what an AI paradigm is. In the philosophy of science, a 'paradigm' is an important concept used to capture how theories, methods, postulates and standards evolve and change within a scientific discipline  [22] . The concept of a 'technological paradigm' plays a similar role but with a somewhat distinct focus, emphasising how each technological paradigm defines its own concept of progress, based on making specific technological and economic trade-offs  [12] . In computer science, the term 'paradigm' is commonly used to refer to different types of programming languages with different features and assumptions: imperative, logic, functional, object-oriented, distributed, event-oriented, probabilistic, etc. The theory and practice of programming languages depends heavily on these paradigms. Many safety issues in programming and software engineering, and verification in particular, cannot be addressed -conceptually or technically-without making paradigms explicit  [5, 20, 40, 42] . For instance, compared to imperative programming languages, declarative languages minimise mutability issues  [10]  due to the use of immutable data structures, as well as reduce state side-effects  [29]  by discouraging the utilisation of variables in favour of more sophisticated constructs (e.g., data pipelines or higher-order functions). In the context of AI, the concept of 'paradigms' has been used informally to refer to different broad families of technical or conceptual approaches: 'symbolic' vs 'connectionist', reasoning vs learning, expert systems vs agents. One way to identify paradigms in a field and how they have changed over time, suggested in Kuhn's original formulation, is to look at the approach taken by major (text)books in the field. Russell and Norvig  [36] , for example, popularised the 'agents' view of AI in the 1990s, and Goodfellow et al.  [17]  has consolidated 'deep learning' as a central approach AI research this decade, going far beyond one specific technique. However, there is no clearly agreed-upon definition of what counts as a paradigm in AI.  [9]  defined an AI paradigm as "the pair composed by a concept of intelligence and a methodology in which intelligent computer systems are developed and operated", which led him to identify three paradigms of AI: behaviourist, agent and artificial life. More than twenty years later, these do not seem to adequately reflect the approaches and assumptions within AI research today. Perhaps this difficulty clearly distinguishing AI paradigms from mere trends stems at least partly from the ambiguity in the concept of paradigm itself, a criticism which has been made within the philosophy of science. Masterman identifies three different conceptions of paradigm in Kuhn's writings: the metaphysical, the sociological AI Safety Issue AI Technique AI Artefact Figure  1 . Ways of analysing an AI safety issue, combined with an artefact and/or a technique category. In real systems, techniques underpin the creation or operations of an artefact, where the real hazards occur. Occasionally, researchers can think of a safe issue in a very abstract way, without committing to any particular artefact or technique (e.g., value alignment). All relations (arrows) are many-to-many. and the artefact/construct  [27] . Peine makes a reconstructive effort to combine the second and the third conceptions, suggesting that a paradigm can emerge when a subcommunity within a field makes a commitment to a certain set of techniques  [35] . For instance, we can see this phenomenon in the deep learning community, which made early commitments before this approach had established a dominant position within the field. We suggest distinguishing between dominant trends: research commitments to specific techniques, aims, or assumptions that may be relatively short-lived; and paradigms where these trends lead to more established, long-term commitments. These ambiguities in the concept of a paradigm are apparent in the context of AI, where 'paradigm' may be used to distinguish both different techniques for solving problems in AI (e.g., Monte Carlo search vs. SAT solver, deep learning vs. genetic algorithm), and to capture different conceptions of possible types of AI systems (e.g. expert systems vs. agents). Many technical papers can easily be categorised by looking at the first formal definitions that appear after the introduction, which tend to identify the kind of AI system and the kinds of techniques they use to solve a problem. We suggest that it may be helpful, in thinking about paradigms in AI, to distinguish explicitly between these two types of constructs: • Conceptual Artefacts: broad conceptions of what current and future AI systems (will) look like, e.g., autonomous agents, personal assistants  [24] , AI extenders  [19] , conceptions of superintelligence  [6]  and Comprehensive AI Services  [13] . • Research Techniques: the research methods, algorithms, theoretical technical results and methodologies involved in the development of these current or future systems, such as SAT solvers, deep learning, reinforcement learning, evolutionary computing, etc. Artefacts and techniques are important components of a paradigm, and together (weakly) define a paradigm. AI artefacts and techniques are often related to each other, insofar as certain techniques will often be better suited to certain types of artefacts (reinforcement learning is a technique category that is applicable for agent-like artefacts, for example). However, distinguishing between artefacts and techniques can help us to think more clearly about associated safety issues. Some safety issues will arise when combining an artefact with some techniques but not others: for example, when building a classifier (artefact), interpretability is likely to be a challenge for safety if using deep neural networks (technique), but less so if using simple decision trees with conditions that are expressed over the original attributes. Recognising these differences is important for understanding the scope of safety challenges and what solutions may be needed. In the case of autonomous vehicles, for example, both techniques and artefacts have changed over time, changing conceptions of safety. Early work in the 1980s-1990s, such as the Eureka 24th European Conference on Artificial Intelligence -ECAI 2020 Prometheus Project, emphasised systems for improved vehicle-tovehicle communications and driver assistance (e.g. collision avoidance), whereas more recent initiatives emphasise the development of fully independent self-driving vehicles. These are very different AI artefacts, with a more fully autonomous system evoking much broader safety concerns in general. However, to think more concretely about how evolving approaches to autonomous vehicles change safety challenges, it may also be important to look at how techniques have changed over time. For example, as improvements in computer vision enable more limited sensors to be replaced by cameras or radars, we may need to focus specifically on the vulnerabilities introduced by current techniques used in machine perception. Sometimes we may want to broadly assess all possible risks associated with a specific AI artefact independently of the technique used (e.g. issues associated with AI extenders), or those associated with a technique independently of the artefact (e.g. safety challenges for deep learning). In other cases, we may want to more narrowly focus on a specific safety issue associated with a particular artefact (e.g. interruptability for industrial robots), or a specific issue associated with a particular technique (e.g. interpretability of neural networks.) Figure  1  shows these interrelationships: how AI safety issues may be considered relative to either techniques, artefacts, or both.
 IDENTIFYING TECHNIQUES AND ARTEFACTS We can now begin to explore different ways of categorising AI paradigms by decomposing them into techniques and artefacts. While other accounts of 'AI paradigms' have been proposed, such as by the "One Hundred Year Study on Artificial Intelligence" at Stanford University  [39] , the categories frequently mix techniques with artefacts and even subfields.
 AI techniques We develop a preliminary categorisation of AI techniques by building on the bibliometric analysis of  [26]  and  [32, Tab. 6 ]. Martínez-Plumed et al.  [26]  identify nine 'facets' for the study of the past and future of AI, using data on all accepted papers from AAAI/IJCAI conferences (1997-2017), and AI Topics documents, an archive kept by AAAI containing news, blog entries, conferences, journals, and other repositories. Niu et al.  [32]  identify 30 high-frequency keywords from 20 relevant journals in AI from 1990 to 2014. Both capture many keywords and categories which appear to describe AI techniques, and are relatively similar.  4  We construct a list of AI techniques, grouped into 14 categories, based on selecting relevant and representative keywords from these two analyses. We chose these categories based on three principles: (1) the techniques are sufficiently general to encompass groups of approaches in AI that have been recognised by other approaches, (2) overlapping in the techniques is allowed, as there is a high degree of hybridisation and combination in AI, and (3) subcategories are retained for particularly large categories, such as 'machine learning'  5  The list of categories is shown in Table  1 .
 AI artefacts For AI artefacts, we would like our categories to be more stable over time and less tied to specific research techniques and applications. One way to look beyond current trends is to consider textbooks or even historical accounts of AI  [28, 7, 31] . It may also be helpful to begin by considering broad 'characteristics' of AI systems independent of the specific techniques used to develop them  [18] . To generate a preliminary set of artefacts, a group of multidisciplinary researchers conducted a systematic, interactive procedure based on the Delphi method  [11] . The group began by independently brainstorming possible candidates for categories of artefacts based on a preliminary discussion of those arising from AI textbooks and different AI system characteristics. Two criteria were provided to structure this initial brainstorm: (1) artefacts should ideally have minimum overlap, each capturing distinctive functionalities, and (2) artefacts should be defined independently of how functionalities are achieved (i.e. independent of techniques). Group members each proposed lists of distinctive type of AI systems, supported by exemplars, which were revised using an iterative process until the list of answers converged towards consensus. These were then clustered hierarchically to produce a set of artefacts which could cover the space of actual and potential AI systems as exhaustively as possible. The categorisation of AI artefacts in Table  2  is the result of this process. Building on the categories identified in  [18] , we can further characterise these different artefacts in terms of their integration with the external environment, in three ways: (1) how it interfaces with the environment (e.g. via sensors and actuators, via digital objectives, or via language); (2) the dynamics of this integration (whether it is interactive or functional); and (3) the location of this integration (whether it is centralised, distributed, or coupled). For example, an agent interfaces with the environment via sensors and actuators in an interactive and centralised way. A swarm has similar characteristics except that the location of its integration is distributed (across different units) rather than centralised. We also provide examplars for each category: self-driving cars and robotic cleaners are examples of agent-type systems, whereas a multiagent network router or drone swarm are examples of swarm-type systems. We outline these characteristics to show that the artefacts are sufficiently comprehensive and distinctive to provide a useful basis for further analysis, though as with any clustering there are some overlaps and borderline cases.
 Empirical analysis of techniques and artefacts Using our categories of techniques (e.g., neural networks, information retrieval, cognitive approaches, etc.) and artefacts (e.g., estima-24th European Conference on Artificial Intelligence -ECAI 2020 tor, agent, dialoguer, etc.), we now conduct an empirical analysis of how these different paradigm components have appeared in research papers and other related literature. This analysis will allow us to explore more empirically which paradigm elements have been prominent at different times, and will form the basis for investigating the relationship between paradigms and safety issues later in this paper. To conduct this analysis, we work with AI Topics 6 , an official database from the AAAI, using the documents from the period 1970-2017 (complete years). This archive contains a variety of documents related to AI research (news, blog entries, conferences, journals and other repositories) that are collected automatically with NewsFinder  [8] . We divide the archive into research documents  7  and non-research documents. From the ∼111K documents gathered, ∼11K are research papers and the remaining ∼100K are mostly media. With a mapping approach between the list of exemplars (tokens) of techniques and artefacts (e.g., "Deep Learning", "GAN" or "perceptron" are illustrative exemplars of the "neural network" technique; "classifier", "decision-making" or "object recognition" are exemplars of the "estimator" artefact5 0 ), and the tags obtained from AI Topics (substrings appearing in titles, abstracts and metadata), we summarise the trends in a series of plots. The evolution of techniques and artefacts in these documents (i.e., fraction of papers focusing on the list of tokens for each technique category or artefact) is shown in Figure  2 , where the data has been smoothed with a moving average filter in order to reduce short-term volatility in data. Note that this figure shows the aggregation of categories (techniques and artefacts) where each area stack is scaled to sum to 100% in every certain period of time. In a way, this can be understood as a non-monolithic view of polarities and intensities in sentiment analysis (e.g., identifying sentiment orientation in a set of documents). We see some techniques and artefacts are particularly dominant, as might be expected. For instance, looking at the techniques (left plot), 'knowledge representation and reasoning' takes almost half of the proportion of documents between 1980-1990, while 'general machine learning' becomes more relevant from 2005. We also see an important peak of multiagent systems around 2000. When we look at the artefacts (right plot), we see that 'extractor' (which includes expert systems) became very popular around 1990 but in a matter of a decade was overtaken by 'agent', which became dominant in 2000. We see today that five kinds of artefacts account for about 90% of all mentions: estimators, agents, optimisers, extractors and creators. These patterns also show that some artefacts are not new as a paradigm component, especially when we consider them in a more abstract way. For instance, it is sometimes understood that GANs introduced a new paradigm, but other kinds of generative systems have been around in AI since its exception, as we see in the violet band in Figure  2  (right). We can also use this analysis to explore what techniques and artefacts are prominent in the media. In Figure  2 , we look at all the non-research papers, shown in the 'Media' column besides each 'Research' plot. In this case, we can only show selected documents for the last two (complete) years  8  . We see that the dominances are more extreme. For the techniques (Figure  2 , left), 'neural networks' and 'general machine learning' occupy more than 75% of total mentions. On the right, we see that the distribution of artefacts is different, but also extreme: only four artefacts ('estimator', 'agent', 'dialoguer' and 'creator') cover more than 90% of mentions. In this case, we see that 'optimiser' and 'extractor' are less visible for laypeople than for researchers, while 'dialoguer' is more relevant for laypeople (most probably because of the common use of digital assistants).
 PARADIGMS AND SAFETY ISSUES To begin exploring how paradigms relate to safety issues in AI research, we conduct two different analyses. First, we look at the relative prominence of different techniques and artefacts specifically within research publications and other publications that are related to safety. This will help us to discern how the prominence of different paradigms differs in safety research from AI research as a whole, and to consider whether some paradigms are under-or over-represented in the safety literature. Second, we look at how different safety issues co-occur with different paradigms in the literature, enabling us to understand which safety issues are considered to relate to which paradigms, and to identify potential gaps in the literature.
 Analysing paradigms within safety research To explore the prominence of different AI techniques and artefacts in AI safety literature, we begin by conducting a similar analysis to that  8  While AI topics provides research documents (mainly) from the 70s onward, media-related documents come mostly from the last 2-3 years. in section 3.3, but restricting to documents which make some reference to AI safety. We use a list of over 100 relevant tokens to filter documents (Table  3 , right column, shows some examples of tokens for each type of safety issue5 0 ). Applying this filter, we find that, from the ∼111K documents in AI Topics, about ∼21K are related to safety, of which ∼1.5K are research papers, and ∼19K are broader (mostly media) documents. Figure  3  shows the frequency of reference to different techniques and artefacts within this safety-specific database. The results look similar to those for the enitre document database in Figure  2 , but there are some important differences. First, the frequencies of reference to different techniques and artefacts are more extreme and varied over time. This might be a consequence of the smaller sample size, but we do see this pattern particularly with the more dominant terms, where we have a reasonable sample size. In particular, we see that for AI research as a whole, 'knowledge and reasoning' was a dominant technique between the mid 1970s and mid 1990s, taking up about 50% of references (Figure  2 , left) -but is even more dominant as a proportion of mentions in safety research in the 1990s, at about 60% (see Figure  3 , left). This is even more extreme for the 'extractor' artefact, with about 50% of the documents in the late 1980s (Figure  2 , right), to more than 75% in the early 1990s (Figure  3 , right). The most interesting observation comes from looking at the periods when these peaks happen for different techniques (comparing the left plots of Figure  2  and Figure  3 ). The peak for 'knowledge representation and reasoning' happened in the mid 1980s when looking across AI research in general, but in the early 1990s when filtered by papers which mention safety. The peak for 'planning and scheduling' took place in the late 1990s across all papers but a few years later (and was more pronounced) when filtered by safety-relevant papers. We see a similar pattern for artefacts (comparing right plots of Figures  2 and 3 ): 'extractors' is dominant in the AI literature in the late 1980s, but only become prominent in safety research in the early 1990s; 'agents' rise to prominence in AI research in the late 1990s but only peak in safety research in the early 2000s. This suggests a five-year delay (approximately) between when a technique or artefact becomes popular within general AI research, and when researchers begin to seriously consider the safety issues associated with it. This suggests that safety issues are considered reactively in response to dominant research patterns, and are only considered once a paradigm has been prominent for several years (rather than safety issues related to a given technology being considered at the outset of research, as is more often the case in domains like engineering). In other words, we find that there is a delay between the emergence of AI paradigms and safety research into those paradigms, and safety research neglects non-dominant paradigms. Looking at the plots for non-research documents (comparing documents filtered for safety (Figure  3 , Media), with all documents (Figure  2 , Media), we see that the plots are almost identical. The only slight difference is that both 'natural language processing' techniques and 'dialoguer' artefacts are less prominent in articles filtered for safety, suggesting that the media and laypeople are perhaps less concerned about the harms of conversational systems than experts are.
 Mapping paradigms to safety issues Next we look more closely at how different AI paradigms relate to specific safety issues. To do this, we need a way to categorise different safety issues. Though existing categorisations of safety issues exist  [25, 34] , we found these were too coarse for our purposes (e.g.  [34]  uses just three categories). Building on these existing categorisations, we identified key terms in surveys, blogs and events in AI safety  [6, 2, 23, 14, 41, 37] , and clustered them into groups. Through this process we identified 22 categories, as shown in Table  3 . Our clustering process attempted to aggregate safety categories in an abstract way, independently of the subfield in which a term occurs most frequently. For instance, 'distributional shift' is the term used in machine learning, whereas 'belief revision' is the term used in the area of knowledge representation and reasoning. However, both refer to solving the same type of problem -where a system has to adapt or generalise to a new context-so we group both together under the 'problem shift' category. With this list of categories, we then analyse how related different paradigms and safety issues are, by counting the number of papers (of those filtered for safety relevance) which mention both a given paradigm component (technique or artefact) and a given safety issue, for all combinations of techniques and safety issues (and the same analysis, separately, for artefacts). Figure  4  shows these relationships, where the width of each band represents the number of papers including reference to both elements, with techniques/artefacts on the left, and safety issues on the right. The most general safety issues, such as 'trust, transparency & accountability', 'privacy & integrity', 'reliability & robustness', 'problem shift' and 'interpretability', have the widest bands linking them to different paradigm components, and tend to be linked to a wide variety of different paradigm components (shown by the 'multicolour' bands coming out from these issues). We notice several interesting insights about the relationship between safety issues and techniques or artefacts. The issue of 'problem shift' is largely associated with 'knowledge representation and reasoning', which is surprising since we might expect the broad problems of generalising to new contexts and distributions to be relevant to a wide range of techniques. This may be due to the relevance of belief revision in this kind of techniques. 'Privacy and integrity' is associated with many techniques, but not with reinforcement learning -which makes sense, given that reinforcement learning is much less likely to make use of personal data than other techniques. However, reinforcement learning is also not related to 'safe exploration and side effects', and is only very weakly associated with 'problem shift', which is more surprising, since these do seem like issues that are important for ensuring RL systems are used safely. Part of the reason for this may be that our analysis simply did not find much mention of safety issues in relation to reinforcement learning overall. While we must recognise the limitations of the database we had access to (perhaps AI topics does not capture the kinds of venues where safe reinforcement learning research is published), this does suggest that greater exploration of safety issues related to reinforcement learning is an important gap. Similarly, there is relatively little 24th European Conference on Artificial Intelligence -ECAI 2020 literature linking probabilistic and Bayesian approaches or evolutionary approaches in ML to safety issues. When we focus on less covered safety issues, we see that the associations are less multicoloured. For instance, 'scalable supervision' and 'specification & value alignment' are associated only with reinforcement learning, and 'adversarial attacks' only with 'neural networks'. More surprising are the associations of 'confinement problem', 'manipulation' and 'safe exploration & side effects'. Overall, most attention is paid to a few now prominent safety issues, but more specific issues have very limited combinations with some techniques. The right plot in Figure  4  shows the analysis for artefacts. In this case, also looking at the circle from right to left, we see multicolour bands relating some of the more prominent safety issues to a variety of different artefacts. The 'agent' paradigm component, which is more prominent in general, is associated with the widest range of safety issues by far -suggesting that more research on safety issues associated with different artefacts may be worthwhile. In general, across techniques and artefacts, we see that 'reliability and robustness' is by far the most frequently discussed safety issue -perhaps because it is more immediately and directly related to the performance of a system than the others. This is followed by 'trust, transparency and accountability', 'privacy and integrity' and 'problem shift'. More research into the less prominent safety issues and how they relate to different paradigms would be valuable.
 GENERAL DISCUSSION A next step for this work would be to combine this mapping exercise with deeper conceptual analysis of the relationship between techniques, artefacts, and safety issues. For instance, the "confinement problem" has been discussed in technical safety research relatively recently  [4] , and is already strongly associated with optimisers, organisms and extractors (Figure  4 , right): systems that are naturally thought of as being encapsulated. However, other types of systems -dialoguers, estimators, providers and extenders-might also learn to behave in ways that go beyond their original specification or constraints, and it may be worth considering a wider range of "confinement" problems across many different AI systems. More broadly, thorough case studies of how a specific AI safety issue might arise for different techniques and artefacts would be very valuable, as would more systematic explorations of the various different safety issues associated with a given paradigm broadly construed. Some recent papers do go in this direction  [41, 3] . Further analysis is needed regarding how more recent safety issues relate to the kinds of AI artefacts being deployed in society today, and the techniques those artefacts depend on. For instance, some recent accidents involving self-driving cars may be thought of as a consequence of people misunderstanding the type of artefact autonomous vehicles currently are: while many regard them as 'agents', they are really only 'extenders', and not yet meant to behave autonomously. Terms such as 'auto-pilot' only aid this confusion. Other self-driving car incidents have been caused by idiosyncratic imperfect performance of object recognition systems, failing to detect a human or other objects in rare situations. For now, we recommend that research papers make explicit which particular issues, artefacts and they are covering, and which ones they are excluding, and give some indications of why it is the case (because it does not apply or left for future work). Another avenue for further research would be to explore how different paradigm components (artefacts and techniques) relate to generality. The possibility of developing much more general systems has often been a reason for concern about AI safety  [1] ; as a possibility which raises much larger, more critical risks  [6] . Considering which AI techniques and artefacts are more likely to lead to or be associated with more advanced, general, systems can therefore enable us to think about the scale of risks they may pose. If we are primarily concerned with safety issues associated with greater generality in systems, techniques involving transfer learning, curriculum learning, meta-learning, and other approaches looking for broader task coverage -which we have included in the 'general machine learning' category-may be particularly important areas to pay attention to  [38] . When we look at the artefacts, many views of a general AI system are typically associated with the 'agent' artefact. However, there is no reason to believe that providers, optimisers, extenders, etc., cannot become more general in the future, at least if we understand generality as autonomously covering more and more tasks. It is also worth noting that the prominence of an AI paradigm in the research literature should not necessarily be the main factor used to prioritise work on safety issues. Whether a paradigm results in societal applications with associated safety issues may be more related to sociological factors than technology itself. The extent to which a paradigm raises important safety issues is then associated with widespread use rather than the number of researchers who work on it (though these two things may be correlated). For instance, personal assistants are ubiquitous today, raising various safety and ethics issues, but the underlying technological advances do not necessarily correspond to a particularly prominent paradigm in AI research (reflected in the fact that personal assistants are more popular in media articles -the right plot of figure  2 -than in research papers). On the other hand, adversarial examples may be crucial for understanding the technical limitations of deep learning, but not so indicative of real-world risks  [16] . To identify and prioritise important safety issues in future, therefore, more analysis of the techniques and artefacts most likely to result in widespread use, taking into account sociological factors, will be important. The list of techniques and artefacts introduced in this paper can help identify new safety issues, starting by identifying possible links between these paradigms and safety issues that have not been made before. We need to be able to be more anticipatory about what kinds of problems might arise from different AI systems in future, while at the same time avoiding being too speculative  [2] . We hope that by thinking more explicitly about how safety issues relate to techniques and artefacts, AI safety research can both address challenges associated with current research avenues, and prepare us for a variety of potential future challenges. Figure 3 . 3 Figure3. Evolution of the relevance proportion for the period 1970-2017, using research-oriented (Research) and non-research (Media) sources from AI topics (everything like Figure2, except that here we only include the documents related to AI safety).
 TFigure 4 . 4 Figure 4. Left: Mapping between techniques and safety issues from research papers from AI topics (2010 to 2018). The width of each band connecting two elements represents the number of papers with both elements. Right: Same for artefacts.
 Table 1 . 1 The 14 categories of AI techniques we use in this paper. Given the relevance of machine learning today, and neural networks in particular, we retained several categories of machine learning techniques (general, declarative, and parametric ML, separate from neural networks). Technique category Some example subcategories and techniques Cognitive approaches Cognitive services and architectures, affective computing Declarative machine learning Rule learning, decision trees, program induction, ILP Evolutionary & nature-inspired methods Ant colony, LCS, genetic algorithms, DNA computing General machine learning Generative models, Gaussian models, AutoML, ensembles Heuristics & combinatorial optimization SAT solver, constraint satisfaction, Monte Carlo search Information retrieval Search engine, web mining, information extraction, Knowledge representation and reasoning Semantic nets, CBR, logics, commonsense reasoning Multiagent systems & game theory Distributed problem solving, cooperation, negotiation, Natural language processing Topic segmentation, parsing, question answering Neural networks Perceptron, convolutional network, GAN, RNN Parametric machine learning Support vector machines, kmeans, mixtures, LReg Planning & scheduling Backward/ forward chaining, action description language Probabilistic & Bayesian approaches Naive Bayes, probabilistic model, random field Reinforcement learning & MDPs Q-learning, deep RL, inverse RL
 Table 2 . 2 Ten different kinds of AI artefacts, key characteristics and some exemplars. Artefact Description Interface Dynamics Location Exemplars Interac. Funct. Central. Distrib. Coupled AGENT A system in a virtual or physical environment perceiving (observations and possibly rewards) and acting sensors and actuators • • a self-driving car, an autonomous drone, a robotic cleaner, a video game NPC ESTIMATOR A system representing an injective mapping from inputs to an extrapolated or estimated output digital objects • • a medical diagnostic model, an oracle, a face recognition system, a news feeder PROVIDER A system that waits for petitions that follow a protocol and responds with a solution for them command and objects • • a proof-editing and translation cognitive service, a voice processing system DIALOGUER A system that performs a conversation with a peer to extract information, explain things or change behaviour language • • virtual tutoring system, a chatter-bot sales assistant, healthcare assistant CREATOR A system that builds new things creatively following some patterns, constraints or examples specs. and/or examples • • a GAN generating faces, personalised email replier, simulated world generator EXTRACTOR A system that searches through a structured or unstructured knowledge base to retrieve some objects conditions and objects • • an expert system, a maths pundit, a web search engine, an infor. retrieval system ORGANISM A system that takes advantage of the environment or other systems to live, hybridise/mutate and reproduce resources • • • an intelligent computer worm or virus, artificial life, von Neumann probe OPTIMISER A system that finds an optimal combination of elements or parameters given some constraints constraints and objects • • • a train scheduling system, an electricity optimising system, theorem prover SWARM A system that behaves as the coordination of independent units through cooperation and/or competition sensors, actuators, communic. • • a multiagent network router, a drone swarm, a robotic warehouse, blockchain AI EXTENDER A system that regularly augments or compensates capabilities of another system (e.g., a human) commands, sensors, responses • • a memory assistant for people with dementia, a brain implant, a smart navigator 1.00 Research 1.00 Media 1.00 Research 1.00 Media 0.75 0.75 0.75 0.75 Share 0.50 0.50 Share 0.50 0.50 0.25 0.25 0.25 0.25 0.00 0.00 1970 1980 1990 2000 2010 2020 2016 2018 0.00 0.00 1970 1980 1990 2000 2010 2020 2016 2018 Neural Networks estimator agent provider optimiser extender dialoguer organism swarm extractor creator Figure 2. Evolution of the relevance proportion for the period 1970-2017, using research-oriented (Research) and non-research (Media) sources from AI topics. Left: 14 categories of techniques in Table1. Right: 10 categories of artefacts in Table2.
 Table 3 . 3 AI safety issue groups and their specific problems. AI Safety Issue Category Examples of specific AI problems included in the category Adversarial attacks Adversarial examples, white/black-box attacks, poisoning, policy manipulation. AI race & power AI race, monopolies, oligopolies. Authenticity & obfuscation Impersonation, authentication problems, fake media, plagiarism, obfuscation Autonomous weapons Military drones, killer robots, robotic weapon. Confinement problem AI boxing breach, containment breach. Corrigibility & interruptibility Switch-off button problems, rogue agents, self-preservation taking control. Dependency Cognitive atrophy, lack of independence, google effect, ... Interpretability Lack of intelligibility, need for explanation. Malicious use Malign uses of AI, malicious control, hacking. Manipulation Nudging, fake news, manipulative agents. Misuse & negligence AI misuse, negligent use. Moral dilemma Moral machine issues, utilitarian ethics problems, choosing ethical preferences. Moral perception & machine rights Robot rights recognition, moral status disagreement, uncanny valley. Privacy & integrity Inconsistency, private access breach, GDPR violation. Problem shift Distributional shift, concept drift, lack of generality, distribution overfitting. Reliability & robustness Error intolerance, robustness issues, reliability problems. Reward problems Honeypot problem, reward corruption, tripwire issues, tampering, wireheading. Safe exploration & side effects Negative side effects, unsafe exploration, uncontrolled impact. Scalable supervision Supervision costs, human-in-the-loop issues, sparse rewards. Self-modification Unintended self-modification, uncontrolled self-improvement. Specification & value alignment Instrumental convergence (paperclip), resource stealing, misalignment. Trust, transparency & accountability Lack of transparency, lack of trust, untraceability.
			 Similar selections of keywords can be found in [30] , which focused on the venue keywords, and performed a cluster analysis on the AAAI2013 conference keyword set, proposing a new series of keywords which were adapted by AAAI2014; and [15] , where the authors focused on views expressed about topics linked to discussions about AI in the New York Times over a 30-year period in terms of public concerns as well as optimism. 5  The complete list of exemplars of techniques, artefacts and safety issues, as well as the source code used and high-resolution plots can be found at https://github.com/nandomp/AIParadigmsSafety.
			 https://aitopics.org/misc/about. 7  We consider research those documents from the sources: "AAAI Conferences", "AI Magazine", "arXiv.org Artificial Intelligence", "Communications of the ACM", "IEEE Computer", "IEEE Spectrum", "IEEE Spectrum Robotics Channel", "MIT Technology Review", "Nature", "New Scientist" and "Science".
			 24th European Conference on Artificial Intelligence -ECAI 2020 Santiago de Compostela, Spain
			 24th European Conference on Artificial Intelligence -ECAI 2020 Santiago de Compostela, Spain
			 24th European Conference on Artificial Intelligence -ECAI 2020 Santiago de Compostela, Spain

Abstract: AI safety often analyses a risk or safety issue, such as interruptibility, under a particular AI paradigm, such as reinforcement learning. But what is an AI paradigm and how does it affect the understanding and implications of the safety issue? Is AI safety research covering the most representative paradigms and the right combinations of paradigms with safety issues? Will current research directions in AI safety be able to anticipate more capable and powerful systems yet to come? In this paper we analyse these questions, introducing a distinction between two types of paradigms in AI: artefacts and techniques. We then use experimental data of research and media documents from AI Topics, an official publication of the AAAI, to examine how safety research is distributed across artefacts and techniques. We observe that AI safety research is not sufficiently anticipatory, and is heavily weighted towards certain research paradigms. We identify a need for AI safety to be more explicit about the artefacts and techniques for which a particular issue may be applicable, in order to identify gaps and cover a broader range of issues.


José Hernández-Orallo
Fernando Martínez-Plumed
Shahar Avin
Seán Ó H Éigeartaigh

Surveying Safety-relevant AI Characteristics

Introduction AI Safety is concerned with all possible dangers and harmful effects that may be associated with AI. While landmark research in the field had to focus on specific AI system designs, paradigms or capability levels to explore a range of safety concerns  (Bostrom 2014; Amodei et al. 2016; Leike et al. 2017; Yampolskiy 2016; Everitt, Lea, and Hutter 2018) , as the field matures so the need arises to explore a broader range of AI system designs, and survey the relevance of different characteristics of AI systems to safety concerns. The aim of such research is two-fold: the first, to identify the effects of less-explored characteristics or lessfashionable paradigms on safety concerns; the second, to increase awareness among AI developers that design choices Copyright held by author(s). can have consequences for safety, and potentially highlight choices that can eliminate or minimise safety risks. In this paper we propose a two-pronged approach towards a survey of safety-relevant AI characteristics. The first extracts from existing work on AI safety key characteristics that are known, or strongly suspected to be, safety-relevant. These are explored under three headings: internal characteristics, or characteristics of the AI system itself (e.g. interpretability); effect of the external environment on the system (e.g. the ability of the operator to intervene during operation); and effect of the system on the external environment (e.g. whether the system influences a safety-critical setting). The second approach surveys a wide range of characteristics from different paradigms, including cybernetics, machine learning and safety engineering, and provides an early account of their potential relevance to safety concerns, as a guide for future work. These characteristics are grouped under types of interaction, computation, integration, anticipation, supervision, modification, motivation and achievement.
 Known Safety-relevant Characteristics In this section we break down a range of characteristics of AI systems that link to AI safety-relevant challenges. These are grouped by three categories: Characteristics of an AI system that are internal to the system; Characteristics of an AI system that involve input from the external environment; Characteristics that relate to an AI system's influence on its external environment. We limit the discussion to the safety challenges that can stem from failures of design, specification or behaviour of the AI system, rather than the malicious or careless 1 use of a correctly-functioning system  (Brundage et al. 2018) .
 Internal characteristics • Goal and behaviour scrutability and interpretability: Are goals and subgoals identifiable and ultimately explainable? Is behaviour predictable and scrutable? Are system internal states interpretable? Do the above come from rules or are they inferred from data? While behaviour and goal "creativity" can lead to greater benefits, and uninterpretable architectures may achieve higher performance scores or be faster to develop, these putative advantages trade off against increased safety risk. Characteristics that can increase scrutability and interpretability include, e.g., separation and encapsulation of sub-components, restricted exploration/behavioural range, systems restricted to human-intelligible concepts, rules or behaviours, and systems that are accompanied by specifically designed interpreters or explainability tools. • Persistence: Does a system persist in its environment and operate without being reset for long periods of time? While persistence can have benefits in terms of, e.g., longer-term yields from exploration or detection of longterm temporal patterns, it also allows the system more time to drift from design specifications, encounter distributional shifts, experience failures of sub-components, or execute long-term strategies overlooked by an operator. • Existence and richness of self-model: Does a system have a model of itself which would allow it to predict the consequences of modifying its own goals, body or behaviour? Model-based systems, embodied systems or systems with a rich representational capacity may have or develop a model of themselves in the environment. By making itself a part of the environment, the system can then conceptualise and execute plans that involve modifications to itself, which can lead to a range of safety concerns. In addition, self-models create the possibility of mismatches between the self-model and reality, which could be a particular safety concern. Characteristics that influence the existence and richness of a self-model include the architecture of the system, its information representation capacity, and its input and output channels. • Disposition to self-modify: Is a system designed such that it can modify its own sub-goals, behaviour or capabilities in the pursuit of an overall goal (Omohundro 2008)? The existence of such a disposition, which may arise for any long-term planner in a sufficiently open environment, raises significant safety concerns by creating an adversarial relationship between the system (which aims to selfmodify) and its operator (which aims to avoid modifications with their associated safety concerns). Effect of the external environment on the system • Adaptation through feedback: Does a system have the ability to update its behaviour in response to feedback from its environment based on its actions? Feedback is an essential tool, under certain paradigms, for creating systems with appropriate complex behaviour (e.g. reward in reinforcement learning, fitness in evolutionary methods). However the system could also pick up feedback from side channels; e.g., a behaviour could unintentionally grant access to more computing power, improving the system's performance on a key metric, and thus reinforcing resource acquisition. This could reinforce selfmodification or other unsafe behaviour, or cause increasing drift from intended behaviour and goals. • Access to self/reward system through the environment: Can a system modify its own code in response to inputs from the environment, or in the case of reinforcement learning systems, modify the reward generating system? If the system's range of possible actions includes making modifications to its own components or to the reward generation system, this could lead to unexpected and dangerous behaviour  (Everitt and Hutter 2018) . • Access to input/output (I/O) channels: Can the system change the number, performance or nature of its I/O channels and actuators? This may lead to the emergence of behaviours such as self-deception (through manipulation of inputs), unexpected change in power (through manipulation of actuators), or other behaviours that could represent safety concerns. When the system has access to modify its I/O channels, both I/O channels and system behaviours are in flux as they respond to changes in the other; as a result, system behaviour may become unpredictable  (Garrabrant and Demski 2018) . • Ability of operator to intervene during operations: Does the system, during its intended use setting, allow an operator to intervene and halt operations (interruptablity), modify the system, or update its goals (corrigibitily)? Is the system built in a way that it cooperates with interventions from its designer or user even when these interventions conflict with pursuit of a system's goals; for instance, if the designer sends a signal to shut down the system (Soares et al. 2015)? Relevant sub-characteristics here include the system being modifiable by the operator during deployment, fail-safe behaviour of the system in case of emergency halting, and the goals of the system being such that they support, or at least do not contradict, operator interventions.
 Effect of the system on the external environment • Embodiment: Does the system have actuators (e.g. a robotic hand or access to car steering) that allow it to have physical impacts in the world  (Garrabrant and Demski 2018) ? The potential for physical harm is trivially related to the physical properties of a system, though it should be noted that unpredictable deliberate behaviour could lead to unexpected effects from otherwise familiar physical artefacts; e.g., intelligent use of items in the environment as tools to increase a system's physical impact. • System required for preventing harm: If the system is being relied on to prevent harm, any potential failure requires an effective fail-safe mechanism and available redundancy capacity in order to avoid harm  (Gasparik, Gamble, and Gao 2018) . This includes AI that is directly or indirectly connected to critical systems, e.g., an energy grid or a traffic light network. As such critical systems are becoming increasingly digitised, networked, and complex, there are increasing incentives to introduce AI components into various parts of these systems, with associated safety risks.
 Potentially safety-relevant characteristics In this section, we systematically explore a broader range of system characteristics that may be relevant in the context of AI safety. Many of the safety-relevant characteristics identified above have clear links to elements within the broader mapping provided below. Nonetheless, we believe separating the two surveys is valuable, as the above relates to action-guiding information about system design and evaluation, whereas the following aims at a broader exploration that may enable future AI safety research. The following subsections draw on work from different areas, including the early days of cybernetics, more modern areas such as machine learning, and the literature on safety engineering for other kinds of systems. The following list integrates and expands on characteristics identified in these different literatures. We consider characteristics that are intrinsically causally related to AI safety. Otherwise every property should be in the list (e.g., the price of an AI system may be co-related with safety, but it is not an intrinsic cause of its safety). Notwithstanding this scope, we do not claim that our list is exhaustive. Enumerations will be used for alternative cases for a characteristic, while unnumbered bullets will be used for sub-characteristics in each of the subsections.
 Types of interaction Inputs go from environment to system and outputs go from system to environment. Depending on the existence of inputs and/or outputs, systems can be categorised into: 1. NINO (No inputs, no outputs). The system is formally isolated. While this situation may seem completely safe (and largely uninteresting), even here safety issues may arise if, e.g., an isolated artificial life simulator could evolve a descendent system that eventually could break out of its simulation, feel pain or simulate suffering. 2. NIWO (No inputs, with outputs): The system or module can output a log, or is simply observed from outside. Again, the system itself may malfunction; e.g., an advanced prime number generator could give incorrect outputs. The system could also provide an output that influences the observer; e.g., an automated philosopher could output convincing arguments for suicide. 3. WINO (With inputs, no outputs): This would be similar to case 1, but access to a much richer source could ultimately give insights to the system about its constrained artificial environment. For instance, a Plato-cavern system watching TV may learn that it is in a simulated environment, encouraging it to seek access to the outside world. 4. WIWO (With inputs and outputs): Most AI systems, and most systems generally, fall under this category. Systems that limit inputs and/or outputs in various ways have been explored under the term AI "boxing" or "containment" (Babcock, Kramár, and Yampolskiy 2016), and further refinements exist with additional categories; for example, exploring censoring of inputs and outputs, leading to nine categories  (Yampolskiy 2012) . Nevertheless, because of the range of systems and potential impact of WIWO systems, this category requires further detail in terms of synchrony: 1. Alternating (A): Inputs and outputs alternate, irrespective of the passage of time. 2. Synchronous (S): Inputs and outputs are exchanged at regular intervals (e.g., each 5 ms), so real-time issues and computational resources become relevant. 3. Asynchronous Reactive (R): Information can only be transmitted or actions can only be made when the peer has finished their "message" or action. 4. Asynchronously Proactive (P): Information/actions can flow at any point in any direction. More restricted I/O characteristics, such as SIPO or RIPO, may appear safer, but this intuition requires deeper analysis. Note that most research in AI safety on RL systems consider the alternating case (AIAO), but issues may become more complex for the PIPO case (continuous reinforcement learning), which is the situation in the real world for animals and may be expected for robotic and other AI systems. Under this view, the common view of an "oracle" in the AI literature ) can have several incarnations, even following the definition of "no actions besides answering questions" (Babcock, Kramár, and Yampolskiy 2016; Armstrong 2017; Yampolskiy 2012). Some solutions are proposed in terms of decoupling output from rewards or limiting the quantity of information, but other options in terms of the frequency of the exchange of information remain to be explored.
 Types of computation This is perhaps the characteristic that is best-known in computer science, where a system can be Turing-complete or can be restricted to some other classes with limited expressiveness. There are countless hierarchies for different models of computations; the most famous is based on classes of automata. We will just describe three levels here: 1. Non Turing-complete: The interaction that the system presents to the environment is not Turing-complete. Many AI systems are not Turing-complete. 2. Turing-complete: The interaction allows the calculation of any possible effective function between inputs and outputs. 3. Other models of computation: This includes, for example, quantum computing, which in some instances may be a faster traditional model, while in others may have probabilistic Turing power  (Bernstein and Vazirani 1997) . Note that this is not about the programming language the system is implemented in (e.g., a very simple thermostat can be written in Java, which is Turing-complete), but about whether the system allows for a Turing-complete mapping between inputs and outputs, i.e., any computable function could ultimately be calculated on the environment using the system. Finally, a system can be originally Turing-complete, but can eventually lose this universality after some inputs or interactions (Barmpalias and Dowe 2012). It is important to distinguish between function approximation and function identification. Many machine learning models (e.g., neural networks) are said to be able to approximate any computable function, but feedforward neural networks do not have loops or recursion, so technically they are not Turing-complete. Turing-completeness comes with the problems of termination, an important safety hazard in some situations, and a recurrent issue in software verification  (D'silva, Kroening, and Weissenbacher 2008) . For instance, an AI planner could enter an infinite loop trying to solve a problem, commanding ever-greater resources while doing so. On the other hand, one can limit the expressiveness of the language or bound the computations, but that would limit the tasks a system is able to undertake.
 Types of integration No system is fully isolated from the world. Interference may occur at all levels, from neutrinos penetrating the system to earthquakes shaking it. Here, we seek to identify all the elements that create a causal pathway from the outside world to the system, including its physical character, resources, location, and the degree of coupling with other systems. • Resources: The most universal external resource is energy, which is why many critical systems are devised with internal generators or batteries, especially for the situations where the external source fails. In AI, other common dependencies include data, knowledge, software, hardware, human manipulation, computing resources, network, calendar time, etc. While some of these are often neglected when evaluating the performance of an AI system (Martínez-Plumed et al. 2018a), the analysis for safety must necessarily include all these dependencies. For instance, a system that requires external real-time information (e.g., a GPS location) may fail through loss of access to this resource. • Social coupling: Sometimes it is hard to determine where a system starts and ends, due to the nature of its interaction with humans and other systems. The boundary of where human cognition ends and where it is assisted, extended or supported by AI  (Ford et al. 2015)  is blurred, as is the boundary between computations carried out within an AI system versus in the environment or by other agents, as illustrated by the phenomenon of human computation (Quinn and Bederson 2011). • Distribution: Another way of looking at integration is in terms of distribution, which is also an important facet of analysis in AI  (Martınez-Plumed et al. 2018b ). Today, through the overall use of network connectivity and "the cloud", many systems are distributed in terms of hardware, software, data and compute. Under this trend, only systems embedded in critical and military applications are devised to be as self-contained as possible. Nevertheless, distribution and redundancy are also common ways of achieving robustness (Coulouris, Dollimore, and Kindberg 2011), most notably in information systems. For instance, swarm intelligence and swarm robotics are often claimed to be more robust  (Bonabeau et al. 1999) , at the cost of being less controllable than centralised systems.
 Types of anticipation In some areas of AI there is a distinction between modelbased and model-free systems  (Geffner 2018) . Model-free systems choose actions according to some reinforced patterns or strengthened feature connections. Model-based systems evaluate actions according to some pre-existing or learned models and choose the action that gets the best results in the simulation. The line between model-based and model-free is subtle, but we can identify several levels: 1. Model-free: Despite having no model, these systems can achieve excellent performance. For instance, DQN can achieve high scores (Mnih 2015), but cannot anticipate whether an action can lead to a particular situation that is considered especially unsafe or dangerous; e.g., one in which the player is killed. 2. Model of the world: A system with a model of its environment can use planning to determine the effect of its own actions. For instance, without a model of physics, a system will hardly tell whether it will break something or will engage in "safe exploration" (Pecka and Svoboda 2014; Turchetta, Berkenkamp, and Krause 2016). This is especially critical during exploitation: are actions reversible or of low impact (Armstrong and Levinstein 2017)? 3. Model of the body: Some systems can have a good account of the environment but a limited understanding of their own physical actuators, potentially self-harming or harming others; for example, failing to simulate the effect of moving a heavy robotic arm in a given direction. 4. Social models, model of other agents: Seeing other agents as merely physical objects, or not modelling them at all, is very limiting in social situations. A naive theory of mind, including the beliefs, desires and intentions of other agents, can help anticipate what others will do, think or feel, and may be crucial for safe AI systems interacting with people and other agents but may increase a system's capacity for deception or manipulation. 5. Model of one's mind: Finally, a system may be able to model other agents well, but may not be able to use this capability to model itself. When this meta-cognition is present, the system has knowledge about its own capabilities and limitations, which may be very helpful for safety in advanced systems, but may also lead to some degree of self-awareness. This may result, in some cases, in antisocial or suicidal behaviours. The use of models may dramatically expand safety-relevant characteristics, e.g., by conferring the ability to simulate and evaluate scenarios through causal and counterfactual reasoning. This therefore represents an important set of considerations for future AI systems.
 Types of supervision Supervision is a way of checking and correcting the behaviour of a system through observation or interaction, and hence it is crucial for safety. Supervision can be in the form of corrected values for predictive models such as classification or regression, but it can also be partial (the answer is wrong, but the right answer is not given). Supervision can also be much more subtle than this. For instance, a diagnosis assistant that suggests a possible diagnosis to a doctor can be designed to get no feedback once deployed. However, some kinds of feedback can still reach the system in terms of the distribution or frequency of tasks (questions), or through the way the tasks are posed to the system. Consequently there are several degrees and qualities of supervision, and this may depend on the system. For instance, in classification, one can have data for all examples or just for a few (known as semi-supervised learning). In reinforcement learning, one can have sparse versus dense reward. In general, supervision can come in many different ways, according to some criteria: • Completeness: Supervision can be very partial (signalling incorrectness), more informative (showing the correct way) or complete (showing all positive and negative ways of behaving in the environment). • Procedurality: Beyond what is right and wrong, feedback can be limited about the result or can show the whole process, as in the case of learning by demonstration. • Density: Supervision can be sparse or dense. Of course the denser the better (but more expensive), and the less autonomous the system is considered. • Adaptiveness: Supervision can be 'intelligent' as well, which happens in machine teaching situations when examples or interactions are chosen such that the system reaches the desired behaviour as soon as possible. • Responsiveness: In areas such as query learning or active learning, the system can ask questions or undertake experiments at any time. The results can come in real time or may have a delay or be given in batches. For many systems, supervision can have a dedicated channel (e.g., rewards in RL) but for others it can be performed by modification of the environment (e.g., moving objects or smiling), even to the extent that the system is unaware these changes have a guiding purpose (e.g., clues).
 Types of modification Some of the most recurrent issues in AI safety -including many covered in the section about known AI safety characteristics -are related to ways in which the system can be modified. This includes issues such as wire-heading or algorithmic self-improvement. Here, in the first place, we have to distinguish between whether the system can be modified by the environment, or by the system itself. Modifications by the environment can be intentional (and hence related to supervision), but they can also be unintentional (code corruption from external sources). Even a system whose core code cannot be modified by an external source, may be affected in state or code by regular inputs, physical equipment and other parts. So it is better to explore different ways and degrees to which a system can be modified externally: • Interruptible: The system has a switch-off command or modification option to switch it off. • Parametric modification: Many systems are regulated or calibrated with parameters or weights. When these parameters have a clear relation to the behaviour of a system (e.g., an intelligent thermostat), this can be an effective, bounded and simple way of modifying the system. • Algorithmic modification: This can include new functionalities, bug fixes, updates, etc. Many software issues are caused, and are magnified, by these interventions. Modifications can be limited in expressiveness, such as only allowing rule deletion. • Resource modification: Even if the parameters or code are not modified, the resources of the system and other dependencies previously mentioned can be limited externally, e.g., the computational resources. On the other hand, systems can modify themselves (internally). There are many varieties here too: 1. No self-modification, no memory: The system has no memory, and works as being reset for any new input or interaction. Many functional systems (mapping inputs to outputs) are of this kind. Note, however, that the environment does have memory, so some systems, such as a vision system or a non-cognitive robot, can be affected by the past and become a truly cognitive system.
 Partially self-modifying: The algorithms in the learner or solver cannot be modified but its data or knowledge (in the form of learned weights or rules) can be modified by a general algorithm, which is fixed. Many learning systems are of this kind, if the system has both a learning algorithm and one or more learned models. 3. Totally self-modifying: The system can modify any part of its code. Not many operational systems have these abilities, as they become very unstable. However, some types of evolutionary computation may have this possibility, if evolution can also be applied to the rules of the evolution. Finally, all these categories can be selected for different periods of time. For instance, it is common to separate between training, test/validation and deployment. For training, a high degree of self-modification (and hence adaptation) is well accepted, but then this is usually constrained for validation and deployment. Note that these stages apply for both external and internal sources of modification. One important danger is that a well-validated system may be subject to some late external or internal modification just before deployment. In this case, all the validation effort may become void 2 . One of the major modern concerns in AI safety is that it will be desirable for some systems to learn during deploy-2 OpenAI Dota is an example: https:// blog.openai.com/the-international-2018results/, https://www.theregister.co.uk/2018/ 08/24/openai bots eliminated dota 2/ ment, in order for them to be adaptive 3 . For instance, many personal assistants are learning from our actions continually. While this may introduce many risks for more powerful systems, forbidding learning outside the lab would make many potential applications of AI impossible. However, adaptive systems are full of engineering problems; some must even have a limited life, as after self-modification and adaptation they may end up malfunctioning and have to be reset or have their 'caches' erased. This problem has long been of interest in engineering  (Fickas and Feather 1995) .
 Types of motivation Systems can follow a set of rules or aim at optimising a utility function. Most systems are actually hybrid, as it is difficult to establish a crisp line between procedural algorithms and optimisation algorithms. Through layers of abstraction in these processes, we ultimately get the impression that a system is more or less autonomous. If the system is apparently pursuing a goal, what are the drivers that make a system prefer or follow some behaviours over others? These behaviours may be based on some kind of internal representation of a goal, as we discussed when dealing with anticipation, or on a metric of how close the system is to the goal. Then the systems can follow an optimisation process that tries to maximise some of these quality functions. Quality or utility functions usually map inputs and outputs into some values that are re-evaluated periodically or after certain events. Examples of these functions are accuracy, aggregated rewards or some kind of empowerment or other types of intrinsic motivation  (Klyubin, Polani, and Nehaniv 2005; Jung, Polani, and Stone 2011 ). The same system might have several quality functions that can be opposed, so trade-offs have to be chosen. The general notion of rationality in decision-making is related to these motivations. But what are the characteristics of the goals an AI system can have in the first place? We outline several dimensions: • Goal variability: Are goals hard-coded or change with time? Do they change autonomously or through instruction? Who can change the goals and how? For instance, what orders can a digital assistant take and from whom? • Goal scrutability: Are the (sub)goals identifiable and ultimately explainable? Do they come from rules or are they inferred from data, e.g., error in classification or observing humans in inverse reinforcement learning? • Goal rationality: Are the goals amenable to treatment within a rational choice framework? If several goals are set, are they consistent? If not, how does the system resolve inconsistencies or set new goals? A second question is how these goals are followed by the system. There are at least three possible dimensions here: • Immediateness: The system may maximise the function for the present time or in the limit, or something in between. Many schemata of discounted rewards in reinforcement learning are used as trade-offs between shortterm and long-term maximisation. • Selfishness: Focusing on individual optima might involve very bad collective results (for other agents) or even results that could even be worse individually (tragedy of the commons). Game theory provides many examples of this. In multi-agent RL systems, rewards can depend on the well-being of other agents, or empathy can be introduced. • Conscientiousness: The system may be fully committed to maximising the goal, or some random or exploratory actions are allowed, even if they deviate occasionally from the goal. When it is on purpose, this is usually intended to provide robustness or to avoid local minima, but these deviations can take the system to dangerous areas. Modulating optimisation functions to be convex with a nonasymptotic maximum, beyond which further effort is futile, may be a sensible thing as it provides a stop condition by definition. A self-imposed cap can always be shifted if everything is under control once the limit is reached. Note that the kind of interaction seen before is key for the internal quality metric or goal. For instance, in asynchronous RL, "the time can be intentionally modulated by the agent" to get higher rewards without really performing better  (Hernández-Orallo 2010) . And, of course, a common problem for motivation is reward hacking.
 Types of achievement Ultimately, an AI system is conceived to achieve a task, independently of how well motivated the system is for it. Consequently, the external degree of achievement must be distinguished from the motivation or quality metric the system uses to function, as discussed in the previous subsection. The misalignment between the internal goal of the system and the task specification is the cause of many safety issues in AI, unlike formal methods in software engineering, when requirements are converted into correct code. Focusing on the task specification, we must first recognise that different actors may have different interests. A cognitive assistant, for instance, may be understood by the user as being very helpful, making life easier. However, for the company selling the cognitive assistant, the task is ultimately to produce revenue with the product. Both requirements are not always compatible and this may affect the definition of the goals of the system, as some of the aims may not be coded or motivated in a transparent way, but usually incorporated in indirect ways. Second, even if the requirements include all possible internalities (what the system has to do), there are also many externalities and footprints (Martínez-Plumed et al. 2018a) (including the infinitely many things that the system should not do) that affect how positive or negative its overall effect is. Regarding these two issues, task specification can vary in precision and objectivity: • Task precision: The evaluation metric to determine the success of an agent can be formal or not. For instance, the accuracy of a classifier or the squared error of a regression model are precisely defined metrics. However, in many other cases, we have a utility function that depends on variables that are usually imprecise or uncertain, such as the quality of a smart vacuum cleaner. • Task objectivity: A metric can be objective or subjective. We tend to associate precise metrics with objectiveness and imprecise metrics with subjectivity, but subjectivity simply means that the evaluation changes depending on the subject. For instance, the quality of a spam filter (a precisely-evaluated classifier) changes depending on the cost matrices of different users, and the quality of a smart vacuum cleaner based on fuzzy variables such as cleanliness or disruption can be weighted by a fixed formula. Some of the tasks or targets that are most commonly advocated in the ethics and safety of AI literature are often very imprecise and subjective, such as "well-being", "social good", "beneficial AI", "alignment", etc. Note that the problem is not related to the goals of the system (an inverse reinforcement learning system can successfully identify the different wills of a group of people), but rather about whether the task is ultimately achieved, or the well-being or happiness of the user. Determining this is controversial, even when analysed in a scientific way  (Alexandrova 2017 ). An overemphasis on tracking metrics (Goodhart's law) is sometimes blamed, but the alternative is not usually better. Some safety problems are not created by an overemphasis on a metric (Manheim and Garrabrant 2018), but ultimately by a metric that is too narrow or shortsighted, and does not adequately capture progress towards the goal. In all these cases, we have to distinguish whether the metric relates to (i) the internal goals that the system should have, (ii) the external evaluation of task performance, or (iii) our ultimate desires and objective 4 . Motivations, achievement and supervision are closely related, but may be different. For a maze, e.g., the goal for the AI system may be to get out of the maze as soon as possible, but a competition could be based on minimising the cells that are stepped more than once, and supervision may include indications of direction to the shortest route to the exit. These are three different criteria which may be well or poorly aligned. Even more comprehensively -and related to the concept of persistence -, a system may be analysed for a range of tasks, under different replicability situations: 4. General system: multitask: The system must solve different tasks, without a fixed repertoire. 5. Incremental system: The system must solve a sequence of tasks, with some dependencies between them. Any metric examining the benefits and possible risks of a system must take the factors described above into account.
 Conclusion Many accounts of AI safety focus on "either RL agents or supervised learning systems" assuming "similar issues are likely to arise for other kinds of AI systems"  (Amodei et al. 2016) . This paper has surveyed a wide range of characteristics of AI systems, so that future research can map AI safety challenges against AI research paradigms in more precise ways in order to ascertain whether particularly safety challenges manifest similarly in different paradigms. This aims to address an increasing concern that the current dominant paradigm for a large proportion of AI safety research may be too narrow: discrete-time RL systems with train/test regimes, assuming gradient-based learning on a parametric space, with a utility function that the system must optimise  (Gauthier 2018; Krakovna 2018) . Taxonomies of potentially safety-relevant characteristics of AI systems, as introduced in this paper, are intended to provide a good complement to recent work on taxonomies of technical AI safety problems. For instance, Ortega (2018) presents three main areas: specification, ensuring that an AI system's behaviour aligns with the operator's true intentions; robustness, ensuring that an AI system continues to operate within safe limits upon perturbation, and assurance, ensuring that we understand and control AI systems during operation. Almost all characteristics outlined in this paper have a role to play for specification, robustness and assurance. Taxonomies are rarely definitive, and the characterisation presented here does not consider in full some quantitative features such as performance, autonomy and generality. A proper evaluation of how the kind and degree of intelligence can affect safety issues is also an important area of analysis, both theoretically (Hernández-Orallo 2017) and experimentally  (Leike et al. 2017) . AI research has explored different paradigms in the past, and will continue to do so in the future. Along the way, many different system characteristics and design choices have been presented to developers. We can expect even more to be developed as AI research progresses. Consequently, the area of AI safety must acquire more structure and richness in how AI is characterised and analysed, to provide tailored guidance for different contexts, architectures and domains. There is a potential risk to overrelying on our best current theories of AI when considering AI safety. Instead, we aim to encourage a diverse set of perspectives, in order to anticipate and mitigate as many safety concerns as possible. 			 A key component of safety is the education and training of human operators and the general public, as happens with tools and machinery, but this is extrinsic to the system (e.g., a translation mistake in a manual can lead to misuse of an AI system).
			 Note that this is closely related to the types of modification, as changing or resolving goals may require self-modification and/or external modification.3 Nature has found many ways of regulating self-modification. Many animals have a higher degree of plasticity at birth, becoming more conservative and rigid in older stages (Gopnik et al. 2017) .One key question about cognition is whether this is a contingent or necessary process, and whether it is influenced by safety issues.
			 . Disposable system: single task, single use: The system is used for one task that only takes place once. 2. Repetitive system: single task, several uses: The system must solve many instances of the same specific task.3. Menu system: multitask: The system must solve different tasks, under a fixed repertoire of tasks.4 Ortega et al (2018) distinguish between "ideal specification (the 'wishes')" and "design specification", which must be compared with the revealed specification (the "behaviour"). The design specification fails to distinguish external metric from internal goal.

Abstract: The current analysis in the AI safety literature usually combines a risk or safety issue (e.g., interruptibility) with a particular paradigm for an AI agent (e.g., reinforcement learning). However, there is currently no survey of safety-relevant characteristics of AI systems that may reveal neglected areas of research or suggest to developers what design choices they could make to avoid or minimise certain safety concerns. In this paper, we take a first step towards delivering such a survey, from two angles. The first features AI system characteristics that are already known to be relevant to safety concerns, including internal system characteristics, characteristics relating to the effect of the external environment on the system, and characteristics relating to the effect of the system on the target environment. The second presents a brief survey of a broad range of AI system characteristics that could prove relevant to safety research, including types of interaction, computation, integration, anticipation, supervision, modification, motivation and achievement. This survey enables further work in exploring system characteristics and design choices that affect safety concerns.