Christian P Janssen
Linda Ng Boyle
Wendy Ju
Andreas Riener
Ignacio Alvarez

Agents, environments, scenarios: A framework for examining models and simulations of human-vehicle interaction

Introduction Within the transportation, automotive, and user interface research communities, there is occasional confusion as to what is implied by "simulation" or "model." For example, the following are all false assumptions: a model or simulation implies tight control with no testing in a naturalistic environment, a model always involves simulating people or the traffic environment, and a concrete, falsifiable research question cannot be achieved without a model or simulation and testing in the real world. These false assumptions overlook the fact that researchers and practitioners in these fields have various approaches to modeling the agents in the car, the driving environment, and the scenarios under consideration of study. The different approaches and associated labels can create confusion as to which methods are most effective to examine specific research questions regarding human-vehicle interaction. The authors of this paper have used different types of simulations in driving studies and other domains, in part due to their different backgrounds in psychology, artificial intelligence (AI), safety science, design, and engineering. During a meeting in 2016  (Riener et al., 2016a) , the authors identified thatup until thenthey meant different things when talking about "a simulation" or "a model" and that each author held some incorrect assumptions about these terms. To move the field forward and to avoid these mistakes, there is a need for (a) more specific terminology to guide the scientific and practical dialogue and (b) a common framework in which each research effort can be mapped and compared. Such a framework is needed given the interdisciplinary focus of transportation research, in which different disciplines (e.g., engineering, design, social sciences, safety sciences, AI) might also use different terminology. In this paper, a classification framework for examining models and simulations of human-vehicle interaction is introduced. Within the domain of human-vehicle interaction, simulations can happen along three dimensions: agents, environments, and scenarios. These are illustrated in Fig.  1 . Within each of these dimensions, there can be multiple styles or approaches of simulation and modeling. Each of these approaches differs in the extent to which they truly emulate the realistic performance of the agent, the environment, or the driving scenario. The framework enables researchers to map their choice of research methods and tools and compare with other literature, as well as identifying areas of effort that could advance the field.
 Intended contribution and audience A first contribution of this paper is to explicitly define the differences between three dimensions of modeling: agents, environments, and scenarios. This is achieved by describing what is entailed in each dimension. By separating out these three broad dimensions, it becomes clear that the aforementioned assumptions (in the introduction) are flawed, as each assumption only considers a subset of the dimensions of modeling. Being more explicit about these differences provides the field of transportation research with more precision. Such precision is needed to compare results across studies and to aid replication of study results and implementation of ideas and results in actual transportation systems. A second contribution of this paper is to identify areas that are as-yet unexplored or underrepresented within the transportation research community. This is achieved by describing the studies completed for various combinations of simulated agents, environments, and scenarios. In studies where either the environment is simulated or the scenario is constrained, but not both, there is the possibility for future research that allows for tighter control where needed, while also providing insights on a wider, open-ended set of human behaviors. The intended audiences of this paper are researchers and practitioners who are consumers of these simulations, as well as industry and regulatory agencies. For all these parties, the framework provides a way to classify studies and to decide upon the best research method for new studies: what type of simulation is needed? Although the discussion within this paper mostly focuses on human-(motorized) vehicle interaction, simulation is also used in other transportation domains such as trains and flights. Having the three dimensions distinguished is not only important for academic purposes, but also for product development and the testing cycle (e.g., V-Model, Scrum)  (Friedrich et al., 2008) .  1  Testing products with real users (agents), in real environments (the world), with actual everyday scenarios provides the highest ecological validity. However, that might also come with potential disadvantages such as (1) weak reproducibility and generalizability due to changing sensor data, weather, or participants' cognitive states, (2) the impossibility of testing under extreme conditions and, (3) its negative impact on release cycle times. A possible alternative that might help to reduce field testing while ensuring functional safety and reliability is performing safety assessment by stochastic virtual simulation  (Kompass et al., 2015) . Still, a pure virtual test as suggested in the past by for example Google  (Harris, 2014)  is barely able to represent reality with its overwhelming complexity, for instance because of performance differences of virtual sensors, lack of realism and flexibility of driver models and shallow modulation of environment and surroundings. That's why (California's) regulations still stipulate autonomous vehicles must be tested under "controlled conditions" (e.g., a test track or temporarily closed public road  Harris, 2014) . The automotive industry is searching for a new standardized testing process  (Kompass et al., 2015)  to cope with the issues highlighted before. Open questions in this regard are if and to which extent real field trials can be substituted by various levels of virtual simulation  (Riener, 2010) , how to seamlessly integrate different validation methods (e.g., virtual simulation, driving simulator tests, X-in-the-Loop simulation), and, how to guarantee reproducible test conditions. To counteract, tests can adjust the real and virtual parts in various dimensions of the test setup (agents, environments, scenarios). Such adjustments create a "mixed-reality testing framework" (see also,  Riener et al., 2016b) , which is explicit about which components (agents, environments, scenarios) are simulated or not. The  framework that is put forth in this paper, and which is described next, thereby helps to better position such mixed-reality efforts.
 Classification framework for (models and simulations of) humanvehicle interaction The framework distinguishes three dimensions that can be examined in studies of human-vehicle interaction: agents, environments, and scenarios. Each dimension can involve some form of simulation or modeling, and will now be discussed in turn.
 Dimension 1: agent An agent is 'anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators'  (Russell and Norvig, 2003, page 32) . This general definition can be applied to both humans and artificial (non-human) entities, and therefore also to simulations and models of humans. Simulations of human agents are typically assumed to be created by software, but can also rely on hardware components (i.e., in embodied, situated robots  Pfeifer and Scheier, 2001) . The reasons for modeling human behavior can differ, but for the traffic domain typically include: a need to determine the operational domain of the system, a need to influence system or user behavior under conditions that might be risky or unsafe for humans, a need to benchmark human behavior against alternative theoretical predictions of behavior, or as a way to ground system behavior in theory. In the driving context an agent is in charge of perception, judgement and actuation on the driving task or a subset of it. At a broad level, an agent in a vehicle is either human, or artificial. However, within the simulated artificial (non-human) agent classes many distinctions can be made. Three dimensions are discussed next: (A) whether accurate simulation of the human's internal thinking process matter, (B) level of abstraction (or: what part of human behavior or thinking is of interest to the modeler), and (C) modeling approach.
 Does accurate simulation of a human's thinking process matter? A first differentiation is whether simulating the details of the human's internal thinking process matter. Some models might not care about mimicking human behavior and thinking at all. For example, when implementing Society of Automotive Engineers (SAE) level-5  (SAE International, 2018) , the vehicle itself makes automated (or autonomous) decisions, but might not always make the same decisions as humans, or not be at fault to human biases and limitations (e.g., fatigue). Other models might only approximate a small subset of human thought or behavior; for example models that test the impact of random human actions  (Thimbleby, 2007) , or models of traffic flow that only focus on crude estimates of perception and action  (Hoogendoorn and Bovy, 2001) . For models where simulation of human thought and behavior are more crucial, there are gradations in level of detail, ranging from models for rapid prototyping and testing of interfaces  (John et al., 2004; Salvucci et al., 2005) , to testing the effect of specific theories such as strategies of task interleaving (e.g.,  Brumby et al., 2018; Janssen et al., 2012; Jokinen et al., 2020 online first; Kujala and Salvucci, 2015; Lee and Lee, 2019) , to developing detailed broader theories of human thinking (e.g., cognitive architectures  Liu et al., 2006; Salvucci and Taatgen, 2011; Zhang and Hornof, 2014) . In other words: creating an artificial agent can be seen in line with Turing's imitation game  (Turing, 1950) : the goal is to have the agent achieve some behavior, but the details of how this behavior was achieved by the agent do not matter to every modeler. In the classification of agents so far, only the extremes were classified: an agent is either human or artificial. However, due to the advent of semi-automated vehicles, less clear cut examples in between these extremes might emerge. Artificial agents might be in control of part of the driving task, while human agents are in control of other parts. And at times it might not be so clear to a human whether an agent is human or artificial (cf. Turing's imitation game  Turing, 1950) . It is not yet clear how to best classify these in between states, therefore the focus in this paper is on the extreme ends first.
 Level of abstraction A second classification, when trying to model human behavior through an artificial agent, is the level of abstraction. In essence, the question here is: what part of human behavior or thinking is of interest to the modeler? David McClelland phrased it as follows: [cognitive models] "are explorations of ideas about the nature of cognitive processes. In these explorations, simplification is essentialthrough simplification, the implications of the central ideas become more transparent"  (McClelland, 2009) . The quote by McClelland beautifully captures that one model of human behavior or thought (so far) cannot capture all of the complexity of human thinking, but instead requires focus. Various frameworks have been proposed to adjust focus systematically to the objective of the model at hand. Two are discussed next: Marr's levels of abstraction  (Marr, 1982 ) and Newell's timescales of action  (Newell, 1990) . Marr  (Marr, 1982)  proposes to approach human thinking from three levels: computational, algorithmic, and implementation. These levels require an increasing level of detail (see also special issue,  Peebles and Cooper, 2015) . Agent models of driving related tasks occur at each level. Computational models specify why specific behaviors might be appropriate or efficient, without specifying what is done by an agent to achieve this. For example, such models might concern why multitasking in the car can sometimes be experienced as efficient by the user, even though objectively it is distracting  (Janssen et al., 2012) , or why it might be efficient to forget information in general  (Anderson and Schooler, 1991) . Algorithmic level models specify what strategies people use to achieve a goal, without specifying why these are used (computational question) or how this is implemented in the brain. Examples are models of visual attention in multitasking scenarios (e.g.,  Salvucci and Taatgen, 2011; Salvucci et al., 2005; Lee and Lee, 2019; Zhang and Hornof, 2014) . Finally, implementation level models describe how the algorithms are physically realized in the brain, without focusing on what task is achieved (algorithmic) and why (computational). Although such detailed implementation level models are available of cognition in more controlled tasks (e.g.,  Eliasmith, 2013) , to the best of the authors' knowledge there are not yet detailed implementation level models that implement multiple facets of driving. The level of abstraction influences the type of questions the models can address: why, what, or how behavior is realized. Another way of classifying the level of abstraction is using Newell's time scales of action  (Newell, 1990) . This framework requires one to specify at what time scale the behavior is modelled and therefore also what type of data can be used to validate the model (see also  Anderson (2002)  and Chapter 1 of  Salvucci and Taatgen (2011) ).  Newell (1990)  distinguishes four bands, which again are all relevant for specific aspects of driving: Biological (actions over ms, e.g. brain processes underlying ms level differences in braking response times,  Gray, 2011; Lahmer et al., 2018) , cognitive (actions over seconds, such as how eye-movements affect steering movements,  Salvucci and Taatgen, 2011; Kujala and Salvucci, 2015; Lee and Lee, 2019) , rational (actions over multiple seconds to minutes, e.g., how to best interleave attention,  Janssen et al., 2012; Janssen et al., 2019c) , and social (actions over multiple minutes to years, such as development of trust,  Forster et al., 2018) . Again, the time scale of a model determines what types of (research) questions can be addressed and also what type of data is needed to validate such theories, as these need to be in sync with the model: milliseconds (e.g., EEG, fMRI), seconds (e.g., eye-tracking, steering actions), minutes (e.g., behavioral choices), or hours (e.g., duration of travel, fuel efficiency) (see also chapter 1 in  Salvucci and Taatgen, 2011) . Other frameworks for classifying the abstraction level of the model of an agent might also exist. Yet, the core question is: through which "lens" is one looking at human behavior and thought. Do minor changes over milliseconds in the physical implementation of an (artificial network) model matter (i.e., implementation level, biological band), or is there a need to understand why behavior at a societal level changes over years  (i.e., computational level, social band) . Whatever the focus is, some simplifications are needed to allow focus of the research  (McClelland, 2009) . What is important is that these simplifications are not made within the area that is of interest most.
 Modeling approach A third way of classifying models is in the modeling style or approach that is taken. For example, is it mostly conceptual in nature (e.g.,  Janssen et al., 2019a; Janssen et al., 2019d; Wickens, 2008) , describing a theory of a (mechanistic) process (e.g.,  Salvucci and Taatgen, 2011; Lee and Lee, 2019; Zhang and Hornof, 2014; Brumby et al., 2018) , or a (machine learning) data-driven model (e.g.  Fridman et al., 2018) ? These three rough classes (conceptual, process, data-driven) rely increasingly less on theory and more on available data, and can all be relevant for driving models. There is a large breadth and depth in modeling styles available for theoretical studies and applied work  (Oulasvirta, 2019) .
 Summary of modeling the agent and further refinements To summarize, models of humans can be captured under the general label of "agent." Agent models can be classified in various ways of which three options were discussed. Although within each of these classification schemes specific classes were identified as well, sometimes models are a blend. For example, models might be able to tackle all three of Marr's levels (e.g.,  Lewis et al., 2014) , address behavior over multiple of Newell's time-scales (e.g.,  Janssen et al., 2012) , or combine datadriven (machine learning) methods with theoretically-driven process models (e.g.,  Anderson et al., 2016) . In driving situations where part of the driving task is automated (e.g., SAE levels 2,3, SAE International, 2018), there are situations where both a human agent and an artificial agent (i.e., the car's internal reasoning system) are involved in aspects of the driving task. In that sense, the expectation of the authors is that research in the years to come will focus more on shared control between human and artificial agents. In modeling the artificial agent that controls the car, many of the same considerations as were mentioned above hold. These techniques can range from simplified state machines with tight control-loops to more conceptual (flexible) models inferred from naturalistic data behaviors using machine learning. State of the art approaches to modeling an artificial agent are approaching the performance of humans at particular driving tasks by means of deep neural network architectures.
 Dimension 2: environment When environments are discussed in the transportation research communities, the environment can be considered as "the world" which is being modelled or the model which is being presented. Researchers and practitioners often draw a sharp distinction between 'laboratory' or 'simulated' on the one hand, and 'on-road' or 'real' or 'open-ended' driving experiments on the other hand, but do not do a lot to explicate the environment further. At first sight, one might assume that testing in an on-road study provides the richest environment for a study, and therefore to provide the most detailed insight about the human behavior. However, this is not necessarily the case. Whether the richness of the on-road driving environments (i.e., the track or roads on which the vehicle is driving) is captured depends heavily on the model of the environment that is made from the data captured by an instrumented car  (Brackstone et al., 1999) . In an extremely limited instrumentation environment, for example, there might only be one camera pointed at the driver or out the window, and the model of the environment after the fact is very sparse. A richer model of on-road drives may capture the in-cabin environment, the traffic surrounding the car, the GPS, and the data from the CAN Bus of the car. The important thing is that the model enables an understanding of the relevant aspects of the environment for analysis. Limitations to the sensors and the internal model of the simulated environment can hinder the ability to address a research question because of missing contextual cues. By contrast, in a virtual driving environment, the environment is specified by the driving simulation. Even within the virtual environment, the model has variations; passing traffic can be randomly generated, for example, or specified exactingly, car by car. Another consideration for modeling virtual environments has to do with whether the virtual environment is completely fictional (e.g., driving on a different planet than earth), or if it is a virtual replica of a real-world environment; the latter better allows for translational experiments that compare virtual and real world performance for similar driving scenarios  (Blaauw, 1982) .
 Dimension 3: scenarios With the agent and the environment identified, one can then specify the scenarios for testing. If the simulated environment can be seen as "the world", then the scenario can be considered as "the way one moves through the world". That is: what types of situations are encountered or not? The spectrum of scenarios that are being considered in the model or simulation can range from unconstrained where nothing in the environment (real or virtual) is manipulated to highly constrained, where everything that the human user observes was somehow designated by the researcher. This often relates to the operational design domain  (SAE International, 2018)  or the context in which the study is being examined. For example, given any specific world (naturalistic or virtual), one might be interested in how a system and agent act in a scenario where there is fog, or a specific construction works scenario. In more unconstrained (open-ended) scenarios, specific scenarios will be encountered (e.g., fog, construction works) but only as they naturally occur during a drive. Most naturalistic driving studies such as SHRP2 (The National Academies of Sciences Engineering and Medicine, 2019) and UDRIVE (Udrive Consortium, 2019) are designed to be more toward the unconstrained end of the spectrum. The researcher does not constrain the destination, driving behavior, or performance of the user. In other words, the scenarios that people encounter are left open-ended. The models behind naturalistic driving studies are typically designed for observation only, without any intervention. The participants know that their vehicles are instrumented, and, as such, might drive more carefully; nevertheless, the environments (or: the world) that they drive through are not affected by any research interventions, and so the scenarios that occurwhether they are everyday traffic, distracted driving, or near missesare natural. While there are many studies that can be conducted in a real world setting, they are not all 'unconstrained'. Hence, the distinction between the environment and scenario. One might have a fully functional car of which high level of detail can be measured (i.e., high on the 'real' dimension of the environment), but testing is only intended under specific conditions such as construction works, or under conditions of rain (i.e., highly constrained). There are also projects in between these extremes. For example, Volvo's "Drive Me" project  (Victor et al., 2017)  was intended to have everyday drivers experience driving in a car that has higher levels of automation, with some in-car technology achieving SAE-level 4 automation on specific roads  (SAE International, 2018) . Although the project has since scaled back in ambition  (Bolduc, 2017) , the original study was intended to include a fully operational vehicle with high level of data collection and instrumentation (i.e., high realism on the environment scale). However, the automated technology within the vehicle could only operate under specific operational design domains, thereby limiting the scenarios under which researchers could study human behavior in automated vehicles. Although the aforementioned might seem to imply that open-ended scenarios can only be run in naturalistic environments, this is not the case. Simulated environments can also run relatively open-ended, unconstrained studies, depending on how open-ended and realistic the environment is modelled in the simulator and to what extend it allows freedom in actions for the driver. If the simulator has for example a detailed map of all roads in a city (i.e., a "microworld"), and the car has almost all the functionality of a real car, then a wide scala of open-ended scenarios might be possible to simulate.
 What is a "controlled" study? One of the things that the classification framework (Fig.  1 ) makes clearer is the different ways that a "controlled study" is controlled. Each dimension (agent, environment, scenario) can have its own level of control. To make this even more specific, a distinction can be made between different degrees of regulation and fidelity for the agent, environment, and scenario.
 Regulation What is called "regulation" is often referred to as "manipulation" or "control" in experimental settings; it is the degree to which different participants experience the same thing. Regulation can be applied to the agent, the environment, and the scenario.
 Regulation of environment and scenario The most obvious coupling of regulation is perhaps with environment and scenario. Within the environment and scenario, there can be a high degree of variation in control, even within laboratory studies. Simulation environments are often associated with high-degrees of regulation. On the far end, in simulated automated driving scenarios, a driving simulation environment can be so controlled that every single participant experiences exactly the same setting, with exactly the same cars passing at exactly the same time in the simulated experience (i.e., a highly regulated scenario). In such tightly regulated studies, the only variations might come from human action such as the human's eye-gaze or steering actions. Although these studies are appropriate for testing parameters of human behavior and thought (e.g.,  Janssen et al., 2012; van der Heiden et al., 2019) , they are less generalizable to everyday traffic scenarios. Therefore, in the more typical laboratory driving simulation studies, other vehicles in the simulation environment around the participant vehicle are spawned stochastically, so that there is some bounded variation in participant experience; any experiment in which the participant drives introduces yet another source of variation. On the far less regulated end, experimenters such as  Feuerstack et al. (2016)  use the simulation environment as a "theater" in which drivers can collaborate to play out the interactions they have on the road improvisationally  (Schindler et al., 2013) . In on-road research, there is also more or less regulation that is possible. While it is not usually possible to make it so that every participant experiences exactly the same experience, on-road experiments can feature set courses, where every participant experiences the same roads, in roughly the same times of day (like,  Baltodano et al. (2015) ), or set situations, where every participant experiences the same scenario (for example, commuting home) even if they have wholly separate routes, like  Zafiroglu et al. (2012) . In other words, regulation, manipulation and control can be exerted on both the environment (i.e., what types of vehicle interactions are allowed) and on the scenarios that are experienced by the participant. Control can be loose or tight on none, one, or both of these dimensions.
 Regulation of agents Regulation can also be applied to the agent, especially for studies with human agents. A first decision on regulation is how the study participants are sampled. On the very tightly regulated end, participants might come from a very specific population, such as psychology undergraduate students. Tight regulation yet more variation in human behavior might be observed when participant statistics meet those of a specific driving population (e.g., match the national ratio male-female drivers, or distribution of drivers' ages, as in, van der  Heiden et al., 2019) . Finally, regulation might be loose when participants are gathered in a less structural way (e.g., opportunity sampling). Sampling might be particularly important in cases where the behavior might be tied to characteristics of the sample -for example years of driving experience, familiarity with local cultural or societal norms or expectations, experience with semi-automated vehicles, or experience with particular road configurations (e.g., driving on the left-side or right-side of the road). A second decision on regulation is whether the participants are asked to perform with or without manipulation of their cognitive state. For example, participants might be performing after tight administration of alcohol, drugs, or other substances (e.g.,  Martin et al., 2013; Veldstra et al., 2012; Wester et al., 2010) . Or participants might be sleep deprived (e.g.,  Eoh et al. (2005) , for models see e.g.  Gunzelmann et al. (2011)) , or be manipulated into a specific affective state  (Jeon, 2015) . A third decision on regulation of agents is how free the agents are in their actions and how participants are instructed. On one extreme end, naturalistic driving studies (e.g., The National Academies of Sciences Engineering and Medicine, 2019; Udrive Consortium, 2019) might not place any constraints on the driving task: participants drive where and when they want to drive. On the other extreme end, the user's task might be very narrowly defined. This might be akin to for example Fitts' law experiments, in which participants are instructed to make specific ballistic movements over and over  (MacKenzie, 1992) . In between these extremes are studies in which some overarching criteria, such as a user's general priority, is manipulated (e.g., safe driving or fast performance of a secondary task while driving,  Brumby et al., 2011; Janssen et al., 2012) . In other words, regulation can also be applied to the agents of the task, and might be exerted implicitly due to the sampling of participants or the instructions for the task. Although regulation was mostly discussed for human agents, similar concerns apply when human behavior is captured in artificial agents: are these models based on data sets with a wide variety of human participants, or only a tightly regulated sample in a tightly regulated task?
 Fidelity Fidelity has to do with the level which the simulation or model of the agent, environment, or scenario mimics real-world or anticipated futureworld drivers and driving situations. In the lab, the fidelity of simulated environments can range from low, where the scenarios are observed from a bird-eye view, and the operator controls the vehicle using a keyboard, to very high, where drivers are fully immersed in the driving environment. The low fidelity environments are appropriate if the primary goal is to represent strategic level decisions. The more typical driving simulation environments have much higher fidelity and feature different degrees of immersion, where a full-vehicle chassis with a motion base delivers the experience most like driving on the road. Open source game engines like the Unity and Unreal Engines that enable development of rich-graphics simulators such as CARLA  (Dosovitskiy et al., 2017)  has enabled high fidelity driving simulation tools to be more readily accessible to a broader range of researchers and designers. In on-road environments, the fidelity is usually pretty high with real road, weather, and lighting conditions, but the location of the testing environment (for example, on a test track vs on (closed) public roadways) can affect the fidelity of the driving task. For the fidelity of the agent, one can consider artificial agents and human agents. For artificial agents, fidelity can again differ between modeling approaches. Automated driving systems have traditionally employed low-level perception-controllers such as PID to perform driving tasks. These models typically lack flexibility and performance when compared to human agents in variable environments. However richer, higher-level driving automation models are being developed using machine learning techniques that rival or even outperform human drivers  (Grace et al., 2018) . Concurrently, when modeling human agents and their thought processes, some approaches care about the fine-grained details of the cognitive process (e.g.,  Zhang and Hornof, 2014)  while others only use approximations of human perception and action (e.g.,  Hoogendoorn and Bovy, 2001) . For human agents, in some sense the fidelity is "high": an actual human performed the task. However, depending on how the human sample and the behavior of the sample was regulated, behavior of these agents might be more or less representative for a wider population of humans and a wider set of human cognitive and affective states. Another aspect to consider in relation to fidelity is the method and process of data collection. Even in a study with a high fidelity set-up (i.e., human agents, driving in the real world, in an unconstrained scenario), the reliability and usefulness of the study outcomes may be negatively impacted if the data collection protocol was not carefully designed. It is therefore crucial to understand the limitations of your equipment in terms of quality, accuracy, frequency (and resolution) of sampling, and the reliability of the measurement as a proxy for the intended variable (s) of interest. Take for example the fidelity of data collection on a human agent. Let's say a study is interested in capturing a cognitive aspect of the human driver using the relationship between eye movements and vehicle steering movements (actions over seconds cf.  Newell, 1990) . In such a study, eye gaze data would need to be collected at a level that can detect differences within seconds. Further, the eye-tracking device will need to be calibrated and checked for stability and tracking before data collection on each participant. If such protocols are not in place, any inferences made from the data would be incorrect. This becomes an even larger concern when methods and tools are used that have not been independently validated or for which the underlying algorithms are not known. For example commercial devices that can predict "alertness" based on physiological measures may have limited validity if the underlying algorithms cannot be shared or verified. In such cases, the conclusions drawn from such tools would be questionable. Data quality also impacts the dimensions of environment and scenario. For these dimensions, the sensors (or simulation code) might also not register relevant information (e.g., what other traffic is on the road), or not register it frequently and detailed enough (e.g., estimate distance to other cars in increments of 10 cm, whereas increments of 1 cm are needed for analysis) or reliable enough (e.g., when a sensor of the car is obscured). The interface between the data, systems, applications, software and platforms are also important to consider. A seemingly small detail or choice in the set-up of a study can have large implications in the inferences that are made. For example, if the eye-tracking glasses shifted during the study, one might conclude that a participant did not pay sufficient attention to the road, whereas this conclusion was reached due to measurement error. Similarly, if a simulation was not calibrated correctly to identify the distance between the test vehicle and surrounding vehicles, then one might incorrectly conclude that an appropriate distance was held at all times. If these results are not further tested (e.g., by formalizing them in computer simulations of underlying cognitive process, or testing them in replication studies), then the wrong conclusions can steer the larger field in the wrong direction. For example, incorrect results can lead to suggesting designs or software that are not effective or based on incorrect principles (e.g., assuming that a vehicle holds appropriate distance to other vehicles, whereas it does not). 4. Actual, virtual, and mixed reality through simulation on different dimensions: where are the research gaps and opportunities? Using the three dimensions (agents, environments, scenarios), one can now more clearly position research that simulates none, one, two, or three of these dimensions. Studies where none of the dimensions is simulated can be considered "actual reality", studies were all dimensions are simulated can be considered "virtual reality", and those where at least one but not all dimensions are simulated can be considered "mixed-reality".
 Collaboration between human and artificial agents As was already mentioned in the section on agents, one important emerging area is that where human and artificial agents interact in a single environment. This is the case for humans that interact with semi-automated technology (e.g., SAE levels 2,3, SAE International, 2018). In these instances, the reasoning system behind the automation can be considered as an artificial agent that senses and acts upon the environment, but also depends on input from the human. Such environments require a good understanding of the mental model of the human and the mode of the vehicle  (Janssen et al., 2019a) . Another area where human and artificial agents interact is in studies of dyadic interaction. In the bottom of Fig.  1 , such studies are placed in the bottom-right quadrants of studies with human agents (left) and studies with artificial agents (right). In dyadic interaction studies, two or more people are involved in a simulated world, and can see each other's actions through an avatar (or other car) that moves around in their shared virtual world. This form of interaction involves both human agents, but also a virtual representation of the agent, therefore positioning this work both in the cluster of human and artificial agents. A conceptual example of such a study is for example described in  (Doric et al., 2016) . The remainder of this section will explicitly discuss studies in which there is either a human agent, or an artificial (simulated human) agent.
 Simulated scenarios and/or environments with human agents The first consideration is of cases where human agents are involved in the driving (e.g., bottom-left of Fig.  1 ), but either the scenario or the environment might be simulated or constrained. Perhaps one holy grail of research is to observe driving in real environments with open-ended, unconstrained scenarios (top-right quadrant). Examples can be found in naturalistic driving studies (e.g., The National Academies of Sciences Engineering and Medicine, 2019; Udrive Consortium, 2019). The challenge with running these studies is that they can require more resources in terms of time, equipment, money, and personnel to run than traditional simulation studies. They are therefore typically overseen by large consortia, and not a realistic choice or option for individual researchers outside of such a consortium. On the other extreme, both the scenario and the environment can be simulated (bottom-left quadrant in Fig.  1 ). Examples are classical driving simulator studies and test track studies. Within this quadrant, there is a gradation of realism, but in general it makes use of simulation on both axes. Studies like these are more common in the transportation research communities. The reason might be that although they also require extensive resources (e.g., to buy and maintain a simulator), these are more one-off expenses, and cheap alternatives are available, such as a combination of commercial steering wheels with open-source driving environments such as Open-DS  (Math et al., 2012) , and off-the-shelf in-vehicle infotainment simulation environments such as Skyline  (Alvarez et al., 2015) . The more interesting, and relatively under-explored quadrants are those in which either the environment ór the natural scenario is simulated/controlled, but not both. It could be argued that current automated driving systems with functionalities at SAE levels 3 and 4 are tests of constrained scenarios in real environments (top-left quadrant, "on-road real world autonomous driving"). The reason is that such vehicles can function in specific operational design domains (e.g., an adaptive cruise control might only function under regular highway conditions), or to use the terminology from the framework in this paper: in specific (controlled) scenarios. Another example is the Ghost Driver project  (Rothenbücher et al., 2016) , where a very controlled scenario (namely: a car that seems to drive without humans inside it, due to camouflage) is placed in a real naturalistic environment (everyday pedestrian crossings on a campus). This allowed for rapidly testing how humans interact with future technology. The second relatively under-explored area is open-ended scenarios with simulated environments. This can for example be achieved through openended Wizard of Oz simulation studies and improvisational or theater studies  (Mok et al., 2015; Feuerstack et al., 2016; Schieben et al., 2009) . In these cases, the world is simulated in some form (e.g., through a driving simulator or through theater enactment), while also allowing the participant to experience a wide set of scenarios. The benefit of only simulating the environment or the scenario, and not both, is that it requires less resources compared to the naturalistic driving simulators, while at the same time allowing for studies of more naturalistic and less controlled human interaction. The authors see high potential in these research methods for transportation research that wants to explore human interaction with novel (in-car) vehicle technology. With the rapid development of automated technology  (Janssen et al., 2019b) , simulation of the environment and/or scenario allows studies of human interaction with automated technology even if such technology is not (yet) commercially available, or not matured enough to test on the open road.
 Simulated scenarios and/or environments with artificial agents The large majority of studies with simulated human agents (bottomright Figure in Fig.  1 ) also simulate the environment and control the scenario. A special and emerging case is formed by studies from automated driving companies that use full simulation to develop their automated driving technologies. Each billion of miles of driving experience collected on real roads with test fleets are complemented with several orders of magnitude more in simulated environments. Using tools like Carcraft  (Madrigal, 2017) , technology companies can identify interesting driving scenarios and iterate through a large number of derived conditions using virtual models of vehicles and other road users in a cost-effective manner. For these studies, the artificial agent acts somewhat like a human, but the focus is mostly on the impact that the human has on the technology, road behavior, and safety. A related but different perspective is taken by studies where simulations of a human agent are used to better understand the human mind. Perhaps one of the best examples is Distract-R  (Salvucci et al., 2005) , and its associated cognitive models  (Salvucci and Taatgen, 2011) . In Distract-R, a simulated agent drives in the same virtual environment as is used in human studies. Distract-R interacts with the car and the environment through its virtual hands and eyes. More often than not, other agent models have even more controlled and limited interaction with the environment. For example, steering actions might be achieved through a simple mathematical function (e.g.,  Janssen et al., 2012) , or the model might simply model traffic flow of a hypothetical scenario as the concern is not with the behavior of any individual model but with the collection/flock of cars (e.g.,  Hoogendoorn and Bovy, 2001) . Cases where a simulated agent acts only in a real environment, but with a constrained scenario (top-left quadrant in Fig.  1 ) include crash test dummies. These simulate a specific scenario (i.e., a crash) in a real environment to test the impact the crash has on the vehicle and the simulated human (a dummy) inside it. Other studies might use configurations of the automation to elicit particular emotional responses on drivers and passengers as a result of its driving reactions to road events  (Alvarez et al., 2019) . Yet another example includes the Stanley parking robot  (Stanley Robotics, 2019) , which can park your car within a constrained scenario (parking garage). Cases where a simulated agent acts in a simulated environment but a more open-ended scenario (bottom-right quadrant) include studies of dyadic interaction. Note that these involve in a sense both artificial agents (avatars) and real human agents. Other examples of work in this environment are simulators where people perform relatively open-ended interaction with automation: a manual driver encountering an autonomous car in the simulated road, pedestrian-AV interaction (e.g.  Mahadevan et al., 2018) , or bicyclist-AV interaction (e.g.  Faghri and Egyháziová, 1999) . The final fourth quadrant (top-right) is one where a simulated human drives in a natural scenario within a real environment. The ideal example is formed by studies using a SAE level 5 (SAE International, 2018) fully automated and autonomous vehicle. At the moment such technology does not yet exist. Instead, systems with limited automation, that can drive in specific operational design domains (e.g., specific scenarios) achieve part of this functionality. There are also other examples around, for example recent studies have used dummies of pedestrians (that walk similar to real pedestrians through motion capture studies) to test how automated vehicles respond to these pedestrians in various open-ended natural scenarios in a real environment  (Doric et al., 2017; Cañas et al., 2018) . This can be interpreted as a more open-ended form of the "crash test dummies", as the dummy in the pedestrian study has more movement and can act in more unconstrained scenarios.
 General discussion This paper provides a framework for examining human-vehicle interaction with respect to three dimensions that can involve simulation or modeling: agents, environments, and scenarios. The claim is not that one form of simulation is better than others, but rather, each dimension provides insights on different but complimentary (research) goals. Each simulation method targets different objectives, and associated strengths and limitations. Moreover, within one research project, researchers might be simulating none, one, or many of these three dimensions. Although modeling and simulating is sometimes thought of as a way to exert control (i.e., exert regulation while achieving fidelity), each of the dimensions can differ in how much regulation is exerted and how much fidelity is achieved. Having more precise terminology to study modeling and simulation is useful for transportation sciences, given its interdisciplinary nature. Contributions to the field of transportation science are made from among others engineering, design, social sciences, and safety sciences. Each of these fields brings its own terminology and the same words or terms might differ in meaning across fields. The aim behind the framework in this paper (i.e., Fig.  1 ) is to aid precision when discussing models and simulations. The development of this framework helped to identify areas that are currently not frequently used within the transportation research communities. Specifically, the paper identified that there is room for studies which simulate either the environment or the scenario, and not both. Such studies are useful as they require less resources compared to the naturalistic driving studies (that simulate none of the dimension), while at the same time allowing for more natural (high fidelity) and less regulated human interaction than studies that simulate both the environment and the scenarios. In effect, this allows researchers to study interaction with future prototypes relatively easily in naturalistic settings. Another area that was identified are studies in which human and artificial agents interact. Examples include studies of human interaction in semiautomated vehicles (e.g., at SAE levels 2 and 3, SAE International, 2018) and studies of dyadic interaction in simulated environments. A practical concern of researchers is that they might not always have the right resources, infrastructure and skills to conduct studies in all of the identified quadrants in Fig.  1 . Fortunately, the areas that were identified as having potential for future work (by comparison) do not necessarily rely on large infrastructure or resources.
 Benefit of the framework for the transportation research community Another benefit of the framework for the transportation research community is that it provides a systematic way to structure studies in which simulated and non-simulated studies can be tested. The need for a consistent tool chain containing both virtual and real tests was already highlighted (e.g., by,  Spies and Spies, 2006; Schuldt et al., 2015) . The value of the framework for the community is that it makes explicit that there are three dimensions on which the extent between "virtual" and "real" can be varied. For example, to allow testing of different (safety-critical) scenarios (dimension 3) without the risk of injuries, the proposed framework identifies the need to integrate both automated and manual vehicles (dimension 2: environment) with modelled and real human agents (drivers, pedestrians, cyclists) (dimension 1: agents) into a single test bed. Based on the actual configuration of the individual (real) parts, subjects can face different levels of immersion. A human participant can, for instance, feel the real kinesthetic experience when driving with a real vehicle on a closed test-track as the actual scenery is presented to them using virtual reality devices. On the other hand, motion tracking of participants in a simulated environment (e.g., CAVE  (Cruz-Neira et al., 1992)  or driving simulator) allows integrating vulnerable road users into the same scene in a similar manner. The resulting scenario, now containing realistic human behavior, can finally be injected into the control unit or sensor system of the test vehicle (automated car) or presented to it using movable dummies on a real test track. In that sense, the proposed framework tries to bridge the gap between simulation and expensive real driving tests  (Frey, 2016; Kühbeck and Huber, 2015) . Different researchers and practitioners will have different (research) questions. These questions will influence which dimension (agent, environment, scenario) is most important to keep under little or high regulation and fidelity. For example, in a study that is mostly focused on studying the human mind, tight control over the environment and scenarios might be needed to ensure that one can study the cognitive principle of interest. By contrast, in a study of an actual vehicle, one might want to keep the environment as realistic as possible and also allow humans to express a wide variety of behaviors, so as to be able to see the impact of such naturalistic behavior.
 Translational research An open question is how results from one setting can be translated into another settings. Specifically: how can conclusions that were drawn in a simulated environment translate to its non-simulated equivalent? Within the field of cognitive science, there have been various studies that looked at how simulations of theories of human behavior and thought relate to actual human performance (see section on modeling the agent). For the dimensions of environment and scenarios, studies comparing "simulated" with "real" environments have typically looked at these dimensions conjoined, by comparing the validity of real (on-road) experiments with simulator studies, e.g.,  (Wang et al., 2010; Riener, 2010; Frey, 2016) . The framework of this paper (Fig.  1 ) already suggests that there is value in separating these two dimensions. Nonetheless, lessons can already be learned about transfer from one setting to the next. For example, in  (Riener, 2010) , driving performance and interaction with an in-car interface was compared between both a lowfidelity driving simulator and an on-the-road driving experiment (i.e., mostly manipulating environment, and slightly manipulating scenario due to the natural variety of traffic conditions in on-the-road driving). Results indicated that drivers respond faster to steering requests in the driving simulator (by 13%) as compared to real driving. The explanation for this difference can most likely be derived from the fact that participants encountered fewer demands in the first (simulated environment) compared to the second (naturalistic environment) setting. However, there were also parallels. For example, the rank-order of performance with different incar interfaces was the same in the simulator and in the on-the-road study. Unfortunately, based on this study and others, the consensus is that it is not possible to derive a simple (e.g., linear) conversion factor or table to describe effects emerging in the reality with results from simulation or simulator studies. Although translational research in transportation sciences is typically focused on translating from simulation (or model) to the realworld, there is an important reason to look at translation in both directions: simulations and models are helpful in exploring a wider variety of situations and contexts, including extreme cases that might not always happen even after prolonged testing in real-world conditions (humans in real environments with unconstrained scenarios). Take the example of Ghost Driver  (Rothenbücher et al., 2016) , which was aimed at testing how humans react to technology that is not yet commercially available (fully self-driving cars). By simulating the scenario (a car that seems to drive without humans inside the vehicle using camouflage) in a real naturalistic environment (everyday pedestrian crossings on a campus), it allowed for rapidly testing how humans interact with future technology. Without the simulation of the car, the response of real pedestrians in their everyday environment would have been difficult if not impossible to test. Another value of simulations and models is that they can help distill the scientific principles and mechanisms that are at the heart of a behavior or situation, and thereby aid understanding. This is consistent for example with the original ambitions of AI, as expressed in the proposal of the famous 1956 Dartmouth workshop: "The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it"  (McCarthy et al., 2006) . Given the value of both simulated and non-simulated research, one implication is that research should cover multiple of the quadrants in Fig.  1  before definitive behavior about human-vehicle interaction are made, for example for regulatory purposes. This aids in understanding both how behavior is in less constrained, real world conditions but also allows for more rigorous, focused testing under controlled experimental conditions.
 Limitations There is an opportunity to position future studies within each of the three dimensions. In this paper, the dimensions were based on the extreme conditions to contrast human agents with non-human agents, simulated road environments with real (on-the-road) environments, and constrained scenarios with open-ended, unconstrained scenarios. For each dimension these extremes might be clear, and a relative position of any two studies might also be possible for each dimension. However, it will be difficult to associate each study with an absolute number, position, or rank on each dimension such that studies can be directly quantified and compared to future studies. Although some form of ranking or rating might be desirable in practice, it would not do justice to the inherent diversity of options that is available within each dimension. For example, even within the set of simulated vehicles there is a myriad of characteristics that can differ along multiple dimensions (e.g., visual realism, ability to induce motion, types of actions that are allowed by the human driver), and equating each dimension into a number would require comparing apples with oranges. Similarly, even for a category such as human or non-human agents, there might be more than a binary distinction, in that artificial agents can differ on multiple dimensions (e.g., Marr's level of abstraction,  Marr (1982)  and Newell's time scale  Newell (1990) ) and modelled using various frameworks  (Oulasvirta, 2019) . Moreover, in line with Turing's "imitation game"  (Turing, 1950) , future artificial agents might be hard to distinguish from human agents. Apart from refinement within the levels, there is also room to consider other dimensions that have so far not yet been included explicitly in the framework. For example, the discussion of the agents dimension in this paper has mostly considered human drivers, yet there can also be humans that have other roles inside the car (e.g., passenger, navigator) and outside the car (e.g., cyclists, pedestrians,  Doric et al., 2016) .
 Conclusion Studies of human-vehicle interaction can entail modeling or simulating the agent, the environment, or the scenario. Although colloquially researchers in the transportation research community and related communities sometimes only distinguish "simulated" from "non-simulated" settings, this paper identified that most studies typically model only some of these three dimensions, and that different levels of regulation and fidelity can be exerted on each dimension independently. The explicit distinction of agent, environment, and scenario can aid researchers and practitioners who are consumers of these simulations, as well as industry and regulatory agencies. The framework provides a way to classify studies and assist researchers, engineers, and designers make better decisions regarding the simulation tool to use for the research question of interest.
 CRediT authorship contribution statement The author sequence was determined using the SDC approach. All authors contributed to the conceptualization and the writing of the paper. Christian Janssen coordinated all activities. Linda Boyle developed the visualization. Fig. 1 . 1 Fig. 1. Three dimensions of simulating related to human-vehicle interaction with example studies indicated (see also text).

Abstract: This paper provides a framework for examining human-vehicle interactions with respect to three dimensions that can involve models or simulations: the agents, the environments, and the scenarios. Agents are considered on a spectrum from human to artificial actors. Environments are considered on a spectrum from simulated to real. Scenarios are considered on a spectrum from constrained to unconstrained. It is argued that these three dimensions capture key differences in research approaches within the field of human-vehicle interaction, and that explicitly situating research and discussions within this framework will allow researchers to better compare and contrast research outcomes and contributions. The framework is used to locate different disciplines in the community with respect to one another, and to identify areas which are as-yet unexplored.


Formal Verification of Hybrid Systems

REVIEW OF CURRENT APPROACHES Model-based design offers a promising approach for detecting and correcting errors in early stages of system design  [33, 37, 49] . In this methodology, a designer first constructs a model, with mathematically precise semantics, of the system under design, and performs extensive analysis with respect to correctness requirements before generating the implementation from the model. Embedded systems, such as controllers in automotive, medical, and avionic systems, consist of a collection of interacting software modules reacting to a continuously evolving environment. The appropriate mathematical model for design of embedded control systems is hybrid systems that combines the traditional models for discrete behavior with classical differential-and algebraicequations based models for dynamical systems. Such models can capture both the controller -the system under design, and the plant -the environment with continuously evolving physical activities in which the system operates. Given that (1) automated verification tools have recently been successful in finding bugs in "real-world" hardware protocols and device drivers  [13, 15] , (2) tools such as Stateflow/Simulink are commonly used in automotive and avionics industry for modeling, and (3) high assurance is a necessity in safetycritical applications that deploy embedded software, formal verification of hybrid systems has been a vibrant research area for the past 20 years. This article is an overview of current research directions, aimed at providing an introductory "roadmap" rather than a comprehensive survey.
 Modeling In early 1990s, formal models for discrete reactive systems were integrated with models for dynamical systems  [2, 41] . The model of hybrid automata  [1, 2, 29]  has emerged to be a popular choice. A hybrid automaton is an extended finitestate machine whose state consists of a mode that ranges over finitely many discrete values and a finite set of realvalued variables. Each mode is annotated with constraints that specify the continuous evolution of the system, and edges between modes are annotated with guards and updates that specify discrete transitions. For example, the behavior of a self-regulating thermostat can be described by a hybrid automaton with two modes, on and off, and a single real-valued variable T modeling the temperature. To specify how the temperature changes in the mode on, we can annotate the mode with a differential equation, say, Ṫ = k(70 − T ), for a constant parameter k. Alternatively, we can use a differential inequality, say, k 1 ≤ Ṫ ≤ k2, for two constants k 1 and k2, to specify the dynamics approximately using only bounds on the derivative. Associating the mode on with an invariant constraint T ≤ 70 specifies that the system must exit this mode before the temperature exceeds 70. An edge from the mode on to the mode off with the guard T ≥ 68 specifies that, whenever the temperature exceeds 68, the system can discretely change its state by updating the mode to off. A hybrid automaton can naturally be interpreted as an infinite-state transition system, and this forms the basis for formalizing classical notions such as safety verification, property-preserving abstractions, and simulation relations, for hybrid systems. Hybrid automata are analogs of state machines, with little support for structured descriptions, and consequently, a number of formalisms have been proposed to facilitate modular descriptions of complex systems. These include modeling environments such as Shift  [17]  and Ptolemy  [19]  for hierarchical specifications of hybrid behavior; models such as hybrid I/O automata  [40] , hybrid modules  [6] , and Charon  [3] , for compositional treatment of concurrent hybrid behavior; and differential dynamic logic for logic-based specification and compositional analysis of sequential hybrid behavior  [43] . The commercial modeling tools such as Stateflow/Simulink (see www.mathworks.com) and Modelica (see www.modelica. org) are routinely used in a wide range of industries. Conceptually, it should be possible to compile models expressed in such tools into formal notations such as hybrid automata. This has turned out to be difficult in practice due to the richness of features in commercial tools and the lack of a standardized formal semantics of such features (see  [35]  for efforts aimed at semantics-preserving transformations across modeling notations).
 Symbolic reachability analysis In the safety verification problem for hybrid systems, we are given a hybrid systems model M , a set I of initial states of M , and a set S of "safe" states of M , and we want to check whether every execution of M starting in an initial state always stays within the set of safe states, and if not, report a violating execution as a counter-example. For instance, given the hybrid systems model of a collision avoidance protocol, we want to check whether the distance between two vehicles stays greater than the safety threshold for given initial conditions. Let us first note that the safety verification problem of hybrid systems cannot be solved algorithmically: with the exception of classes that severely restrict the allowed dynamics of real-valued variables such as timed automata  [5]  and initialized rectangular automata  [32] , the safety verification problem is undecidable  [1, 32] . Symbolic reachability algorithms for safety verification try to compute the set R of reachable states of a hybrid system M in an iterative manner starting from the set I of initial states. The algorithm checks, at every step of the iteration, if the current set R of reachable states is a subset of the set S of safe states, and if not, it terminates with a counter-example. In general, there is no termination guarantee, as the algorithm may keep adding more and more states to R without being able to deduce that the system is safe. The key challenge to efficient implementation is to identify a suitable representation for the set of states that supports the operations used by the iterative reachability computation. The tool HyTech was the first model checker to implement symbolic reachability analysis for hybrid systems  [7, 31] . For a hybrid automaton with n real-valued variables, each reachable set is represented by associating a finite union of ndimensional polyhedra with each mode, where a polyhedron is represented as a conjunction of linear inequalities over variables. Such a polyhedra-based representation is appealing due to the use of polyhedra in many computing applications and the availability of open-source libraries for manipulating them (c.f.  [12] ). The models are restricted to the class of linear hybrid automata (LHA): guards, updates, and invariants involve only linear expressions, and the dynamics is specified using differential inequalities that are linear constraints over first-order derivatives. For example, the LHA-admissible dynamics ẋ = ẏ ∧ 1 ≤ ẋ ≤ 2 describes two-dimensional motion along the diagonal line with bounds on speed. For LHA, the polyhedral representation is closed under both discrete transitions corresponding to mode-switches and continuous evolution according to differential constraints in a mode: given a polyhedron R describing the set of current states, the set R of states that the system can reach after a discrete mode-switch to a mode m, is a polyhedron that can be computed effectively from R, and the set R of states that the system can reach as a result letting it evolve continuously according to the dynamics associated with the mode m, is also a polyhedron that can be computed effectively from R . The most commonly used dynamics in mathematical design of control systems involves linear differential equations: if x represents the vector of state variables, u represents the vector of input variables, a linear system is described by the equation ẋ = Ax + Bu, where A and B are matrices of appropriate dimensions. Such dynamics is not allowed in LHA. Linear hybrid systems (LHS) refers to the class of hybrid automata where guards, updates, and invariants involve only linear expressions, and the dynamics is specified using linear differential equations. First note that the polyhedral representation is not closed under continuous evolution specified by linear differential equations: if x = 1 is the initial state and the dynamics is given by the linear differential equation ẋ = x, the set of reachable states is the exponential curve given by e t , for real numbers t ≥ 0. Since manipulating transcendental functions is computationally difficult, a popular strategy, first advocated by the tool Checkmate  [14] , and later refined by the tool d/dt  [11] , is to compute overapproximations of reachable sets using polyhedral representations. Given a polyhedron R representing the set of current states, to compute the set R of states that are reachable within a fixed time horizon Δ according to a given linear differential equation, the algorithm implements the following strategy. For every corner vertex v of R, it first computes the state v of the system at time Δ starting in state v. Then it computes the convex hull R 1 of all the corner vertices v of R and their respective images v . We are guaranteed that all the system trajectories start in the polyhedron R 1 and end in R 1 at time Δ, but are not necessarily inside R1 at intermediate times. The final step involves computing the polyhedron R 2 obtained by "face-lifting" R1: the number and normal vectors for facets of R 2 coincide with R1, but the facets are shifted outwards so as to include all reachable states upto time Δ. All these steps can be efficiently implemented for linear systems, and the resulting polyhedron R 2 is guaranteed to be a superset of the desired set R . The process can be repeated for successive time intervals, and the resulting approximation of the reachable set is called the flowpipe approximation. The complexity of operations on polyhedra is exponential in the number of dimensions (that is, the number of real-valued variables of the hybrid system), and since such operations are invoked repeatedly in symbolic analysis based on polyhedral representation, a significant body of work has been aimed at battling this complexity and/or replacing polyhedra with alternative representations  [11, 21, 36, 42] . Representation using zonotopes and support functions  [22, 26]  has so far resulted in the most scalable approach for the analysis of linear hybrid systems, leading to the tool SpaceEx that is able to analyze, for instance, a complex 28-dimensional helicopter controller  [20] .
 Deductive verification In deductive verification, a designer interacts with a mechanized theorem prover to generate proofs of correctness of systems. For safety verification of discrete systems, a classical proof principle relies on the notion of inductive invariants: to show that all executions of a system M starting in an initial set I stay within a safe set S, we identify a state property ϕ such that (1) all initial states satisfy ϕ; (2) ϕ is a subset of the desired safety property S; and (3) the property ϕ is preserved locally across system transitions (that is, no transition of the system changes the value of ϕ from 1 to 0). In interactive verification, the user proposes a property ϕ, and the analysis tool checks if ϕ is an inductive invariant. The concept of inductive invariants for discrete systems has been generalized and adopted to continuous-time dynamical (and hybrid) systems. We will informally explain the first such notion, called barrier certificates  [47, 48] . To show that a dynamical system M with dynamics ẋ = f (x) with initial set I satisfies a safety property S, we identify a function ψ from the states to reals such that (1) in all initial states, the value of ψ is nonnegative; (2) in all unsafe states (that is, states not in the safe set S), the value of ψ is negative; and (3) the Lie derivate of ψ with respect to the vector field f is positive on the boundary set (called the barrier) characterized by ψ(x) = 0. The first two conditions ensure that the barrier separates the initial states from the unsafe states, and the third condition ensures that system trajectories cannot escape the barrier from inside as at the barrier the flow field points inwards. Together, these conditions imply that ψ(x) ≥ 0 is an inductive invariant of the system. Typically, the desired function ψ is a polynomial function of the system variables. It is also worth noting that the barrier certificates are closely related to the notion of Lyapunov certificates for stability in classical control theory. The notion of differential invariants generalizes barrier certificates  [44] : it relaxes the third condition, and also allows more general forms of logical assertions as potential invariants. Verification of safety properties based on identifying inductive invariants avoids the iterative calculation of reachable state sets and is not limited to linear systems. The tool KeYmaera offers support to prove correctness of hybrid systems using deductive verification  [43, 44] . To check whether a given polynomial certificate satisfies all the conditions necessary for it to be a barrier certificate, the tool needs to perform symbolic differentiation, and calculations such as simplification and quantifier elimination, with formulas in the theory of reals with arithmetic operators. To fully automate deductive verification, such a tool needs to automatically generate candidates for inductive invariants, and this remains an active area of research (see  [28, 50]  for automatic generation of invariants by instantiating templates and  [45]  for generating invariants by fixpoint computation).
 Abstraction An abstraction A of a model M is a "simplified" model obtained from M such that proving safety and temporal properties of A is a sufficient condition for proving the corresponding properties of M . Abstraction is an effective strategy for scalability of verification tools, provided there is a way to compute A from M in a tool-supported manner, and a way to refine the current abstraction A if it is not adequate to prove the desired properties. In the case of hybrid systems, the simplicity of the abstract model A can be of various forms: A can be discrete while M is continuous; A can have linear dynamics while M has non-linear dynamics; and A can be a linear model of dimensionality lower than that of M . There is an extensive literature on automatic abstraction of hybrid systems. We note three representative examples. We have already seen that the dynamics admissible in the model of linear hybrid automata is simple enough to permit exact computation of reachable states using polyhedral representation. In phase portrait approximation  [30] , the dynamics ẋ = f (x) in a mode m of a hybrid system is replaced by l ≤ ẋ ≤ u, where the vectors l and u represent the lower and upper bounds on the function f over the range of values specified by the invariant constraint associated with the mode m. This clearly yields an over-approximation of the allowed system trajectories in each mode. The error introduced by the approximation can be reduced if we split the mode m into submodes, each corresponding to a different region of the state-space. Predicate abstraction is a powerful technique for extracting finite-state models from complex, potentially infinite-state, systems, and has been extended and adopted for hybrid systems  [4, 16] . In this approach, the input to the verification tool consists of a linear hybrid system, the safety property to be verified, and a finite set of Boolean predicates over system variables to be used for abstraction. An abstract state is a valid combination of truth values to the Boolean predicates, and thus, corresponds to a polyhedral set of the concrete state-space. The verifier performs an on-the-fly search of the abstract system by symbolic manipulation of polyhedra, where the computation of continuoustime successors of abstract states can be performed using flow-pipe approximations. The key computational benefit is that the continuous reachability computation is applied only to an abstract state, instead of intermediate sets of arbitrary complexity generated during iterative computation. If the initial choice of predicates is too coarse, the search finds abstract counter-examples that are infeasible in the original hybrid system, and such counter-examples can be analyzed to discover new predicates that will rule out related spurious counter-examples. A classical notion of equivalence of nondeterministic systems is simulation: a relation between states of two systems is a simulation relation, if (1) two related states have identical observations, and (2) whenever two states are related, for every transition from the first state, there exists a matching transition from the second state such that the targets of the transitions remain related. For simpler classes of hybrid sys-tems such as timed automata and O-minimal systems, one can algorithmically compute the maximal simulation relation over the states of a given system, and use the discrete quotient with respect to this relation as the abstract system which can replace the original system for verification purposes  [8] . In the context of hybrid systems, since states contain real-valued vectors, there is a natural metric over states, and this can be used to also define a coarser notion of simulation called approximating simulations: an approximating simulation relation with parameter ε requires observations of two related states to be ε-close of one another, and transitions can be matched step-by-step by staying ε-close  [23] . The resulting theory of approximating relations leads to algorithms for constructing lower-dimensional abstractions of linear systems  [52] .
 EMERGING RESEARCH DIRECTIONS Automated verification is a computationally intractable problem. Consequently, even though there are many demonstrations of interesting analyses using tools for verification of hybrid systems, scalability remains a challenge, and a significant fraction of the current research is aimed at addressing this challenge. A complementary challenge is to integrate verification tools and techniques in the design flow so as to improve the overall system reliability. We conclude this article by discussing some promising research directions towards this goal.
 Symbolic simulation In simulation, a possible execution of the model upto a finite time horizon is obtained using numerical methods, and this is a well-accepted industrial practice. A single simulation corresponds to a specific choice of inputs. A promising idea is to analyze a simulation trace corresponding to a particular choice of inputs using symbolic analysis techniques to compute the space of inputs that are close enough to the chosen one so that no inputs from this space need to be considered for subsequent simulations (see  [9, 18, 34]  for recent efforts). This integration of simulation and symbolic analysis can lead to improved coverage, and is similar to concolic testing which has proved to be effective in debugging of large-scale software systems  [24] . An added benefit is that such an approach can be implemented directly within the native simulation engine of an industrial-strength modeling environment such as Simulink/Stateflow.
 Synthesis Historically, synthesis refers to the process of computing an implementation (the "how") from a specification of the desired behavior and performance (the "what") and the assumptions on the environment (the "where"). In the more recent view, the synthesis tool facilitates the design by consistently integrating different views: a designer expresses her insights about the design using synthesis artifacts of different kinds such as models that may contain ambiguities and declarative specifications of high-level requirements, and the synthesis tool composes these different views about the structure and functionality of the system into a unified concrete implementation using a combination of algorithmic techniques. Illustrative examples of this new view of synthesis include programming by examples for spreadsheet transformations in Microsoft Excel  [27] , and sketching of bit-streaming programs using program skeletons  [51] . We believe that these ideas emerging in the programming languages community should be explored in the context of design of hybrid systems to integrate synthesis in a pragmatic way in the design cycle (see  [53]  for recent work on synthesizing switching conditions in hybrid automata).
 From models to code Generating embedded software directly from high-level models, such as hybrid systems, is appealing, but challenging due to the wide gap between the two. In current practice, this gap is bridged with significant manual effort by exploiting the run-time support offered by operating systems for managing tasks and interrupts. A key challenge to systematic software generation from hybrid models is to ensure that one can infer properties of the software from the properties of the model, and this problem is receiving increasing attention from researchers. Sample research directions include integration of control and scheduling  [10]  and static analysis of errors introduced by finite precision computations  [25] .
 Industrial applications The value of formal modeling and verification on industrially relevant problems has been demonstrated on a number of case studies. Examples of these include design and analysis of vehicle platooning protocols  [17] , identification of optimal tolerances for audio control protocol  [31] , safety verification of collision avoidance protocol for aircrafts  [46, 54] , and verification of adaptive cruise control  [39] . Yet, the level of commitment from embedded software industry remains limited to exploratory projects in collaboration with academic researchers. This is in contrast to, say, Intel's investment in formal hardware verification and Microsoft's investment in static analysis of software, which can be attributed to the identification of specific classes of errors that can be largely eliminated using verification techniques (for example, deadlocks in cache coherence protocols and misuse of API rules by third-party device driver code). Thus, a key challenge for research in formal verification of hybrid systems is to identify a compelling class of errors that designers routinely make and can be eliminated using verification techniques. An alternative path to industrial adoption is to integrate verification tools in the certification process, and this seems plausible in safety-critical domains such as software for medical devices  [38] . 			 Authorized licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 00:46:37 UTC from IEEE Xplore. Restrictions apply.
			 licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 00:46:37 UTC from IEEE Xplore. Restrictions apply.

Abstract: In formal verification, a designer first constructs a model, with mathematically precise semantics, of the system under design, and performs extensive analysis with respect to correctness requirements. The appropriate mathematical model for embedded control systems is hybrid systems that combines the traditional state-machine based models for discrete control with classical differential-equations based models for continuously evolving physical activities. In this article, we briefly review selected existing approaches to formal verification of hybrid systems, along with directions for future research.